Web Crawler

The Project

The client, a leading online shopping firm, outsourced the Web Crawling project to Tricom, to crawl sites that have different complexities. With a new change in developing web applications came a whole set of new challenges that their existing crawler failed to render.
The Challenge

The major challenges confronting the client were:

  • Crawling sites that have JavaScript/AJAX.
  • Defined time for crawling big sites.

The Solution

Tricom’s crawling assisted the client in overcoming the challenges by:

    • Developing customized crawlers that were site specific. We had performed an experiment of running our crawling framework over a number of representative AJAX sites. We analyzed the overall performance of our approach, evaluated the effectiveness in retrieving relevant clicks, assessed the quality and correctness of scraping contents and examined the capability of our crawlers on real sites used in practice.
    • Smartly balancing the load, thereby reducing the total time for crawling the huge site.

     

   Highlights

Specialized in electronic management of business documents

Web Crawler: Case Studies

The client, a leading online shopping firm, outsourced the Web Crawling project to Tricom for crawling sites that have different major complexities … more++