Academic Journals Database
Disseminating quality controlled scientific knowledge

Web Crawler: A Review

Author(s): Dhiraj Khurana | Satish Kumar

Journal: International Journal of Computer Science and Management Studies
ISSN 2231-5268

Volume: 12;
Issue: 01;
Start page: 401;
Date: 2012;
VIEW PDF   PDF DOWNLOAD PDF   Download PDF Original page

Keywords: Crawler | Optimization | Duplicate

In a large distributed system like the Web, users find resources by following hypertext links from one document to another. When the system is small and its resources share the same fundamental purpose, users can find resources of interest with relative ease.However, with the Web now encompassing millions of sites with many different purposes, navigation is difficult. WebCrawler, the Web’s first comprehensive full-text search engine, is a tool that assists users in their Web navigation by automating the task of linktraversal, creating a searchable index of the web, and fulfilling searchers’ queries from the index. Conceptually, WebCrawler is a node in the Web graph that contains links to many sites on the net, shortening the path between users and their destinations.

Tango Rapperswil
Tango Rapperswil

     Affiliate Program