In search engine terminology a spider is part of a search engine that regularly scans the Internet for new or updated content that the search engine needs to compile and maintain its index of Web pages and files. A spider scans, or spiders the Web to collect entries for its search-engine indices. A spider is also known as a crawler or bot. Most search engines rely on a combination of propriety spiders and ranking algorithms for indexation and ranking purposes.
A spider crawls the Internet by following the hyperlinks that connect the Web's pages and reviewing the contents of each of the pages it visits. Based on that review, the spider determines if and how to rank the page in its listings.
