Search engines use what are called spiders, robots or crawlers to scan the web for content. A search engine crawler is programed by the search engine (such as Google) to scan websites for links and to discover other websites, new content or broken content. Once this information is gathered by the search engines then it is indexed in their database. When a user does a search they then scan their database for all the pages that relate to the query that was entered into the search box.

Over the years these search engine spiders have become smarter at determining a pages wealth and to also flag websites that appear as spam. These robots are essential to the workings of search engines and to those who use search engines.