Search robots, also known as bots, wanderers, spiders, and crawlers, are the tools many web indexing engines, such as Google , Bing , and Yahoo! , use to build their databases.
Most robots work like web browsers, except they don’t require user interaction. Robots access web pages, often using links to locate and link to other sites. They can index titles, summaries, or the entire contents of documents much more quickly and thoroughly than a human could.