A guide to online robots/spiders/web crawlers/web ants
Robots are programs which go off and can trawl through websites and all documents that are referenced within. Most people know them for their indexing service like the Googlebot, but they also have a few other functions including: