Andy Jarrett // Code. Develop. Create.

A guide to online robots/spiders/web crawlers/web ants

Robots are programs which go off and can trawl through websites and all documents that are referenced within. Most people know them for their indexing service like the Googlebot, but they also have a few other functions including:
  • Indexing
  • HTML/Link Validation
  • "What's new" monitoring
  • Mirroring
To know more about these programs and how to control their visit through your site check out http://www.robotstxt.org/wc/robots.html

I’m here, learning and working away. If you liked this content and want to keep me going, consider buying me a coffee. Your support keeps this site running and the coffee brewing! ☕️