Spiders, Crawlers and Bots Oh My!

Bots, Spiders, Crawlers are all nick-names for the little programs that are constantly scouring the Internet indexing every file they find. You can see which Bots have visited your Web site in your site’s stats.

For example, Googlebot is Google’s little indexing program. These Bots are how search engines become aware of your site and new additions/pages so you can be listed accordingly. Understanding how they work is pretty important to every site owner because you can control what they index.

If you don’t want to let these critters in to index certain areas of your site, (images, member or private directories for example) it is important that you set up a robots.txt file. These little algorithms, at least the well-behaving ones, will look for that file before they proceed in case you want to designate areas on your site/server that you do not want indexed.

Well-behaving ones? Yep, there are nasty little devils that ignore your robots.txt file requests. Shame on them!

Read up and understand what Webcrawlers, Webrobots, Bots, Crawlers and Spiders are all about @ http://en.wikipedia.org/wiki/Web_crawler.  Then, use that information to create your robots.txt file and control what information on your site is indexed.

At your service,
Judith