Spiders, Crawlers and Bots Oh My!
Bots, Spiders, Crawlers are all nick-names for the little programs that are constantly scouring the Internet indexing every file they find. You can see which Bots have visited your Web site in your site’s stats.
For example, Googlebot is Google’s little indexing program. These Bots are how search engines become aware of your site and new additions/pages so you can be listed accordingly. Understanding how they work is pretty important to every site owner because you can control what they index.
If you don’t want to let these critters in to index certain areas of your site, (images, member or private directories for example) it is important that you set up a robots.txt file. These little algorithms, at least the well-behaving ones, will look for that file before they proceed in case you want to designate areas on your site/server that you do not want indexed.
Well-behaving ones? Yep, there are nasty little devils that ignore your robots.txt file requests. Shame on them!
Read up and understand what Webcrawlers, Webrobots, Bots, Crawlers and Spiders are all about @ http://en.wikipedia.org/wiki/Web_crawler. Then, use that information to create your robots.txt file and control what information on your site is indexed.
At your service,
Judith
You may also enjoy:
- Don’t 4Get Your 404
- Avoiding E-mail Security Issues
- SEO Bullseye Marketing
- WordPress How To: Tired of Comment Spam?
- Where Were You 10 Years Ago?
Tell Me What You Think ...
FYI: I do not approve comments from those who use keywords in lieu of their names. Not only do those comments tend to be self-serving, I prefer to make a real connection with visitors who comment on my Blog -- can't do that with keywords. Please see my comment policy for more ...
