seo and wordpress stuff for all & sundry
You are currently browsing articles tagged Robots.
Crawlers, bots and spiders are those critters that come along when you least expect or you can’t wait ’til they show up uninvited. Once they come and visit and check out your goods they will make a note and come back.
You can identify the bots by looking in your web logs and will discover that the big 3 are:
Google – Googlebot
Yahoo – Yahoo Slurp
Bing – MSNbot (wondering if they’ll change that soon, like Bingbot)
Tags: bandwidth, Bots, Googlebot, indexed, Robots, Spiders, voyager
By checking your web site logs of who is coming and going you will see a number of bots and spiders referencing your pages and indexing for their masters most notably search engines, this happens all the time and at various times of day and night.
You might not want some of your pages made public – here is where you can create a robots.txt file which the bots and spiders are supposed to adhere to. The file protocol follows an industry standard and you can allow/disallow certain search engines to crawl and also protect certain parts of the URL from being indexed. Read the rest of this entry »
