Crawlers, bots and spiders are those critters that come along when you least expect or you can’t wait ’til they show up uninvited. Once they come and visit and check out your goods they will make a note and come back.
You can identify the bots by looking in your web logs and will discover that the big 3 are:
Google – Googlebot
Yahoo – Yahoo Slurp
Bing – MSNbot (wondering if they’ll change that soon, like Bingbot)
As well as the big threee you will see a bunch of other bots, some hide their identity and others are visible, couple of others that i have noticed are:
Relevant Noise – a social media mining site set up to see what others are gossiping about you
Voyager – not quite sure who the voyager bot is yet, there is not much info out there on this – maybe the next big thing.
- bots_spiders_crawlers_voyager
The pic above is a grab from AW stats, it gives you a list of bots and how much bandwith they suck up, some also have links back to their homes so you can see whose spidering on you.
One thing to note is that the bots will consume your all important bandwidth allowance, but you do really want them on your side, if you didn’t or wanted to ban certain ones you can get your robots on the defensive.
No related posts.
Tags: bandwidth, Bots, Googlebot, indexed, Robots, Spiders, voyager
