By checking your web site logs of who is coming and going you will see a number of bots and spiders referencing your pages and indexing for their masters most notably search engines, this happens all the time and at various times of day and night.
You might not want some of your pages made public – here is where you can create a robots.txt file which the bots and spiders are supposed to adhere to. The file protocol follows an industry standard and you can allow/disallow certain search engines to crawl and also protect certain parts of the URL from being indexed. Read the rest of this entry »
