The robots.txt file is not a big deal. It is simply a file that you use to tell search engine spiders what they can and can’t do in your site.
Having a blank robots.txt file means you are telling robots that they can crawl your entire site, and index everything.
There are 3 parts to a robots TXT file:
1 – The User-agent: This specifies to which robot you are giving which command.
2- Disallow: This tells the robot(s) which pages or directories you do not want crawled.
3- Comments: Self explanatory, they are marked by a # at the beginning of the line
Remember that a * is considered a wild card, so if you place User-agent: *, this is referring to all robots.
3 working examples:
On site.com you want all robots to index everything except your image folder, so that would be:
# you can do anything but don’t you step on my blue image folder
User-agent: *
Disallow: /images
On site.com you want all robots to index everything except your image folder, but pepebot can index your image folder:
User-agent: *
Disallow: /images
# Pepe bot rules so I will let him take my pics
User-agent: pepebot
Disallow:
On site.com you don’t want any spiders on any part of your site:
# the world sucks and I want to be a hermit
User-agent: *
Disallow: /
Remember that there are rogue spiders out there that won’t listen to your robots file, and they will do whatever they want. To protect yourself you need to use your htaccess file on your apache server. I am not quite sure how to protect yourself with a windows server, but I vaguely remember hearing that with ISS you can achieve similar restrictions.
Note: the comments were intended as comedy, you can erase/change those on your file
Best Regards,
Nicolas Johnson
GamblingWages Assistant Affiliate Manager
MSN messenger nicolas_1875 at hotmail dot com