- This topic is empty.
-
Posts
-
One thing you should include is an exclusion to the images directory
The robots.txt file is not a big deal. It is simply a file that you use to tell search engine spiders what they can and can’t do in your site.
Having a blank robots.txt file means you are telling robots that they can crawl your entire site, and index everything.
There are 3 parts to a robots TXT file:
1 – The User-agent: This specifies to which robot you are giving which command.
2- Disallow: This tells the robot(s) which pages or directories you do not want crawled.
3- Comments: Self explanatory, they are marked by a # at the beginning of the line
Remember that a * is considered a wild card, so if you place User-agent: *, this is referring to all robots.3 working examples:
On site.com you want all robots to index everything except your image folder, so that would be:
# you can do anything but don’t you step on my blue image folder
User-agent: *
Disallow: /imagesOn site.com you want all robots to index everything except your image folder, but pepebot can index your image folder:
User-agent: *
Disallow: /images# Pepe bot rules so I will let him take my pics
User-agent: pepebot
Disallow:On site.com you don’t want any spiders on any part of your site:
# the world sucks and I want to be a hermit
User-agent: *
Disallow: /Remember that there are rogue spiders out there that won’t listen to your robots file, and they will do whatever they want. To protect yourself you need to use your htaccess file on your apache server. I am not quite sure how to protect yourself with a windows server, but I vaguely remember hearing that with ISS you can achieve similar restrictions.
Note: the comments were intended as comedy, you can erase/change those on your file
Best Regards,
Nicolas Johnson
GamblingWages Assistant Affiliate Manager
MSN messenger nicolas_1875 at hotmail dot comThank you Nicolas for posting this very useful information.
I know that many people had some doubts/problems with what to do about the .htaccess – including me
Never even thought of excluding the images folders – plus what about navigation and redirect folders? In fact, probably all folders, since all we’d want accessed would be the actual pages.
ntaus
You can use the htaccess to protect yourself against referral spam or bad bots. You can also google bad bot traps to protect against bad spiders but then you might have to expressly allow certain agents like google, msn, yahoo etc. in your robots.txt.
Another very handy function of the htaccess is hotlink protection. For good tutorials and an intro to the htaccess google ilovejackdaniels . Gotta love that site
Thanks Nicolas ,
