pokermonger wrote:
Thanks Dominique. I guess I just couldn’t decipher that it didn’t allow robots.
There is no indication here that robots.txt is blocking spiders to that section of the website.
Only certain directories, such as cgi-bin, are blocked. These aren’t directories you’d want spidered anyway.
The links page is listed in google, altavista, msn, etc.. thus it is going to be spidered.