islandmaan wrote:
just one more point of clarification, just because a site implements php that does not disallow the spiders from spidering does it? I see lots and lots of sites use php.
Brian
You are correct.
The robots.txt file tells search engine bots where they can and cannot index content. Your choice of web technology (PHP, ASP.NET, etc.) has nothing to do with this.