TheGooner wrote:
yes – all URLs get spidered if they have a connection to the web – or are listed in a sitemap.
There are a couple other things that affect spidering. The robots.txt file issues instructions to spiders about which pages are disallowed, how intensive they should be, etc. See http://www.robotstxt.org. Also the robots META tag once the spider accesses a page, which tells spiders whether to index the page and follow links from the page.