I used to actually run a spider identification site eons ago, and the type and variety out there are pretty amazing. Just a few of the tasks these things could be up to (in no particular order):
Crawling for inclusion in search engines
Email harvesting (for spam, link swap offers, semi-spam targeted offers, etc)
Image/file crawling/grabbing
Content harvesting (wholesale as well as sentence by sentence)
Link functionality checking (usually done as advertising for the company offering the service)
Link swap verification
Pure log entry advertising
“Archiving”
Form filling
And on and on…
Most legit bots will respect your robots.txt file, but the bad bots aren’t likely to be polite and follow the rules. Those at least posing as legit will include a URL in their user_agent with info about their crawler, but it may be bogus. Some are just trying to get you visit that URL…
Your best bet for blocking specific bots is cloaking/IP or agent delivery where you have a specific IP or user_agent. If you can identify them on the way in, you can deliver whatever you want.
In most cases that’s a bit extreme, but we should all, at a minimum, avoid mailto tags, @ symbols and anything else that might suggest an email address.
Sorry I can’t offer more…