I am an idiot when it comes to this technical mumbo jumbo, but maybe someone can break this down for me. Since mostly all of everyones traffic comes from google, msn and yahoo would it be possible to make your site just allow the spiders of these major search engines while disallowing all other bs spiders. My understanding is most of this scrapping occurs when blackhats send out automated spiders to grab our content, if we can stop those spiders from entering our site then it solves a large part of the problem.
Isn’t there a way on the robots.txt or meta tags robots file to allow only your selected spiders/robots to enter your site while disallowing all others. There are a lot of smart people out there and it would seem to me that someone could come up with a way to just enable the 20-30 spiders or whatever the number is you want entering your site and blocking all others. I understand this will only stop the automated scrapping, but I think that is at least half the battle. My point is there are only a handful of major search engines. All the rest are pretty much bs so any spider coming to your site other than those from the major search engines don’t help much and are potentially the scrapers spiders. I get the feeling when writing this that I truly don’t have a clue what I am talking about lol.