Here is the real problem with blocking bots….
Not all bots (and especially scrapers) will say ‘hey, I am ScraperBot!’. Just like Google has many bots that do not say ‘hey, I am googlebot!’
So – without that information, you are back to blocking IP’s. However, there are so many proxies out there (have a look at the major site-type which sells at Sitepoint Marketplace – they are all proxies, directories, and myspace sites).
SO – You can look at your logs and block a proxy IP. That is all well and good, but if it is popular proxy, people may be using it from work so that they can check out your gambling site. Would you want to block them?
Sorry to kill the illusions out there, but this is fact. The only way to defend, IMO, is to watch Copyscape and either hire someone to bring you site names or spend all your time tracking them down instead of updating content.
In either case…. I think we lose.
It’s a tough nut to crack and it is going to take the help of the AFFILIATE PROGRAMS in order to fix the problem. They are the source of the income. If there is no income – there is no scraping in this industry.
Long enough for ya? :blahblah: