Simoneaton wrote:
This has happened to me.
From my understanding these guys use BOTS and there is a certain comment you can place in your header tag to prevent BOTS crawling your site and scraping content.
That’s only useful if the bots respect robots.txt exclusion protocol. Typically, a malicious bot won’t.
Simoneaton wrote:
I will ask around so this can be prevented in the future. BOTS will continue to exist and crawl sites and the only way we can prevent these affiliates stealing from us is taking action by changing the way we design our website(s).
Really… designing the website differently won’t help. You can ban bots server-side by user agent, but that doesn’t help much since anyone capable of builiding a bot can surf with any user agent they wish.
You can also ban IP addresses server-side as well, but there are easy ways around that too.
The link posted is a really good one, and will stop a certain amount of silly stuff. There’s a large amount of still stuff that will continue.