Hi all,
It took some time and research and as you all know I am a bit busy ( just a bit) ready to launch B4Playing beta software the day after tomorrow ( Aug 15th).
I will post here 4 different posts as the answer is pretty long. I do hope it will help us all to protect our content. Each post is related to each question posted by Dom (Thanks DOM)
First one:
Ok, so here’s the info I promised (thanks, Dominique, for breaking it down into the topics you want to know the most).
“How to find out if your site is being scraped and what to do if your site is being scraped.”
Right, so web scraping is what happens when someone comes along and takes content from your website without your permission. This started out just being text, but it now includes just about everything – images, graphics, video, and other elements. They are either left as they are or converted into another format or they may be utilized within a different context.
To be sure that your own website doesn’t fall victim to web scraping, there are many different things that you can do. Some are easier than others, and they all have their own good sides and bad sides.
To implement an effective anti-scraping strategy, you first need to understand how scrapers actually work. It’s actually quite surprising how simplistic scraping really is. The reason is that the more complex the tactics are, the less effectively they work. Their goal is reliability, not completeness or accuracy. They simply want to obtain content in a way that will make their visitors believe that they have obtained it by legitimate means.
Think about how easy it is to actually scrape a website. Have you ever highlighted text and pasted it into a document file so that you could use it later? Have you ever saved an image to your hard drive? That’s pretty much a form of scraping – or at least that’s how the easiest part of it is done. The difference, though, is that web scrapers will generally parse the HTML supplied by the server to take out the text and images. They use bots to do the “copy and paste” for them. They then use the material for their own sites, within different HTML coding.
This can have negative effects on your search engine ranking, since your content is no longer unique on the web, and search engines are looking for relevant, original content when they dole out the best ranks.
For one thing, you should accept that no matter what you do, it is possible for a determined web scraper to use your web content. So you may begin by ensuring that your content will work for you – whether it’s scraped or not. Essentially, this means that you should work to ensure that you are credited with your own work whether it’s being read on your website or if it’s been scraped and fed onto another site altogether.
Techniques for doing this include:
– Use your name in at the beginning of all of your posts, articles, and/or blogs.
– Hyperlink your name to your “about us” page when you use it at the beginning of your posts, articles, and/or blogs.
– Use absolute URLs.
– Include a link in the first two lines of your posts, articles, and/or blogs. Have the link lead to another page on your site.
– Include the name of your site or blog within the text of your posts, articles, and/or blogs.
– Include a paragraph at the end of your posts, articles, and/or blogs that will lead to “related information” or “other works by the author”, which include links to other places on your website.
– Be sure to quote or reference yourself or your own works within your posts, articles, and/or blogs.
This is extremely effective for ensuring that your readers recognize that the scraper is not the author. Even better, you will be providing the reader with many different pathways to find their way back to your website – where they belong. It may make your web content a bit more challenging to create, but it’s well worth the effort.
There are also some technical steps that you can take to stop – or at least slow down – the bots that scrapers use to swipe your content. Try the following:
– Use a “honeypot” (an easy-to-use technique for capturing limited information such as IP addresses of those visiting the site) or another technique to find out what IP address is sending the bots.
– If you find out that there’s an unwanted bot visiting your site, check out its IP address and block it. All browsing and bots will be blocked from then on from that address.
– Some bots – such as Google’s crawler, “googlebot” – declare themselves when they crawl a site. Therefore, you can block them on this basis. Sorry to say, though, that malicious bots will frequently say that they’re simply normal browsers, so they may not be screened by your efforts to identify them by name.
– You can block bots using excess traffic monitoring.
– You can block bots on your site by using excess traffic monitoring.
– Use techniques such as those created by the CAPCHA project (such as those wavy images of random letters that visitors must enter to progress, whichare used by some websites to prove that you’re a human visitor and not a bot).
– Use carefully crafted JavaScript.
itay