- This topic is empty.
-
Posts
-
Interesting question.
I get a ton of spiders. In the following list, are all the “unknown” ones bad? Or could they be smaller foreign search engines too?
Robots/Spiders visitors Hits Bandwidth Last visit
Inktomi Slurp 718+466 6.08 MB 17 Jan 2005 – 11:03
Googlebot (Google) 416+50 4.68 MB 17 Jan 2005 – 07:05
Unknown robot (identified by ‘crawl’) 341+40 4.33 MB 17 Jan 2005 – 07:27
Unknown robot (identified by hit on ‘robots.txt’) 0+338 8.51 KB 17 Jan 2005 – 09:55
Jeeves 182+30 2.25 MB 17 Jan 2005 – 03:42
Unknown robot (identified by ‘robot’) 126+2 1.37 MB 10 Jan 2005 – 11:56
WISENutbot (Looksmart) 118+4 1.71 MB 17 Jan 2005 – 10:59
Alexa (IA Archiver) 77+41 2.31 MB 16 Jan 2005 – 19:05
Unknown robot (identified by ‘spider’) 41+5 877.11 KB 16 Jan 2005 – 18:42
Walhello appie 14+15 107.47 KB 16 Jan 2005 – 03:48Unknown robot (identified by ‘crawl’) 341+40 4.33 MB 17 Jan 2005 – 07:27
Unknown robot (identified by hit on ‘robots.txt’) 0+338 8.51 KB 17 Jan 2005 –
Unknown robot (identified by ‘robot’) 126+2 1.37 MB 10 Jan 2005 – 11:56
Unknown robot (identified by ‘spider’) 41+5 877.11 KB 16 Jan 2005 – 18:42
[/quote]These are the ones I wonder about. I get all of these and more – One, Unknown robot (identified by ‘crawl’) has incredible spidering activity on my site: 811+97 14.30 MB, almost as much as the big search engines . Wonder what they are?
Could they be from SEO pages, where people are searching your site, looking for clues to success, examining backlinks?
Are they legitimate search engine robots?
I find them worrying because they don’t identify themselves.
I used to actually run a spider identification site eons ago, and the type and variety out there are pretty amazing. Just a few of the tasks these things could be up to (in no particular order):
Crawling for inclusion in search engines
Email harvesting (for spam, link swap offers, semi-spam targeted offers, etc)
Image/file crawling/grabbing
Content harvesting (wholesale as well as sentence by sentence)
Link functionality checking (usually done as advertising for the company offering the service)
Link swap verification
Pure log entry advertising
“Archiving”
Form filling
And on and on…
Most legit bots will respect your robots.txt file, but the bad bots aren’t likely to be polite and follow the rules. Those at least posing as legit will include a URL in their user_agent with info about their crawler, but it may be bogus. Some are just trying to get you visit that URL…
Your best bet for blocking specific bots is cloaking/IP or agent delivery where you have a specific IP or user_agent. If you can identify them on the way in, you can deliver whatever you want.
In most cases that’s a bit extreme, but we should all, at a minimum, avoid mailto tags, @ symbols and anything else that might suggest an email address.
Sorry I can’t offer more…
