Interesting question.
I get a ton of spiders. In the following list, are all the “unknown” ones bad? Or could they be smaller foreign search engines too?
Robots/Spiders visitors Hits Bandwidth Last visit
Inktomi Slurp 718+466 6.08 MB 17 Jan 2005 – 11:03
Googlebot (Google) 416+50 4.68 MB 17 Jan 2005 – 07:05
Unknown robot (identified by ‘crawl’) 341+40 4.33 MB 17 Jan 2005 – 07:27
Unknown robot (identified by hit on ‘robots.txt’) 0+338 8.51 KB 17 Jan 2005 – 09:55
Jeeves 182+30 2.25 MB 17 Jan 2005 – 03:42
Unknown robot (identified by ‘robot’) 126+2 1.37 MB 10 Jan 2005 – 11:56
WISENutbot (Looksmart) 118+4 1.71 MB 17 Jan 2005 – 10:59
Alexa (IA Archiver) 77+41 2.31 MB 16 Jan 2005 – 19:05
Unknown robot (identified by ‘spider’) 41+5 877.11 KB 16 Jan 2005 – 18:42
Walhello appie 14+15 107.47 KB 16 Jan 2005 – 03:48