Unknown robot (identified by ‘crawl’) 341+40 4.33 MB 17 Jan 2005 – 07:27
Unknown robot (identified by hit on ‘robots.txt’) 0+338 8.51 KB 17 Jan 2005 –
Unknown robot (identified by ‘robot’) 126+2 1.37 MB 10 Jan 2005 – 11:56
Unknown robot (identified by ‘spider’) 41+5 877.11 KB 16 Jan 2005 – 18:42
[/quote]These are the ones I wonder about. I get all of these and more – One, Unknown robot (identified by ‘crawl’) has incredible spidering activity on my site: 811+97 14.30 MB, almost as much as the big search engines . Wonder what they are?
Could they be from SEO pages, where people are searching your site, looking for clues to success, examining backlinks?
Are they legitimate search engine robots?
I find them worrying because they don’t identify themselves.