Slurp crawling but not indexing

Its just a theory at this point, but I believe most if not all relevant pages are being targeted and in short order banned for duplicate content.

See:

http://www.casinoaffiliateprograms.com/bb/showthread.php?t=5378

This could explain why undesirable terms are still easily obtained.

How can you tell,by looking at your stats, if your site has been visited by a slurp? Is there a distinctive ip address that I should be looking for?

Awstats lists all the robots like this:

Inktomi Slurp 933+543 7.55 MB 15 Mar 2005 – 08:04

The name, how often has it crawled this month, how often did it hit robots.txt files, what bandwidth does it use, when was the last visit.

Awstats does not count robots as visitors. Some stats do that and it messes things up.

Awstats is one of the stats programs I use and I find it quite useful.

Awstats does not count robots as visitors. Some stats do that and it messes things up.

Very true. That’s why I have very different figures with awstats and webalizer. But it’s a matter of configuration. It’s possible to configure webalizer to not count these robots as visitors.

Good Morning,

I must be missing something with Webalizer.

Some people actually seem to find it useful.

I have always found it to be a completely useless piece of crap!

It gives me a bunch of words and terms that were used to find my site, but no indication as to whether those terms were entered at MSN, Google, Yahoo or wherever.

It’s frustrating, as most hosting plans offer cpanel, which comes with Webalizer pre-installed. .. :rasberry:

AWStats are a bit better .. but still not terribly comprehensive.

I recently spent an entire week searching for hosting plans offering decent stats.

Apollo offers Urchin for a one time installation fee of $35.

I guess you get what you pay for .. Webalizer is free, so most don’t complain.

Can anyone enlighten me as to what Webalizer may be good for?

Hey Professor, thanks for the reply! Isn’t that issue affecting google only? AFAIK it is.

Gravity, as Professor suggested above, you can use awstats or you can simply look at the raw access logs and search for the string “slurp!” which slurp uses as it’s useragent it hits your site.

Hope this helps! ” title=”” class=”bbcode_smiley” />

i’m using awstats…came with the hosting plan…and some of the robots are taken out of the stats, but others are not…inktomi slurp is taken out (they last visited this morning at 7:14)…msnbot and become.com visit often and are recorded as unique users, which is anoying…there are several others that aren’t as easy to figure out, but they are reported showing an equal number of page views as hits, which would be next to impossible for a real person to achieve…for instance i had dotnet3.orcsweb.com visit and go to 2 pages but only register 2 hits, so probably some sort of robot…

I use Webalizer and like it as well, although it took a few changes to the default configuration to make it behave. In particular, the defaults have it counting internal pages as referrers, which is pretty useless. A one-line change in the config fixes that. (I can look it up if anyone needs it.)

Dominique, Webalizer relies on the same weblogs as all the other webstats software you mention. It can’t give you referrer information on type-in traffic, unless the visitor is using a browser that transmits the current webpage as a referrer even if the user typed in a new address rather than clicking a link. I don’t know which browsers do that, but I know some have in the past.

If that information is present in your server logs, any of the noted stats software can access it.

I would think most current browsers would definitely not do that, as it’s potentially a security issue. Imagine if you’re visiting some site with a password and username in the URL, then type in another website. The webmaster on the second site could grab your username and password for the first site from his web logs.

It positively IS effecting Yahoo and conversely MSN