It’s not pretty, but it works for me. I’ve only included the parts relevant to spider detection.
$is_spider = false;
if (strpos($agent,’googlebot’) !== false) { $is_spider = true; }
if (strpos($agent,’slurp’) !== false) { $is_spider = true; }
if (strpos($agent,’webcrawler’) !== false) { $is_spider = true; }
if (strpos($agent,’ia_archiver’) !== false) { $is_spider = true; }
if (strpos($agent,’spider’) !== false) { $is_spider = true; }
if (strpos($agent,’scooter’) !== false) { $is_spider = true; }
if (strpos($agent,’teoma_agent1′) !== false) { $is_spider = true; }
if (strpos($agent,’nameprotect’) !== false) { $is_spider = true; }
if (strpos($agent,’msnbot’) !== false) { $is_spider = true; }
if (strpos($agent,’turnitin’) !== false) { $is_spider = true; }
if (strpos($agent,’naverbot’) !== false) { $is_spider = true; }
if (strpos($agent,’pompos’) !== false) { $is_spider = true; }
if (strpos($agent,’nutch’) !== false) { $is_spider = true; }
if (strpos($agent,’gigabot’) !== false) { $is_spider = true; }
if (strpos($agent,’sohu-search’) !== false) { $is_spider = true; }
if (strpos($agent,’it-bot’) !== false) { $is_spider = true; }
if (strpos($agent,’becomebot’) !== false) { $is_spider = true; }
if (strpos($agent,’ng/2.0′) !== false) { $is_spider = true; }
if (strpos($agent,’webrobot’) !== false) { $is_spider = true; }
if (!$is_spider) { inc_clicks_check_ip($id, $_SERVER[‘REMOTE_ADDR’]); }
The inc_clicks_check_ip part is another routine that ensures that multiple consecutive clicks from the same IP address are only counted once.
Also, whenever I see an unusual number of clicks throughout my site, I usually go look in the logs to see what new bot is hitting me, and add it’s name to the list above.