I've cleaned up a couple small problems using page validators but the net result has been that the Googlebot has gone from visiting my site every minute or so to every second or so. No kidding.
Before I block them altogether, there is one thing I don't understand that I'm hoping someone can explain to me. What does it mean exactly when I get a "No session for session identifier" INFO message in my error_log? There is one of these for each of the Googlebot hits.
--Jeff
Here is a snippet of the access_log (followed by the corresponding error_log snippet):
127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:32] "GET /zippy.html?hunchentoot-session=482%3A1424DD49911CD85 E35CE168E602686DC HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot .html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:32] "GET /robogames-2008.html?hunchentoot-session=483%3AFBDE26 FEB3370A5DC63A565E2611DD5A HTTP/1.1" 200 12724 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.goog le.com/bot.html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:33] "GET /robogames-2008.html?hunchentoot-session=482%3A1424DD 49911CD85E35CE168E602686DC HTTP/1.1" 200 12724 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.goog le.com/bot.html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:34] "GET /zippy.html?hunchentoot-session=485%3A97280661B72EBFC 07C48B8F100FCB60C HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot .html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:34] "GET /robogames-2008.html?hunchentoot-session=456%3A211AA2 04AD4D3A3703E0FA80DF3D0D51 HTTP/1.1" 200 12724 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.goog le.com/bot.html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:35] "GET /zippy.html?hunchentoot-session=289%3A8676A2453C327D6 10DDB4F79A48A61F6 HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot .html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:36] "GET /zippy.html?hunchentoot-session=177%3A6EF66F58D2E0A59 19AACB700DDB5D435 HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot .html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:39:37] "GET /robogames-2008.html?hunchentoot-session=464%3AC5740E 8F90480BCF319EFCD9C7A8EE88 HTTP/1.1" 200 12724 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.goog le.com/bot.html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:42:39] "GET /zippy.html?hunchentoot-session=483%3AFBDE26FEB3370A5 DC63A565E2611DD5A HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot .html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:46:28] "GET /zippy.html?hunchentoot-session=478%3AB00C2E319A15E56 58D72E232C36F30DD HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot .html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:50:18] "GET /zippy.html?hunchentoot-session=230%3A85588DC64DDDBF9 219B71F4DFA746B50 HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot .html)" 127.0.0.1 (66.249.66.195) - [2008-07-04 07:54:08] "GET /zippy.html?hunchentoot-session=145%3A4C7B146A126FC14 ACFC7BFA8100C07D8 HTTP/1.1" 200 4284 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot .html)"
HERE'S THE CORRESPONDING ERROR LOG ENTRIES:
[2008-07-04 07:39:32 [INFO]] No session for session identifier '483:FBDE26FEB3370A5DC63A565E2611DD5A' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:39:33 [INFO]] No session for session identifier '482:1424DD49911CD85E35CE168E602686DC' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:39:34 [INFO]] No session for session identifier '485:97280661B72EBFC07C48B8F100FCB60C' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:39:34 [INFO]] No session for session identifier '456:211AA204AD4D3A3703E0FA80DF3D0D51' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:39:35 [INFO]] No session for session identifier '289:8676A2453C327D610DDB4F79A48A61F6' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:39:36 [INFO]] No session for session identifier '177:6EF66F58D2E0A5919AACB700DDB5D435' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:39:37 [INFO]] No session for session identifier '464:C5740E8F90480BCF319EFCD9C7A8EE88' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:39:57 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {133B61D9}>. [2008-07-04 07:42:38 [INFO]] No session for session identifier '483:FBDE26FEB3370A5DC63A565E2611DD5A' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:42:59 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {13C504E1}>. [2008-07-04 07:46:28 [INFO]] No session for session identifier '478:B00C2E319A15E5658D72E232C36F30DD' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:46:48 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {13C740D1}>. [2008-07-04 07:50:18 [INFO]] No session for session identifier '230:85588DC64DDDBF9219B71F4DFA746B50' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:50:38 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {13C50F59}>. [2008-07-04 07:54:08 [INFO]] No session for session identifier '145:4C7B146A126FC14ACFC7BFA8100C07D8' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:54:28 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {13C84E89}>. [2008-07-04 07:57:59 [INFO]] No session for session identifier '478:B00C2E319A15E5658D72E232C36F30DD' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 07:58:19 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {13C5BF61}>. [2008-07-04 08:01:49 [INFO]] No session for session identifier '493:640F2D5F184258D62527F8BD4AA0B91B' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 08:02:09 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {13C735C9}>. [2008-07-04 08:05:40 [INFO]] No session for session identifier '308:FA7CD1A4C4592F69DA33BE6826383D46' (User- Agent: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', IP: '127.0.0.1') [2008-07-04 08:06:00 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {13C624A9}>.