Zach Beane wrote:
On Sun, Jun 29, 2008 at 04:31:47PM -0700, Jeff Cunningham wrote:
Has anyone else noticed excessive visitation of pages served dynamically by Hunchentoot? On a couple of small sites virtually every page gets hit every five minutes by http://www.google.com/bot.html, regular as clockwork.
Normally, I setup robots.txt to allow google to walk these sites but unless I can think of a way to tell them to back off to once a week or something I'm about to Disallow them. They're just little private sites and Google ends up being 99% of the traffic. It's silly.
FWIW, I haven't had any trouble like that with two different Hunchentoot sites.
Is there something you do to indicate that the page has a fixed date? I was wondering if maybe the fact that dynamically generated pages look "changed" to Google, so they get thrown back in the queue to be rechecked.