Hi!
I'm currently having some trouble with a long-running TBNL/mod_lisp application on a public website which gets about 500,000 requests per month. It's an application delivered with LispWorks professional 4.4.6 and it runs on Linux behind Apache 2.0.54.
From time to time I see errors in the server log file and sometimes
(very rarely) users also have reported errors. It turns out they're all of the "error reading from Lisp" kind. I hacked mod_lisp2.c a bit to make this more specific and I now get one of "error reading from Lisp (fill status)" or "error reading header name." This means that the error happens either when the reply entity is copied from Lisp to the client or when mod_lisp tries to read a header name from Lisp.
I've attached a couple of these error message to the end of this email. What's suspicious for me is that they almost always come in chunks - half a dozen of them or more in a couple of seconds, then several hours without errors.
I'm calling MARK-AND-SWEEP from time to time but at intervals that don't match the time of the error messages. (My first thought was that a full GC blocks LW long enough for mod_lisp to time out but that doesn't seem to be the case.) Other than that I only use locks in admin parts of the website which are clearly not used in the middle of the night when some of these errors happened.
I'm lost. Does anyone have an idea where I should look to find the cause for these problems?
Thanks, Edi.
[Fri Jan 06 00:50:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:09 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:10 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:13 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:14 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:15 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:17 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:34 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:36 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:37 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:38 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:40 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:41 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:43 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:44 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:45 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:46 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:00 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:01 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:08 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:09 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:10 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:12 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 18:07:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Fri Jan 06 21:53:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 04:31:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 05:03:53 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 06:04:10 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 07:05:02 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 07:56:04 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 07:56:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 07:56:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 10:03:19 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 10:28:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:22:53 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:23:57 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:37:46 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:48:43 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:52:56 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:03:12 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:24:57 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:32:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:04:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:10:00 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:11:04 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:12:08 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:13:14 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:14:19 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:15:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:58:12 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:25:11 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:29:14 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:30:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:31:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:46:11 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:49:50 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:53:46 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 21:05:03 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:04 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:12 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:14 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:15 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:19 2006] [error] (70007)The timeout specified has expired: error reading header name