I am having a problem that I can't replicate on demand but has been happening with increasing frequency. I am hoping some of you might suggest ways to troubleshoot it. I don't know that it is a Hunchentoot problem, per se, but it may be an interaction problem between Hunchentoot and Apache2 via mod_lisp2.
The symptom is the server is hung first thing in the morning when I check it. The cpu is at 99% activity on the server image (sbcl). When I look in the error_log I see dozens of these:
[Wed Feb 20 05:10:10 2008] [error] (70007)The timeout specified has expired: error reading from Lisp [Wed Feb 20 05:12:23 2008] [error] (70007)The timeout specified has expired: error reading from Lisp [Wed Feb 20 05:15:14 2008] [error] (70007)The timeout specified has expired: error reading from Lisp [Wed Feb 20 05:18:57 2008] [error] (70007)The timeout specified has expired: error reading from Lisp
These generally correspond to googlebot activity:
66.249.73.165 - - [20/Feb/2008:05:09:10 -0800] "GET /intrepid-obstacle-map.html?hunchentoot-session=18%3A15F2776FDB3BD7CA852EBF74A3B40B2B HTTP/1.1" 500 673 "-" "Mozilla/5.0 (c ompatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.165 - - [20/Feb/2008:05:11:23 -0800] "GET /index.html?hunchentoot-session=17%3A549FDE65186DCCFD1D2552AA79A49871 HTTP/1.1" 500 673 "-" "Mozilla/5.0 (compatible; Googl ebot/2.1; +http://www.google.com/bot.html)" 66.249.73.165 - - [20/Feb/2008:05:14:14 -0800] "GET /intrepid-obstacle-map.html?hunchentoot-session=17%3A549FDE65186DCCFD1D2552AA79A49871 HTTP/1.1" 500 673 "-" "Mozilla/5.0 (c ompatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.165 - - [20/Feb/2008:05:17:57 -0800] "GET /intrepid-robothon-2007.html?hunchentoot-session=15%3A3C06276A2A5CD0567B6A332AE52B3940 HTTP/1.1" 500 673 "-" "Mozilla/5.0 ( compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
And finally, netstat shows a whole bunch of 4242 ports open (these are just a small subset:)
tcp 680 0 127.0.0.1:4242 127.0.0.1:38121 CLOSE_WAIT tcp 517 0 127.0.0.1:4242 127.0.0.1:44273 CLOSE_WAIT tcp 0 0 127.0.0.1:42641 127.0.0.1:4242 ESTABLISHED tcp 0 0 127.0.0.1:4242 127.0.0.1:51379 ESTABLISHED tcp 509 0 127.0.0.1:4242 127.0.0.1:33727 CLOSE_WAIT tcp 515 0 127.0.0.1:4242 127.0.0.1:44279 CLOSE_WAIT tcp 0 0 127.0.0.1:4242 127.0.0.1:48386 ESTABLISHED tcp 561 0 127.0.0.1:4242 127.0.0.1:44283 CLOSE_WAIT tcp 0 0 127.0.0.1:4242 127.0.0.1:42646 ESTABLISHED tcp 557 0 127.0.0.1:4242 127.0.0.1:43923 CLOSE_WAIT tcp 0 0 192.168.1.102:39210 192.168.1.2:993 ESTABLISHED tcp 0 0 127.0.0.1:42651 127.0.0.1:4242 ESTABLISHED tcp 0 0 127.0.0.1:4242 127.0.0.1:51382 ESTABLISHED tcp 589 0 127.0.0.1:4242 127.0.0.1:48377 CLOSE_WAIT tcp 589 0 127.0.0.1:4242 127.0.0.1:56928 CLOSE_WAIT tcp 589 0 127.0.0.1:4242 127.0.0.1:54195 CLOSE_WAIT
The only way I can restart the server is to kill the process, restart it, and restart apache.
Any ideas?
Thanks. --Jeff
On Wed, Feb 20, 2008 at 08:09:18AM -0800, Jeff Cunningham wrote:
I am having a problem that I can't replicate on demand but has been happening with increasing frequency. I am hoping some of you might suggest ways to troubleshoot it. I don't know that it is a Hunchentoot problem, per se, but it may be an interaction problem between Hunchentoot and Apache2 via mod_lisp2.
The symptom is the server is hung first thing in the morning when I check it. The cpu is at 99% activity on the server image (sbcl). When I look in the error_log I see dozens of these:
[snip]
I had trouble like this in the past, and so far in every case it was due to me making a mistake in the handler and going into an infinite loop. I don't know if that's the case for you, but I had good luck debugging it by adding extra info to thread names so I could identify handlers that might be stuck.
http://xach.livejournal.com/132391.html has a writeup.
You can also troubleshoot by interrupting all threads and requesting their backtrace (best to write it to a file or save them to a table or something like that). That can also help identify problems.
Zach
Zach Beane wrote:
On Wed, Feb 20, 2008 at 08:09:18AM -0800, Jeff Cunningham wrote:
I am having a problem that I can't replicate on demand but has been happening with increasing frequency. I am hoping some of you might suggest ways to troubleshoot it. I don't know that it is a Hunchentoot problem, per se, but it may be an interaction problem between Hunchentoot and Apache2 via mod_lisp2.
The symptom is the server is hung first thing in the morning when I check it. The cpu is at 99% activity on the server image (sbcl). When I look in the error_log I see dozens of these:
[snip]
I had trouble like this in the past, and so far in every case it was due to me making a mistake in the handler and going into an infinite loop. I don't know if that's the case for you, but I had good luck debugging it by adding extra info to thread names so I could identify handlers that might be stuck.
http://xach.livejournal.com/132391.html has a writeup.
You can also troubleshoot by interrupting all threads and requesting their backtrace (best to write it to a file or save them to a table or something like that). That can also help identify problems.
Zach _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
Thanks, Zach. I just tried the following:
(mapcar (lambda (thread) (sb-thread:interrupt-thread thread (lambda () (throw 'hunchentoot::handler-done "")))) (sb-thread:list-all-threads))
And it says it can't find the tag 'hunchentoot::handler-done (I assume). Yet, it shows up in the list of package symbols, exported even:
attempt to THROW to a tag that does not exist: HANDLER-DONE [Condition of type SB-INT:SIMPLE-CONTROL-ERROR]
Restarts: 0: [ABORT] Abort SLIME compilation. 1: [ABORT] Return to SLIME's top level. 2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker" {D813B61}>)
Backtrace: 0: ("no debug information for frame") 1: ((LAMBDA NIL)) 2: (SB-THREAD:INTERRUPT-THREAD #<SB-THREAD:THREAD "worker" {D813B61}> #<FUNCTION (LAMBDA NIL) {D84CC55}>) 3: (SB-IMPL::MAP1 #<FUNCTION (LAMBDA (THREAD)) {D84CF95}> ((#<SB-THREAD:THREAD "worker" {D813B61}> #<SB-THREAD:THREAD "hunchentoot-worker-34" {D7CB2D1}> #<SB-THREAD:THREAD "hunchentoot-worker-33" {D7D92B9}> #<SB-THREAD:THREAD "hunchentoot-worker-32" {D777351}> #<SB-THREAD:THREAD "hunchentoot-worker-31" {DA38F11}> #<SB-THREAD:THREAD "hunchentoot-worker-30" {DC5A989}> ...)) :LIST T) 4: (MAPCAR #<FUNCTION (LAMBDA (THREAD)) {D84CF95}> (#<SB-THREAD:THREAD "worker" {D813B61}> #<SB-THREAD:THREAD "hunchentoot-worker-34" {D7CB2D1}> #<SB-THREAD:THREAD "hunchentoot-worker-33" {D7D92B9}> #<SB-THREAD:THREAD "hunchentoot-worker-32" {D777351}> #<SB-THREAD:THREAD "hunchentoot-worker-31" {DA38F11}> #<SB-THREAD:THREAD "hunchentoot-worker-30" {DC5A989}> ...))
--Jeff
Google will also let you rate limit the naughty GoogleBot using their Webmaster Tools thing.
Do individual requests do anything particularly computationally intensive? Is Hunchentoot just hanging doing nothing, or is it using the CPU? If SBCL, have you a moderately recent version? (Old ones had fun interesting nasty threading issues.) Rob
Robert Synnott wrote:
Google will also let you rate limit the naughty GoogleBot using their Webmaster Tools thing.
Do individual requests do anything particularly computationally intensive? Is Hunchentoot just hanging doing nothing, or is it using the CPU? If SBCL, have you a moderately recent version? (Old ones had fun interesting nasty threading issues.) Rob __________________
Yes, I know about the webmaster tools thing - it looks like they're hit rate is a page every twenty seconds or so when they are going through the site. It ought to be able to handle that.
Hunchentoot is hanging with the cpu hammered at 99-100%, like its in an infinite loop. And, no, nothing in the page generation should cause it to do more than blink. These pages are very simple text with a few embedded images.
My sbcl is pretty late model: 1.0.14 from sources. But I've noticed this same problem on a commercial site a couple times where I'm running 1.0.12.
--Jeff
Have you slime running in it? You could possibly attach and poke around a bit; not sure how useful it would be, though. Rob
On Wed, Feb 20, 2008 at 08:46:24AM -0800, Jeff Cunningham wrote:
And it says it can't find the tag 'hunchentoot::handler-done (I assume). Yet, it shows up in the list of package symbols, exported even:
attempt to THROW to a tag that does not exist: HANDLER-DONE [Condition of type SB-INT:SIMPLE-CONTROL-ERROR]
Right. That's because not all threads are handlers, so the non-handler threads will signal an error. That code snippet is not very precise, and I just ignored the errors I got.
Zach
Zach Beane wrote:
Right. That's because not all threads are handlers, so the non-handler threads will signal an error. That code snippet is not very precise, and I just ignored the errors I got.
Zach
Okay - it's doing it right now. 99-100% cpu showing up as sbcl on my webserver user (the only sbcl running under that user). When I list out the notated threads I see the following. I'm not sure what I was expecting to see, but nothing stands out to me as a problem. All of them say "[finished & idle]". Can you interpret?
--Jeff
(#<SB-THREAD:THREAD "repl-thread" {F030409}> #<SB-THREAD:THREAD "reader-thread" {F030339}> #<SB-THREAD:THREAD "control-thread" {F030259}> #<SB-THREAD:THREAD "auto-flush-thread" {F030099}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1136%3A6A13236755E6BFCCC3994C86FF75CE20" [finished & idle]" {DCD92A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1188%3A25D0F65BA3F9CB5F8848D24A394B5A87" [finished & idle]" {DB552D1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1143%3AE13A4EFDF0FA675CCF10FCD34B0C19EB" [finished & idle]" {DCF22A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-philosophy.html?hunchentoot-session=1188%3A25D0F65BA3F9CB5F8848D24A394B5A87" [finished & idle]" {DC4A4E1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1157%3A20E10CFFC8FE2FEDB91EAC6C7C554F8C" [finished & idle]" {DC632A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1148%3A306D0561E30ECFD4918E2DDABE1BBA8D" [finished & idle]" {DAD52A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1167%3A982468B6F34710B4AA6ACF16FE1B05DB" [finished & idle]" {DBD72A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1152%3A5DC6202211E106618E5EEF58F9D1AC91" [finished & idle]" {DAE62A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1151%3AAAAC22F60C7EAB687EA5C6F04355688B" [finished & idle]" {DC242A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1151%3AAAAC22F60C7EAB687EA5C6F04355688B" [finished & idle]" {DB1A2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1196%3A212178577168731D2454185B7D8F0554" [finished & idle]" {DAB52A1}> #<SB-THREAD:THREAD "Handler for "/slime.html" [finished & idle]" {DB2B2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1220%3AAF17981047A17996DFBD8F72406747EB" [finished & idle]" {DA2C2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1219%3ACDBC9E26292EDF35F26E792A23038D5D" [finished & idle]" {EFD02A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1184%3A7808D8B675B055C7CF210957A6037CE5" [finished & idle]" {DA3A2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1217%3ACEF20AE79B65C45111EEEA664ADAADD9" [finished & idle]" {E2E3681}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1189%3ABF3BCA4D06D4C89DD371E187D92D30B6" [finished & idle]" {D9032A1}> #<SB-THREAD:THREAD "Handler for "/ts7200.html" [finished & idle]" {D91E2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1188%3A25D0F65BA3F9CB5F8848D24A394B5A87" [finished & idle]" {D9322A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1117%3A135850BDF1BBA2BAEA4A1EAF54580B72" [finished & idle]" {DA0F2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1182%3AD6355AA121CEDE6E24B3A4586ABCFE50" [finished & idle]" {D8502A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1145%3A6950D17A5213E6C054F90C5F607631D5" [finished & idle]" {D8852A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1155%3A7B38C20E78E979DECDB64E500EC22FA2" [finished & idle]" {D8B32A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1143%3AE13A4EFDF0FA675CCF10FCD34B0C19EB" [finished & idle]" {D7652A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1149%3AF3A5B129D307ED34BF1ECD3EEDE8CE5B" [finished & idle]" {D9792D1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1156%3A6ECDBBD829964A74DB970C7FA9EA9697" [finished & idle]" {D7BB2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-philosophy.html?hunchentoot-session=1166%3A45D7F9FAB0B8C917904F0F553EB246F8" [finished & idle]" {D9582A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1141%3AD26E286D709B5826A51E461D94813B68" [finished & idle]" {D97B2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1128%3A1233C2ACD90A914E382DAD2557276432" [finished & idle]" {D7C02A1}> #<SB-THREAD:THREAD "Handler for "/" [finished & idle]" {D7882A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1153%3A5B822B1CFBCC621168187D094E3E2014" [finished & idle]" {D6DA2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1167%3A982468B6F34710B4AA6ACF16FE1B05DB" [finished & idle]" {D8C42A1}> #<SB-THREAD:THREAD "Handler for "/intrepid/Intrepid_001_th.jpg?hunchentoot-session=1209%3AD5CE6247382306CAF9115845A594C128" [finished & idle]" {D8EA2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1154%3A32344CE5830E371C7B0F192C597257DD" [finished & idle]" {D8102D1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1153%3A5B822B1CFBCC621168187D094E3E2014" [finished & idle]" {D8122A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1154%3A32344CE5830E371C7B0F192C597257DD" [finished & idle]" {D6192D1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1183%3AC7780577DC441B2BB56915A75313BFB3" [finished & idle]" {D56D2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1140%3AF6B64C2D8CCA9783DE0E190057497AE4" [finished & idle]" {D59C2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1194%3A163A0D0028182806A9CDD029D8C07601" [finished & idle]" {D5542A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1214%3AE9E6EEEF287C09954F0C55A2305FBA1E" [finished & idle]" {D6402A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1194%3A163A0D0028182806A9CDD029D8C07601" [finished & idle]" {D5F4719}> #<SB-THREAD:THREAD "Handler for "/ts7200.html" [finished & idle]" {D5F62A1}> #<SB-THREAD:THREAD "Handler for "/ts7200.html" [finished & idle]" {D61B2A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1218%3A339FB8475FBBCAC25B012C5E363635FB" [finished & idle]" {D5762A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1192%3A65574B358B655C0E7E3A0EA43680AED6" [finished & idle]" {D7342A1}> #<SB-THREAD:THREAD "Handler for "/ts7200.html" [finished & idle]" {D6B02A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1219%3ACDBC9E26292EDF35F26E792A23038D5D" [finished & idle]" {D6F32A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1220%3AAF17981047A17996DFBD8F72406747EB" [finished & idle]" {D5D62A1}> #<SB-THREAD:THREAD "Handler for "/intrepid-obstacle-map.html?hunchentoot-session=1202%3AC12DEAF5F5AAD8ECB6FBF9F1D31C1B8F" [finished & idle]" {F17B579}> #<SB-THREAD:THREAD "Handler for "/intrepid-robogames-2007.html?hunchentoot-session=1182%3AD6355AA121CEDE6E24B3A4586ABCFE50" [finished & idle]" {D699F11}> #<SB-THREAD:THREAD "Swank 4005" {D6936D1}> #<SB-THREAD:THREAD "hunchentoot-listener-1" {D5C3F61}> #<SB-THREAD:THREAD "initial thread" {D4C2089}>)
On Sat, Feb 23, 2008 at 06:12:26PM -0800, Jeff Cunningham wrote:
Zach Beane wrote:
Right. That's because not all threads are handlers, so the non-handler threads will signal an error. That code snippet is not very precise, and I just ignored the errors I got.
Zach
Okay - it's doing it right now. 99-100% cpu showing up as sbcl on my webserver user (the only sbcl running under that user). When I list out the notated threads I see the following. I'm not sure what I was expecting to see, but nothing stands out to me as a problem. All of them say "[finished & idle]". Can you interpret?
Nope, I got nothin'. The next thing I'd try, if I were you, is profiling all threads with sb-sprof to see where time is being spent.
Zach
On Wed, 20 Feb 2008 08:46:24 -0800, Jeff Cunningham jeffrey@cunningham.net wrote:
And it says it can't find the tag 'hunchentoot::handler-done (I assume). Yet, it shows up in the list of package symbols, exported even:
attempt to THROW to a tag that does not exist: HANDLER-DONE [Condition of type SB-INT:SIMPLE-CONTROL-ERROR]
The fact that the symbol is there doesn't mean that the catch tag has been established. It looks like you're using THROW (or a function which uses it like REDIRECT) from a place where you shouldn't.
Edi Weitz wrote:
On Wed, 20 Feb 2008 08:46:24 -0800, Jeff Cunningham jeffrey@cunningham.net wrote:
And it says it can't find the tag 'hunchentoot::handler-done (I assume). Yet, it shows up in the list of package symbols, exported even:
attempt to THROW to a tag that does not exist: HANDLER-DONE [Condition of type SB-INT:SIMPLE-CONTROL-ERROR]
The fact that the symbol is there doesn't mean that the catch tag has been established. It looks like you're using THROW (or a function which uses it like REDIRECT) from a place where you shouldn't.
That one went over my head. Does it mean that if I put a function like this:
(defun cancel-all-threads () (mapcar (lambda (thread) (sb-thread:interrupt-thread thread (lambda () (throw 'hunchentoot::handler-done "")))) (sb-thread:list-all-threads)))
into, say, request.lisp (in hunchentoot), that it would find the tag?
On Wed, 20 Feb 2008 09:16:58 -0800, Jeff Cunningham jeffrey@cunningham.net wrote:
That one went over my head. Does it mean that if I put a function like this:
(defun cancel-all-threads () (mapcar (lambda (thread) (sb-thread:interrupt-thread thread (lambda () (throw 'hunchentoot::handler-done "")))) (sb-thread:list-all-threads)))
into, say, request.lisp (in hunchentoot), that it would find the tag?
Why don't you try it out yourself?
CL-USER> (defun endless () (format #.*standard-output* "Interrupted by someone throwing ~S.~%" (catch 'foo (loop (sleep 1))))) ENDLESS CL-USER> (defun stop () (throw 'foo 42)) STOP CL-USER> (sb-thread:make-thread 'endless) #<SB-THREAD:THREAD {AEB0E11}> CL-USER> (sb-thread:interrupt-thread * 'stop) Interrupted by someone throwing 42.
But as I already said the catch tag has to be established for this to work. Look at the code, specifically at server.lisp, and you'll see that you can't simply throw HANDLER-DONE at any time during the lifetime of a Hunchentoot thread. Or simply read the docs where it says that you can throw this tag "during the lifetime of a handler."
Edi Weitz wrote:
That one went over my head. Does it mean that if I put a function like this:
(defun cancel-all-threads () (mapcar (lambda (thread) (sb-thread:interrupt-thread thread (lambda () (throw 'hunchentoot::handler-done "")))) (sb-thread:list-all-threads)))
into, say, request.lisp (in hunchentoot), that it would find the tag?
Why don't you try it out yourself?
CL-USER> (defun endless () (format #.*standard-output* "Interrupted by someone throwing ~S.~%" (catch 'foo (loop (sleep 1))))) ENDLESS CL-USER> (defun stop () (throw 'foo 42)) STOP CL-USER> (sb-thread:make-thread 'endless) #<SB-THREAD:THREAD {AEB0E11}> CL-USER> (sb-thread:interrupt-thread * 'stop) Interrupted by someone throwing 42.
But as I already said the catch tag has to be established for this to work. Look at the code, specifically at server.lisp, and you'll see that you can't simply throw HANDLER-DONE at any time during the lifetime of a Hunchentoot thread. Or simply read the docs where it says that you can throw this tag "during the lifetime of a handler."
I didn't try it out at that point because I didn't understand it well enough to formulate an intelligent test case. Thank you for providing one.
I have in place some machinery now that should enable me to identify all the threads in play and their state the next time it hangs as per Zach's suggestions, so I'll have to wait and see. Can't troubleshoot it until it breaks unfortunately.
Thanks for the ideas. --Jeff