[hunchentoot-devel] Hunchentoot and threads
I seem to recall that, a long time ago, TBNL/hunchentoot required threads. At some point Edi rigged up a singled threaded version but claimed that that was just for development/testing work and shouldn't be used in the real world. Now I understand that Hans has an aversion to threads as the multiplexing abstraction for webservers and that single threaded is "the one true way." I bring this up as background to me real problem, which is that, at least on FreeBSD, I seem to have a number of zombie threads that stick around after requests are made. Before I dig into the problem, I'd be curious to hear what folks' recommendations for running a low-traffic, but nevertheless hopefully reliable, site are. I had reasonably good luck with SBCL+threads and hunchentoot before the big rewrite, but since then, it has been reliable refusing to accept new connections after a week or two of use. Everything starts out fine and I can field some traffic and all is good: CL-USER> (sb-thread:list-all-threads) (#<SB-THREAD:THREAD "reader-thread" RUNNING {59B6B241}> #<SB-THREAD:THREAD "repl-thread" RUNNING {59B6B0F1}> #<SB-THREAD:THREAD "control-thread" RUNNING {59B60359}> #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {59B601D9}> #<SB-THREAD:THREAD "initial thread" RUNNING {598D79B1}>) However, after some period, I see an accumulation of worker threads that won't die and eventually I'm unable to start a new thread to field the request. Who knows where the problem is... Could be in hunchentoot proper, in usocket, in SBCL's FreeBSD threads, etc... At the moment I don't have any debugging info on the problem, as all seems to be working ok since I restarted the web server a few minutes ago. The next time the problem crops up, I'll reply to this with some debugging information. This seems to be the one major "regression" from the old hunchentoot that has me considering going back to the stable release (although I've already ported my code forward for the new API, so I'd hate to have to do that). Better yet would be to either track down and fix the problem or to be convinced that this is just broken and is never going to work and figure out how to make my stuff work in a single threaded lisp. One more question regarding threads, is it possible to use a threaded lisp to act like the single threaded server does? That is to use threads, but to only have one thread doing the hard work, using serve- event for the multiplexing? My initialization code relies on setting some state after starting to listen for requests, so I can't just flip the switch to single-threaded mode and have everything work out of the box. Thanks for all of the hunchentoot rewrite efforts! On the whole it seems like a good thing, but fixing this issue would make me a much happier hunchentoot user. Cyrus
Hi Cyrus, when you see worker threads accumulate, do you also see that there is a large number of connections to Hunchentoot that are not being closed? Or are there just threads, but no connections? Are you running Hunchentoot behind a proxy/frontend, or standalone? How many dead workers did you see? NB: I do certainly think that it is possible to write threaded web servers that work, I just believe that it is hard, in many respects. Threads, in my personal opinion, are not the best hammer to solve the I/O multiplexing problem that needs to be solved in a web server. Certainly, the unbounded worker thread creation strategy that Hunchentoot uses is not suitable for servers that see load peaks, which is why I recommend not using threaded Hunchentoot for such sites. That said: I do have stability issues with my non-threaded Hunchentoot installation on FreeBSD, too. I use a multi threaded SBCL, but run Hunchentoot in a single thread behind squid. In some situations, no new connections are being accepted for no apparent reasons, but I failed to properly analyze the problem last time it happened as the customer was already nervous. I have seen this happen two times in the last four months, on a moderately busy site. Thus, the problem may actually not be related to Hunchentoot's threads usage (I'm running non-threaded, you run threaded), but could as well be located in SBCL's thread implementation of FreeBSD, in usocket or somewhere else. Any further data points would help getting down to the bottom of this. As for future work on Hunchentoot: We do have the new connection manager class in place which is meant to support the implementation of thread pools. Thread pools would help putting limits on the number of threads created, helping with getting through load peaks. I do not personally need such a connection manager, but rather want to spend some time on making Hunchentoot be able to use single threaded I/O multiplexing using select/kpoll/whatever. -Hans On Sun, Nov 16, 2008 at 18:03, Cyrus Harmon <ch-tbnl@bobobeach.com> wrote:
I seem to recall that, a long time ago, TBNL/hunchentoot required threads. At some point Edi rigged up a singled threaded version but claimed that that was just for development/testing work and shouldn't be used in the real world. Now I understand that Hans has an aversion to threads as the multiplexing abstraction for webservers and that single threaded is "the one true way." I bring this up as background to me real problem, which is that, at least on FreeBSD, I seem to have a number of zombie threads that stick around after requests are made. Before I dig into the problem, I'd be curious to hear what folks' recommendations for running a low-traffic, but nevertheless hopefully reliable, site are. I had reasonably good luck with SBCL+threads and hunchentoot before the big rewrite, but since then, it has been reliable refusing to accept new connections after a week or two of use.
Everything starts out fine and I can field some traffic and all is good:
CL-USER> (sb-thread:list-all-threads) (#<SB-THREAD:THREAD "reader-thread" RUNNING {59B6B241}> #<SB-THREAD:THREAD "repl-thread" RUNNING {59B6B0F1}> #<SB-THREAD:THREAD "control-thread" RUNNING {59B60359}> #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {59B601D9}> #<SB-THREAD:THREAD "initial thread" RUNNING {598D79B1}>)
However, after some period, I see an accumulation of worker threads that won't die and eventually I'm unable to start a new thread to field the request. Who knows where the problem is... Could be in hunchentoot proper, in usocket, in SBCL's FreeBSD threads, etc... At the moment I don't have any debugging info on the problem, as all seems to be working ok since I restarted the web server a few minutes ago. The next time the problem crops up, I'll reply to this with some debugging information. This seems to be the one major "regression" from the old hunchentoot that has me considering going back to the stable release (although I've already ported my code forward for the new API, so I'd hate to have to do that). Better yet would be to either track down and fix the problem or to be convinced that this is just broken and is never going to work and figure out how to make my stuff work in a single threaded lisp.
One more question regarding threads, is it possible to use a threaded lisp to act like the single threaded server does? That is to use threads, but to only have one thread doing the hard work, using serve- event for the multiplexing? My initialization code relies on setting some state after starting to listen for requests, so I can't just flip the switch to single-threaded mode and have everything work out of the box.
Thanks for all of the hunchentoot rewrite efforts! On the whole it seems like a good thing, but fixing this issue would make me a much happier hunchentoot user.
Cyrus
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
On 16 Nov 2008, at 19:08, Hans Hübner wrote:
Hi Cyrus,
when you see worker threads accumulate, do you also see that there is a large number of connections to Hunchentoot that are not being closed? Or are there just threads, but no connections? Are you running Hunchentoot behind a proxy/frontend, or standalone? How many dead workers did you see?
NB: I do certainly think that it is possible to write threaded web servers that work, I just believe that it is hard, in many respects. Threads, in my personal opinion, are not the best hammer to solve the I/O multiplexing problem that needs to be solved in a web server. Certainly, the unbounded worker thread creation strategy that Hunchentoot uses is not suitable for servers that see load peaks, which is why I recommend not using threaded Hunchentoot for such sites.
That said: I do have stability issues with my non-threaded Hunchentoot installation on FreeBSD, too. I use a multi threaded SBCL, but run Hunchentoot in a single thread behind squid. In some situations, no new connections are being accepted for no apparent reasons, but I failed to properly analyze the problem last time it happened as the customer was already nervous. I have seen this happen two times in the last four months, on a moderately busy site. Thus, the problem may actually not be related to Hunchentoot's threads usage (I'm running non-threaded, you run threaded), but could as well be located in SBCL's thread implementation of FreeBSD, in usocket or somewhere else.
Any further data points would help getting down to the bottom of this.
As for future work on Hunchentoot: We do have the new connection manager class in place which is meant to support the implementation of thread pools. Thread pools would help putting limits on the number of threads created, helping with getting through load peaks. I do not personally need such a connection manager, but rather want to spend some time on making Hunchentoot be able to use single threaded I/O multiplexing using select/kpoll/whatever.
-Hans
On this, I've had no trouble with multi-threaded Hunchentoot for a moderate traffic site (~10 requests per sec, generally a few threads working at a time), on Linux. Rob
On Sun, Nov 16, 2008 at 7:08 PM, Hans Hübner <hans@huebner.org> wrote:
As for future work on Hunchentoot: We do have the new connection manager class in place which is meant to support the implementation of thread pools. Thread pools would help putting limits on the number of threads created, helping with getting through load peaks. I do not personally need such a connection manager, but rather want to spend some time on making Hunchentoot be able to use single threaded I/O multiplexing using select/kpoll/whatever.
Is there an existing webserver which uses this technique? I'm interested in seeing the details.
On Mon, Nov 17, 2008 at 14:02, Andy Chambers <achambers.home@googlemail.com> wrote:
On Sun, Nov 16, 2008 at 7:08 PM, Hans Hübner <hans@huebner.org> wrote:
As for future work on Hunchentoot: We do have the new connection manager class in place which is meant to support the implementation of thread pools. Thread pools would help putting limits on the number of threads created, helping with getting through load peaks. I do not personally need such a connection manager, but rather want to spend some time on making Hunchentoot be able to use single threaded I/O multiplexing using select/kpoll/whatever.
Is there an existing webserver which uses this technique? I'm interested in seeing the details.
IOLIB (http://common-lisp.net/project/iolib/) has an event based HTTP server and is written in Common Lisp. Twisted (http://twistedmatrix.com/trac/) is an event based communications framework written in Python which includes a HTTP server. xlightweb (http://xlightweb.sourceforge.net/) is a HTTP client/server library written in an event oriented fashion in Java, based on the xsocket library. HTH. -Hans
On Mon, Nov 17, 2008 at 1:40 PM, Hans Hübner <hans@huebner.org> wrote:
On Mon, Nov 17, 2008 at 14:02, Andy Chambers <achambers.home@googlemail.com> wrote:
On Sun, Nov 16, 2008 at 7:08 PM, Hans Hübner <hans@huebner.org> wrote:
As for future work on Hunchentoot: We do have the new connection manager class in place which is meant to support the implementation of thread pools. Thread pools would help putting limits on the number of threads created, helping with getting through load peaks. I do not personally need such a connection manager, but rather want to spend some time on making Hunchentoot be able to use single threaded I/O multiplexing using select/kpoll/whatever.
Is there an existing webserver which uses this technique? I'm interested in seeing the details.
IOLIB (http://common-lisp.net/project/iolib/) has an event based HTTP server and is written in Common Lisp. Twisted (http://twistedmatrix.com/trac/) is an event based communications framework written in Python which includes a HTTP server. xlightweb (http://xlightweb.sourceforge.net/) is a HTTP client/server library written in an event oriented fashion in Java, based on the xsocket library.
Cool. I also found this discussion on the subject which provided yet more pointers. http://www.kegel.com/c10k.htm -- Andy
Hi Andy,
Is there an existing webserver which uses this technique? I'm interested in seeing the details.
Do you mean the thread pool tech or the select technique. AFAIK, aserve uses thread pools; I think nginx (http://www.nginx.net/) uses the select-style. -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM
On Nov 16, 2008, at 11:08 AM, Hans Hübner wrote:
Hi Cyrus,
when you see worker threads accumulate, do you also see that there is a large number of connections to Hunchentoot that are not being closed? Or are there just threads, but no connections? Are you running Hunchentoot behind a proxy/frontend, or standalone? How many dead workers did you see?
So, how do I see if the connections are still open? It's only two so far, but here's what it looks like: CL-USER> (sb-thread:list-all-threads) (#<SB-THREAD:THREAD "repl-thread" RUNNING {5CBD0411}> #<SB-THREAD:THREAD "reader-thread" RUNNING {5CBD02C1}> #<SB-THREAD:THREAD "control-thread" RUNNING {5CBD0169}> #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:35892)" RUNNING {5CDF8E91}> #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:34650)" RUNNING {5CD329A1}> #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> #<SB-THREAD:THREAD "Hunchentoot acceptor (*:443)" RUNNING {5A5C7019}> #<SB-THREAD:THREAD "Hunchentoot acceptor (*:80)" RUNNING {5A3D80B9}> #<SB-THREAD:THREAD "Swank 4005" RUNNING {5A2304A1}> #<SB-THREAD:THREAD "initial thread" RUNNING {598D7BE1}>) Here's a backtrace from thread 4: 0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA #) {5A4E49D5}>)[:EXTERNAL] 1: (BACKTRACE 536870911 #<SYNONYM-STREAM :SYMBOL *TERMINAL-IO* {5812FA99}>) 2: ((FLET SB-UNIX::INTERRUPTION)) 3: ((FLET #:WITHOUT-INTERRUPTS-BODY-[INVOKE-INTERRUPTION]10)) 4: (SB-SYS:INVOKE-INTERRUPTION #<FUNCTION (FLET SB-UNIX::INTERRUPTION) {58049035}>) 5: ("foreign function: call_into_lisp") 6: ("foreign function: post_signal_tramp") 7: ("foreign function: nanosleep") 8: ((LAMBDA ())) 9: (SB-C::INVOKE-WITH-SAVED-FP-AND-PC #<CLOSURE (LAMBDA #) {5A4D929D}>) 10: (SB-UNIX:NANOSLEEP 0 150000016) 11: (SLEEP 0.15) 12: (SWANK-BACKEND::FLUSH-STREAMS) 13: (SWANK-BACKEND::FLUSH-STREAMS)[:EXTERNAL] 14: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) 15: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477)) 16: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {29E3DDE5}> #S(SB-THREAD:MUTEX :NAME "thread result lock" :%OWNER #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> :STATE 1) #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> T) 17: ((LAMBDA ())) 18: ("foreign function: call_into_lisp") 19: ("foreign function: funcall0") 20: ("foreign function: new_thread_trampoline") 21: ("foreign function: _pthread_create") Also, I don't know if it's related, but I see lots of these messages in the log: [2008-11-17 06:59:47 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {5A465A29}>. Thanks for taking a look at this! Cyrus
NB: I do certainly think that it is possible to write threaded web servers that work, I just believe that it is hard, in many respects. Threads, in my personal opinion, are not the best hammer to solve the I/O multiplexing problem that needs to be solved in a web server. Certainly, the unbounded worker thread creation strategy that Hunchentoot uses is not suitable for servers that see load peaks, which is why I recommend not using threaded Hunchentoot for such sites.
That said: I do have stability issues with my non-threaded Hunchentoot installation on FreeBSD, too. I use a multi threaded SBCL, but run Hunchentoot in a single thread behind squid. In some situations, no new connections are being accepted for no apparent reasons, but I failed to properly analyze the problem last time it happened as the customer was already nervous. I have seen this happen two times in the last four months, on a moderately busy site. Thus, the problem may actually not be related to Hunchentoot's threads usage (I'm running non-threaded, you run threaded), but could as well be located in SBCL's thread implementation of FreeBSD, in usocket or somewhere else.
Any further data points would help getting down to the bottom of this.
As for future work on Hunchentoot: We do have the new connection manager class in place which is meant to support the implementation of thread pools. Thread pools would help putting limits on the number of threads created, helping with getting through load peaks. I do not personally need such a connection manager, but rather want to spend some time on making Hunchentoot be able to use single threaded I/O multiplexing using select/kpoll/whatever.
-Hans
On Sun, Nov 16, 2008 at 18:03, Cyrus Harmon <ch-tbnl@bobobeach.com> wrote:
I seem to recall that, a long time ago, TBNL/hunchentoot required threads. At some point Edi rigged up a singled threaded version but claimed that that was just for development/testing work and shouldn't be used in the real world. Now I understand that Hans has an aversion to threads as the multiplexing abstraction for webservers and that single threaded is "the one true way." I bring this up as background to me real problem, which is that, at least on FreeBSD, I seem to have a number of zombie threads that stick around after requests are made. Before I dig into the problem, I'd be curious to hear what folks' recommendations for running a low-traffic, but nevertheless hopefully reliable, site are. I had reasonably good luck with SBCL+threads and hunchentoot before the big rewrite, but since then, it has been reliable refusing to accept new connections after a week or two of use.
Everything starts out fine and I can field some traffic and all is good:
CL-USER> (sb-thread:list-all-threads) (#<SB-THREAD:THREAD "reader-thread" RUNNING {59B6B241}> #<SB-THREAD:THREAD "repl-thread" RUNNING {59B6B0F1}> #<SB-THREAD:THREAD "control-thread" RUNNING {59B60359}> #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {59B601D9}> #<SB-THREAD:THREAD "initial thread" RUNNING {598D79B1}>)
However, after some period, I see an accumulation of worker threads that won't die and eventually I'm unable to start a new thread to field the request. Who knows where the problem is... Could be in hunchentoot proper, in usocket, in SBCL's FreeBSD threads, etc... At the moment I don't have any debugging info on the problem, as all seems to be working ok since I restarted the web server a few minutes ago. The next time the problem crops up, I'll reply to this with some debugging information. This seems to be the one major "regression" from the old hunchentoot that has me considering going back to the stable release (although I've already ported my code forward for the new API, so I'd hate to have to do that). Better yet would be to either track down and fix the problem or to be convinced that this is just broken and is never going to work and figure out how to make my stuff work in a single threaded lisp.
One more question regarding threads, is it possible to use a threaded lisp to act like the single threaded server does? That is to use threads, but to only have one thread doing the hard work, using serve- event for the multiplexing? My initialization code relies on setting some state after starting to listen for requests, so I can't just flip the switch to single-threaded mode and have everything work out of the box.
Thanks for all of the hunchentoot rewrite efforts! On the whole it seems like a good thing, but fixing this issue would make me a much happier hunchentoot user.
Cyrus
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
Hi Cyrus, On Mon, Nov 17, 2008 at 16:17, Cyrus Harmon <ch-tbnl@bobobeach.com> wrote:
On Nov 16, 2008, at 11:08 AM, Hans Hübner wrote:
when you see worker threads accumulate, do you also see that there is a large number of connections to Hunchentoot that are not being closed? Or are there just threads, but no connections? Are you running Hunchentoot behind a proxy/frontend, or standalone? How many dead workers did you see?
So, how do I see if the connections are still open?
On the shell, use "netstat -nf inet | grep [port]", replacing [port] by the port number that the worker thread is serving (35892 and 34650 are the ports for the two workers in the thread list below).
It's only two so far, but here's what it looks like:
So this is the situation in which Hunchentoot does not accept more requests?
CL-USER> (sb-thread:list-all-threads) (#<SB-THREAD:THREAD "repl-thread" RUNNING {5CBD0411}> #<SB-THREAD:THREAD "reader-thread" RUNNING {5CBD02C1}> #<SB-THREAD:THREAD "control-thread" RUNNING {5CBD0169}> #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:35892)" RUNNING {5CDF8E91}> #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:34650)" RUNNING {5CD329A1}> #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> #<SB-THREAD:THREAD "Hunchentoot acceptor (*:443)" RUNNING {5A5C7019}> #<SB-THREAD:THREAD "Hunchentoot acceptor (*:80)" RUNNING {5A3D80B9}> #<SB-THREAD:THREAD "Swank 4005" RUNNING {5A2304A1}> #<SB-THREAD:THREAD "initial thread" RUNNING {598D7BE1}>)
Here's a backtrace from thread 4:
I may be missing something, but to me, it seems as if this backtrace is coming straight from a Slime REPL threads. Are you sure that it is the backtrace of one of the workers?
0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA #) {5A4E49D5}>)[:EXTERNAL] 1: (BACKTRACE 536870911 #<SYNONYM-STREAM :SYMBOL *TERMINAL-IO* {5812FA99}>) 2: ((FLET SB-UNIX::INTERRUPTION)) 3: ((FLET #:WITHOUT-INTERRUPTS-BODY-[INVOKE-INTERRUPTION]10)) 4: (SB-SYS:INVOKE-INTERRUPTION #<FUNCTION (FLET SB-UNIX::INTERRUPTION) {58049035}>) 5: ("foreign function: call_into_lisp") 6: ("foreign function: post_signal_tramp") 7: ("foreign function: nanosleep") 8: ((LAMBDA ())) 9: (SB-C::INVOKE-WITH-SAVED-FP-AND-PC #<CLOSURE (LAMBDA #) {5A4D929D}>) 10: (SB-UNIX:NANOSLEEP 0 150000016) 11: (SLEEP 0.15) 12: (SWANK-BACKEND::FLUSH-STREAMS) 13: (SWANK-BACKEND::FLUSH-STREAMS)[:EXTERNAL] 14: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) 15: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477)) 16: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {29E3DDE5}> #S(SB-THREAD:MUTEX :NAME "thread result lock" :%OWNER #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> :STATE 1) #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> T) 17: ((LAMBDA ())) 18: ("foreign function: call_into_lisp") 19: ("foreign function: funcall0") 20: ("foreign function: new_thread_trampoline") 21: ("foreign function: _pthread_create")
Also, I don't know if it's related, but I see lots of these messages in the log:
[2008-11-17 06:59:47 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {5A465A29}>.
This is normal, although it needs fixing, two. Usually, clients leave HTTP connections to servers open and servers close them whenever they deem it be neccessary as a part of their resource management strategy. Hunchentoot does this, too, by setting a timeout on all TCP sockets which results in a "I/O timeout" error being generated if any connection is idle for more than a configurable number of seconds. This timeout error must really be caught and muted, as it is no exceptional situation, but rather the simple form of resource management that Hunchentoot implements. -Hans
On Nov 17, 2008, at 7:49 AM, Hans Hübner wrote:
Hi Cyrus,
On Mon, Nov 17, 2008 at 16:17, Cyrus Harmon <ch-tbnl@bobobeach.com> wrote:
On Nov 16, 2008, at 11:08 AM, Hans Hübner wrote:
when you see worker threads accumulate, do you also see that there is a large number of connections to Hunchentoot that are not being closed? Or are there just threads, but no connections? Are you running Hunchentoot behind a proxy/frontend, or standalone? How many dead workers did you see?
So, how do I see if the connections are still open?
On the shell, use "netstat -nf inet | grep [port]", replacing [port] by the port number that the worker thread is serving (35892 and 34650 are the ports for the two workers in the thread list below).
Ok, thanks. No, the connections do not seem to be open anymore.
It's only two so far, but here's what it looks like:
So this is the situation in which Hunchentoot does not accept more requests?
No, not yet. This is just when one of those threads was sticking around. Indeed, I did have the wrong thread for the backtrace. This is one of the worker threads that is hanging around: 0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA #) {5D23D5D5}>)[:EXTERNAL] 1: (BACKTRACE 536870911 #<SYNONYM-STREAM :SYMBOL *TERMINAL-IO* {5812FA99}>) 2: ((FLET SB-UNIX::INTERRUPTION)) 3: ((FLET #:WITHOUT-INTERRUPTS-BODY-[INVOKE-INTERRUPTION]10)) 4: (SB-SYS:INVOKE-INTERRUPTION #<FUNCTION (FLET SB-UNIX::INTERRUPTION) {58049035}>) 5: ("foreign function: call_into_lisp") 6: ("foreign function: post_signal_tramp") 7: ("foreign function: select") 8: ((LAMBDA ())) 9: (SB-C::INVOKE-WITH-SAVED-FP-AND-PC #<CLOSURE (LAMBDA #) {5D2157C5}>) 10: ((FLET SB-UNIX::SELECT) #.(SB-SYS:INT-SAP #X2C3DEEF4)) 11: (SB-IMPL::SUB-SUB-SERVE-EVENT 1 0) 12: (SB-IMPL::SUB-SERVE-EVENT 1 0 NIL) 13: (SB-SYS:SERVE-ALL-EVENTS 1) 14: (PROCESS-WAIT #<SB-IMPL::PROCESS 45223 :RUNNING> NIL) 15: (RUN-PROGRAM #P"/usr/home/sly/projects/cyrusharmon.org/www.cyrusharmon.org/hunchy/cgi/gitweb.cgi " NIL)[:EXTERNAL] 16: (HUNCHENTOOT-CGI::HANDLE-CGI-SCRIPT #P"/usr/home/sly/projects/cyrusharmon.org/www.cyrusharmon.org/hunchy/cgi/gitweb.cgi " #<unused argument>) 17: (HUNCHENTOOT::PROCESS-REQUEST #<HUNCHENTOOT:REQUEST {5CE00401}>) 18: ((SB-PCL::FAST-METHOD HUNCHENTOOT::PROCESS-CONNECTION (T T)) #<unavailable argument> #<unavailable argument> #<HUNCHENTOOT::SERVER (host *, port 80)> #<USOCKET:STREAM-USOCKET {5CDECF09}>) 19: ((SB-PCL::FAST-METHOD HUNCHENTOOT::PROCESS-CONNECTION :AROUND (T T)) #<unavailable argument> #S(SB-PCL::FAST-METHOD-CALL :FUNCTION #<FUNCTION #> :PV NIL :NEXT-METHOD-CALL NIL :ARG-INFO (2)) #<HUNCHENTOOT::SERVER (host *, port 80)> #<USOCKET:STREAM-USOCKET {5CDECF09}>) 20: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) 21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477)) 22: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {2C1DEDE5}> #S(SB-THREAD:MUTEX :NAME "thread result lock" :%OWNER #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:35892)" RUNNING {5CDF8E91}> :STATE 1) #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:35892)" RUNNING {5CDF8E91}> T) 23: ((LAMBDA ())) 24: ("foreign function: call_into_lisp") 25: ("foreign function: funcall0") 26: ("foreign function: new_thread_trampoline") 27: ("foreign function: _pthread_create") Hrmm.... this suggests that the problem might be in gitweb.cgi or at least in the way I call it with my HT/CGI interface. Interesting... I'll keep an eye out and see if the rest of the stuck threads are in the same boat. thanks again, Cyrus
participants (6)
-
Andy Chambers -
Cyrus Harmon -
Gary King -
Hans Hübner -
Robert Synnott -
Xristos Kalkanis