Hi Cyrus,
On Mon, Nov 17, 2008 at 16:17, Cyrus Harmon ch-tbnl@bobobeach.com wrote:
On Nov 16, 2008, at 11:08 AM, Hans Hübner wrote:
when you see worker threads accumulate, do you also see that there is a large number of connections to Hunchentoot that are not being closed? Or are there just threads, but no connections? Are you running Hunchentoot behind a proxy/frontend, or standalone? How many dead workers did you see?
So, how do I see if the connections are still open?
On the shell, use "netstat -nf inet | grep [port]", replacing [port] by the port number that the worker thread is serving (35892 and 34650 are the ports for the two workers in the thread list below).
It's only two so far, but here's what it looks like:
So this is the situation in which Hunchentoot does not accept more requests?
CL-USER> (sb-thread:list-all-threads) (#<SB-THREAD:THREAD "repl-thread" RUNNING {5CBD0411}> #<SB-THREAD:THREAD "reader-thread" RUNNING {5CBD02C1}> #<SB-THREAD:THREAD "control-thread" RUNNING {5CBD0169}> #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:35892)" RUNNING {5CDF8E91}> #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:34650)" RUNNING {5CD329A1}> #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> #<SB-THREAD:THREAD "Hunchentoot acceptor (*:443)" RUNNING {5A5C7019}> #<SB-THREAD:THREAD "Hunchentoot acceptor (*:80)" RUNNING {5A3D80B9}> #<SB-THREAD:THREAD "Swank 4005" RUNNING {5A2304A1}> #<SB-THREAD:THREAD "initial thread" RUNNING {598D7BE1}>)
Here's a backtrace from thread 4:
I may be missing something, but to me, it seems as if this backtrace is coming straight from a Slime REPL threads. Are you sure that it is the backtrace of one of the workers?
0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA #) {5A4E49D5}>)[:EXTERNAL] 1: (BACKTRACE 536870911 #<SYNONYM-STREAM :SYMBOL *TERMINAL-IO* {5812FA99}>) 2: ((FLET SB-UNIX::INTERRUPTION)) 3: ((FLET #:WITHOUT-INTERRUPTS-BODY-[INVOKE-INTERRUPTION]10)) 4: (SB-SYS:INVOKE-INTERRUPTION #<FUNCTION (FLET SB-UNIX::INTERRUPTION) {58049035}>) 5: ("foreign function: call_into_lisp") 6: ("foreign function: post_signal_tramp") 7: ("foreign function: nanosleep") 8: ((LAMBDA ())) 9: (SB-C::INVOKE-WITH-SAVED-FP-AND-PC #<CLOSURE (LAMBDA #) {5A4D929D}>) 10: (SB-UNIX:NANOSLEEP 0 150000016) 11: (SLEEP 0.15) 12: (SWANK-BACKEND::FLUSH-STREAMS) 13: (SWANK-BACKEND::FLUSH-STREAMS)[:EXTERNAL] 14: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) 15: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477)) 16: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {29E3DDE5}> #S(SB-THREAD:MUTEX :NAME "thread result lock" :%OWNER #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> :STATE 1) #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}> T) 17: ((LAMBDA ())) 18: ("foreign function: call_into_lisp") 19: ("foreign function: funcall0") 20: ("foreign function: new_thread_trampoline") 21: ("foreign function: _pthread_create")
Also, I don't know if it's related, but I see lots of these messages in the log:
[2008-11-17 06:59:47 [ERROR]] Error while processing connection: I/O timeout reading #<SB-SYS:FD-STREAM for "a socket" {5A465A29}>.
This is normal, although it needs fixing, two. Usually, clients leave HTTP connections to servers open and servers close them whenever they deem it be neccessary as a part of their resource management strategy. Hunchentoot does this, too, by setting a timeout on all TCP sockets which results in a "I/O timeout" error being generated if any connection is idle for more than a configurable number of seconds. This timeout error must really be caught and muted, as it is no exceptional situation, but rather the simple form of resource management that Hunchentoot implements.
-Hans