On Nov 17, 2008, at 7:49 AM, Hans Hübner wrote:
Hi Cyrus,
On Mon, Nov 17, 2008 at 16:17, Cyrus Harmon ch-tbnl@bobobeach.com wrote:
On Nov 16, 2008, at 11:08 AM, Hans Hübner wrote:
when you see worker threads accumulate, do you also see that there is a large number of connections to Hunchentoot that are not being closed? Or are there just threads, but no connections? Are you running Hunchentoot behind a proxy/frontend, or standalone? How many dead workers did you see?
So, how do I see if the connections are still open?
On the shell, use "netstat -nf inet | grep [port]", replacing [port] by the port number that the worker thread is serving (35892 and 34650 are the ports for the two workers in the thread list below).
Ok, thanks. No, the connections do not seem to be open anymore.
It's only two so far, but here's what it looks like:
So this is the situation in which Hunchentoot does not accept more requests?
No, not yet. This is just when one of those threads was sticking around. Indeed, I did have the wrong thread for the backtrace. This is one of the worker threads that is hanging around:
0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA #) {5D23D5D5}>)[:EXTERNAL] 1: (BACKTRACE 536870911 #<SYNONYM-STREAM :SYMBOL *TERMINAL-IO* {5812FA99}>) 2: ((FLET SB-UNIX::INTERRUPTION)) 3: ((FLET #:WITHOUT-INTERRUPTS-BODY-[INVOKE-INTERRUPTION]10)) 4: (SB-SYS:INVOKE-INTERRUPTION #<FUNCTION (FLET SB-UNIX::INTERRUPTION) {58049035}>) 5: ("foreign function: call_into_lisp") 6: ("foreign function: post_signal_tramp") 7: ("foreign function: select") 8: ((LAMBDA ())) 9: (SB-C::INVOKE-WITH-SAVED-FP-AND-PC #<CLOSURE (LAMBDA #) {5D2157C5}>) 10: ((FLET SB-UNIX::SELECT) #.(SB-SYS:INT-SAP #X2C3DEEF4)) 11: (SB-IMPL::SUB-SUB-SERVE-EVENT 1 0) 12: (SB-IMPL::SUB-SERVE-EVENT 1 0 NIL) 13: (SB-SYS:SERVE-ALL-EVENTS 1) 14: (PROCESS-WAIT #<SB-IMPL::PROCESS 45223 :RUNNING> NIL) 15: (RUN-PROGRAM #P"/usr/home/sly/projects/cyrusharmon.org/www.cyrusharmon.org/hunchy/cgi/gitweb.cgi " NIL)[:EXTERNAL] 16: (HUNCHENTOOT-CGI::HANDLE-CGI-SCRIPT #P"/usr/home/sly/projects/cyrusharmon.org/www.cyrusharmon.org/hunchy/cgi/gitweb.cgi " #<unused argument>) 17: (HUNCHENTOOT::PROCESS-REQUEST #<HUNCHENTOOT:REQUEST {5CE00401}>) 18: ((SB-PCL::FAST-METHOD HUNCHENTOOT::PROCESS-CONNECTION (T T)) #<unavailable argument> #<unavailable argument> #<HUNCHENTOOT::SERVER (host *, port 80)> #<USOCKET:STREAM-USOCKET {5CDECF09}>) 19: ((SB-PCL::FAST-METHOD HUNCHENTOOT::PROCESS-CONNECTION :AROUND (T T)) #<unavailable argument> #S(SB-PCL::FAST-METHOD-CALL :FUNCTION #<FUNCTION #> :PV NIL :NEXT-METHOD-CALL NIL :ARG-INFO (2)) #<HUNCHENTOOT::SERVER (host *, port 80)> #<USOCKET:STREAM-USOCKET {5CDECF09}>) 20: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) 21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477)) 22: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {2C1DEDE5}> #S(SB-THREAD:MUTEX :NAME "thread result lock" :%OWNER #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:35892)" RUNNING {5CDF8E91}> :STATE 1) #<SB-THREAD:THREAD "Hunchentoot worker (client: 66.255.53.123:35892)" RUNNING {5CDF8E91}> T) 23: ((LAMBDA ())) 24: ("foreign function: call_into_lisp") 25: ("foreign function: funcall0") 26: ("foreign function: new_thread_trampoline") 27: ("foreign function: _pthread_create")
Hrmm.... this suggests that the problem might be in gitweb.cgi or at least in the way I call it with my HT/CGI interface. Interesting... I'll keep an eye out and see if the rest of the stuck threads are in the same boat.
thanks again,
Cyrus