I've finally been motivated to do some maintenance on my web server and one of the things that has been bothering me for a long time has been some relatively serious instability problems with my production- ish hunchentoot server. I had been using a pre-1.0-but-post-last- release development snapshot and the main problem has been that after a week or so of use my hunchentoot instance would fairly reliably crash, although it wasn't clear exactly when it would crash. After I time begin to suspect that the main culprit use my use of sb-ext:run- program in my hunchentoot-cgi stuff. hunchentoot-cgi is a hunchentoot handler and associated glue to allow hunchentoot to call CGI programs written in, say, perl that I wrote primarily so I could put gitweb behind my hunchentoot server directly, instead using apache or lighttpd or whatever as a front-end to hunchentoot. Like most of my code, it is, at least in the beginning, SBCL specific and it uses sb- ext:run-program to launch the CGI script. So far, so good, but then I kept getting these crashes and it was driving me crazy.
Today I figured out that if I manually close the process, bad things happen a lot less frequently. This may seem obvious, but if after calling sb-ext:run-program I make an unwind-protect block and call sb- ext:process-wait and sb-ext:process-close, things get a lot better. I guess I should have suspected that something like this might have been the problem when I had errors such as:
[2009-06-10 20:37:03 [ERROR]] The value 1026 is not of type (MOD 1025).
in the logs, but I never put two and two together.
I can now bang on the server (calling CGIs) pretty hard and it generally responds well and hasn't crashed on me yet, although it hasn't been all that long. Nevertheless, a thousand or so hits from ab with the old way of doing things would bring down the server and I can for multiple thousands of hits on the new server with no noticeable problems.
I should caveat this by saying that I was seeing the occasional sockout-timeout, writing to a closed pipe, etc... problems with the 1.0 (or thereabouts, whatever is in luis' repo) release, but upgrading to the latest dev version from the ediware svn repo seems to have fixed even those problems. Well, I still occasionally see errors like the following:
[2009-06-10 22:35:25 [ERROR]] Couldn't write to #<SB-SYS:FD-STREAM for "a socket" {5A0E5BA1}>: Broken pipe [2009-06-10 22:35:26 [ERROR]] Error while processing connection: Couldn't write to #<SB-SYS:FD-STREAM for "a socket" {5A0E5BA1}>: Broken pipe
but they neither bring down the server nor cause ab to freak out with a connection reset by peer error, which is nice.
So I guess this all a long-winded way of saying:
1) If you're going to be using sb-ext:run-program :wait nil and reading from the process' stream, make sure you sb-ext:process-close the process.
2) thanks to the hunchentoot team for the 1.0 release and beyond!
While I'm at it, I'll put in another plug for my hunchentoot add-on modules, hunchentoot-cgi, hunchentoot-auth and hunchentoot-vhost, for calling CGIs from hunchentoot, for providing (more of) an infrastructure for user authentication, and for trivially handling multiple "virtual hosts" with hunchentoot, respectively:
http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=hunchentoot-cgi.git http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=hunchentoot-auth.git http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=hunchentoot-vhost.git
All of which should now be working with hunchentoot 1.0 (and beyond).
Thanks again to Edi and Hans.
Cyrus