April 2011 - tbnl-devel@common-lisp.net - mailman3.common-lisp.net

[hunchentoot-devel] Hunchentoot Errors (e.g. Couldn't write to #<SB-SYS:FD-STREAM for "socket ..." ...>)
by Cyrus Harmon 30 Oct '11

30 Oct '11

Hello tbnl-folk, So, I've been seeing transient problems with hunchentoot for as long as I can remember, and I haven't had the chance to try to track down the problems until now. It seems there are a couple of classes of problems. In order of severity, the are: 1. things that crash hunchentoot/SBCL (or, in some cases, the host OS) 2. things that throw "unexpected" errors 3. things that throw "expected" errors (e.g. 404) As for class 1, these are tough to track down as the server is usually crashed by the time I get to it, but some causes I consider plausible are: * running out of some limited resource such as RAM, threads, or network sockets * possible SBCL bug in, say, signal handling or dealing with lots of threads I don't have any data to present on this class, at the moment, so let's move on to the next class. I often see errors such as this in the log: [2011-04-05 00:04:52 [ERROR]] Couldn't write to #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}>: Broken pipe 0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA #) {B71EC4D}> :START 0 :COUNT 536870911) 1: (BACKTRACE 536870911 #<SB-IMPL::STRING-OUTPUT-STREAM {B71EBE9}>) 2: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #<SB-IMPL::STRING-OUTPUT-STREAM {B71EBE9}>) 3: (HUNCHENTOOT::GET-BACKTRACE) 4: ((FLET #:LAMBDA271) #<SB-INT:SIMPLE-STREAM-ERROR "~@<~?: ~2I~_~A~:>" {B71EA51}>) 5: (SIGNAL #<SB-INT:SIMPLE-STREAM-ERROR "~@<~?: ~2I~_~A~:>" {B71EA51}>) 6: (ERROR SB-INT:SIMPLE-STREAM-ERROR :STREAM #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}> :FORMAT-CONTROL "~@<~?: ~2I~_~A~:>" :FORMAT-ARGUMENTS ("Couldn't write to ~S" (#<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}>) "Broken pipe")) 7: (SB-IMPL::SIMPLE-STREAM-PERROR "Couldn't write to ~S" #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}> 32) 8: (SB-IMPL::WRITE-OUTPUT-FROM-QUEUE #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}>) 9: (SB-IMPL::SUB-SUB-SERVE-EVENT NIL NIL) 10: (SB-IMPL::SUB-SERVE-EVENT NIL NIL NIL) 11: (SB-SYS:SERVE-ALL-EVENTS NIL) 12: (SB-IMPL::FINISH-FD-STREAM-OUTPUT #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}>) 13: (FINISH-OUTPUT #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}>) 14: (HUNCHENTOOT:HANDLE-STATIC-FILE #P"/home/sly/projects/www-cyrusharmon-org/static/images/fish-clusters.png" NIL) 15: ((SB-PCL::FAST-METHOD HUNCHENTOOT:HANDLE-REQUEST (HUNCHENTOOT:ACCEPTOR HUNCHENTOOT:REQUEST)) #<unavailable argument> #<unavailable argument> #<HUNCHENTOOT:ACCEPTOR (host *, port 80)> #<HUNCHENTOOT:REQUEST {B701D09}>) 16: ((SB-PCL::FAST-METHOD HUNCHENTOOT:PROCESS-REQUEST (T)) #<unavailable argument> #<unavailable argument> #<HUNCHENTOOT:REQUEST {B701D09}>) 17: ((SB-PCL::FAST-METHOD HUNCHENTOOT:PROCESS-CONNECTION (HUNCHENTOOT:ACCEPTOR T)) #<unavailable argument> #<unavailable argument> #<HUNCHENTOOT:ACCEPTOR (host *, port 80)> #<USOCKET:STREAM-USOCKET {B6FB8F1}>) 18: ((SB-PCL::FAST-METHOD HUNCHENTOOT:PROCESS-CONNECTION :AROUND (HUNCHENTOOT:ACCEPTOR T)) #<unavailable argument> #S(SB-PCL::FAST-METHOD-CALL :FUNCTION #<FUNCTION #> :PV NIL :NEXT-METHOD-CALL NIL :ARG-INFO (2)) #<HUNCHENTOOT:ACCEPTOR (host *, port 80)> #<USOCKET:STREAM-USOCKET {B6FB8F1}>) 19: ((LAMBDA ())) 20: ((FLET #:WITHOUT-INTERRUPTS-BODY-[BLOCK353]358)) 21: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) 22: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]300)) 23: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {B0E14235}> #S(SB-THREAD:MUTEX :NAME "thread result lock" :%OWNER #<SB-THREAD:THREAD "Hunchentoot worker (client: 217.72.218.248:2741)" RUNNING {B6FCE39}> :STATE 1) #<SB-THREAD:THREAD "Hunchentoot worker (client: 217.72.218.248:2741)" RUNNING {B6FCE39}> T) 24: (SB-THREAD::INITIAL-THREAD-FUNCTION) 25: ("foreign function: call_into_lisp") 26: ("foreign function: funcall0") 27: ("foreign function: new_thread_trampoline") 28: ("foreign function: #xB7FC5506") What does this whole "broken pipe" mean? Is this more of an "expected error" or is something going wrong here? On a slightly different note, I'd like to be able to run a relatively reliable hunchentoot server that can serve a modest traffic load without having to resort to caching proxy servers and the like. I recall some discussion of thread pools, epoll, etc... Has the state of the art for high-reliability hunchentoot instances changed in the last couple years? Any suggestions on improving it? thanks! Cyrus

2 1

[hunchentoot-devel] Garbage in REPL
by Andrey Moskvitin 30 Oct '11

30 Oct '11

Hi, For ACCESS-LOG-DESIGNATION default value is *ERROR-OUTPUT*. It is the cause of littering REPL for SLIME users. Maybe better use NIL as default and for suppress logging use a keyword :suppress? (defmethod acceptor-access-log-destination ((acceptor acceptor)) (let ((destination (slot-value acceptor 'access-log-destination))) (case destination ((nil) *error-output*) (:suppress nil) (otherwise destination)))) In this case, the logging will be sent to *inferior-lisp* and will not interfere with. Andrey

2 1

[hunchentoot-devel] socket-connector and ssl-socket-connector?
by Cyrus Harmon 29 Sep '11

29 Sep '11

I've been playing around with some changes to ACCEPTOR and friends that introduce the new classes SOCKET-CONNECTOR and SSL-SOCKET-CONNECTOR. The purpose of these changes are to take the specialized socket handling behavior out of the acceptor and into another class, such that subclassing acceptor isn't required to, say, use SSL streams. As a motivating example, consider the following: (hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4243 :socket-connector (make-instance 'hunchentoot::ssl-socket-connector :ssl-certificate-file (asdf:system-relative-pathname :hunchentoot "ssl/certificate.pem") :ssl-privatekey-file (asdf:system-relative-pathname :hunchentoot "ssl/key.pem")))) (hunchentoot:define-easy-handler (say-yo :uri "/yo") (name) (setf (hunchentoot:content-type*) "text/plain") (format nil "Hey~@[ ~A~]!" name)) Without these changes, one would have to subclass both easy-acceptor and ssl-acceptor in order to have an SSL-using easy acceptor. Comments/suggestions greatly appreciated. The changes can be found in the git repo at: https://github.com/slyrus/hunchentoot/tree/socket-connector and in the attached patch. Cyrus

3 9

[hunchentoot-devel] Patch for ABCL SET-TIMEOUTS
by Mark Evenson 28 Apr '11

28 Apr '11

The attached patch provides an implementation for Hunchentoot's SET-TIMEOUTS running on ABCL. Please consider it for inclusion in the Hunchentoot source. With this patch, [the development versions of ABCL is now able to run Hunchentoot with small additional patches to Quicklisp bordeaux-threads and usocket][1]. Currently one has to add :HUNCHENTOOT-NO-SSL to *FEATURES* for Hunchentoot to work as CFFI doesn't implement the necessary callbacks for CL+SSL to work (but [progress in rectifying this has been made][2]. And testing via Drakma needs trivial-garbage for which we [only recently implemented the necessary finalizers][3] but haven't wrestled out a suitable patch. [1]: http://detroit.slack.net/~evenson/abcl/hunchentoot [2]: http://detroit.slack.net/~evenson/abcl/cffi-abcl-20101231a.diff [3]: http://trac.common-lisp.net/armedbear/changeset/13250 -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."

1 0

[hunchentoot-devel] dispatcher class
by Cyrus Harmon 18 Apr '11

18 Apr '11

Continuing my efforts to solve the problem of wanting, for instance, an SSL-accepting easy-acceptor without creating a subclass of both SSL-ACCEPTOR and EASY-ACCEPTOR, I've created a new class called DISPATCHER which implements the dispatch-request generic function. The advantage of this class is that one can add custom dispatching behavior without needing to subclass the acceptor. For instance, the EASY-ACCEPTOR is now an EASY-DISPATCHER and can be invoked as: (hunchentoot:start (make-instance 'hunchentoot:acceptor :port 4243 :dispatcher (make-instance 'hunchentoot:easy-dispatcher))) New dispatching behavior can be implemented by subclassing DISPATCHER and setting the dispatcher slot of the acceptor to point to an instance of the subclass. The changes can be found in git at: https://github.com/slyrus/hunchentoot/tree/dispatcher and in the attached patch. Cyrus

1 0

[hunchentoot-devel] Hunchentoot Request Processing
by Cyrus Harmon 17 Apr '11

17 Apr '11

[Before diving in to try to "fix" some things in the way hunchentoot handles requests, I figured it would be good to try to more fully understand how hunchentoot handles incoming requests, so I decided to take some notes on the whole process. I thought these might be food for thought or discussion for the list, so here they are - cyrus] Hunchentoot has a flexible and extensible model for managing connections between the hunchentoot server and its clients. In olden times, there was a hunchentoot SERVER class which received incoming requests and dispatched these requests to handlers that would produce the appropriate responses. In recent versions of hunchentoot, this functionality has been split into a few different classes, or families of classes. The ACCEPTOR and TASKMASTER classes (or, more precisely, they and their subclasses) now work together to listen for incoming connections and to dispatch them appropriately. The logic of the TASKMASTER and ACCEPTOR classes is somewhat baroque, although this is probably not without good reason. Let's walk through a typical process of starting up hunchentoot. * Initializing the Server 1. Create an Acceptor and a Taskmaster Usually, this is done in one fell swoop with a call such as: (make-instance 'hunchentoot:acceptor) This creates an acceptor that listens for incoming connections on a given port, 80 by default. There's an important detail here that can usually be ignored by the user, which is that the acceptor needs a taskmaster to manage the process of actually receiving connections from the lower level networking infrastructure. The reference to the taskmaster is stored in the ACCEPTOR's TASKMASTER slot and the value of this slot is usually provided directly by the default-initargs section of the ACCEPTOR class definition which reads: :taskmaster (make-instance (cond (*supports-threads-p* 'one-thread-per-connection-taskmaster) (t 'single-threaded-taskmaster))) So, when running on a suitably threaded lisp implementation, a ONE-THREAD-PER-CONNECTION-TASKMASTER is created, or on a single-threaded lisp a SINGLE-THREADED-TASKMASTER is created, and this value is stored in the TASKMASTER slot of the ACCEPTOR. 2. ACCEPTOR / START method Once the ACCEPTOR has been created and its TASKMASTER slot suitably initialized, the START method of the ACCEPTOR is called. START then calls the ACCEPTOR's START-LISTENING method. 3. ACCEPTOR / START-LISTENING method The START-LISTENING method tells the underlying networking infrastructure to listen for incoming connections on a (single) port. On lispworks, the acceptor starts a "server" process with comm:startup-server and then stops this process, with it ready to be restarted in ACCEPT-CONNECTIONS. On non-lispworks lisps, this method sets the LISTEN-SOCKET slot to a USOCKET:STREAM-SERVER-USOCKET as returned by USOCKET:SOCKET-LISTEN. 4. Establish reciprocal connection from TASKMASTER back to ACCEPTOR Back in the ACCEPTOR's START method, and now that the ACCEPTOR is listening for traffic, we establish the link from the TASKMASTER back to this ACCEPTOR (notice that there is a tight 1:1 coupling between ACCEPTORs and TASKMASTERs, although the decoupling of the two classes allows for subclassing of one independently from the other -- a good thing!). 5. TASKMASTER / EXECUTE-ACCEPTOR method The TASKMASTER's EXECUTE-ACCEPTOR doesn't do a lot of work -- it turns around and calls the ACCEPTOR's ACCEPT-CONNECTIONS method, but the critical bit here is that the (particular subclass of) TASKMASTER is free to spawn a new thread in which to call ACCEPT-CONNECTIONS (as is done in the ONE-THREAD-PER-CONNECTION-TASKMASTER), or it can call it in its (the TASKMASTER's and, presumably, the ACCEPTOR's) own thread (as in the SINGLE-THREADED-TASKMASTER). 6. ACCEPTOR / ACCEPT-CONNECTIONS ACCEPT-CONNECTIONS either listens for input using USOCKET:WAIT-FOR-INPUT on non-lispworks lisps, or, on lispworks, wakes the server process using MP:PROCESS-UNSTOP. On non-lispworks lisps, once the stream is ready for input, ACCEPT-CONNECTIONS calls its TASKMASTER's HANDLE-INCOMING-CONNECTION method. On Lispworks, the ACCEPTOR's TASKMASTER's HANDLE-INCOMING-CONNECTION method is called from the callback provided to COMM:START-UP-SERVER. In either case, we transfer control back to the taskmaster once we have a connection ready to provide input. * Processing Requests 7a. TASKMASTER / HANDLE-INCOMING-CONNECTION Once again, the TASKMASTER comes into play, as we have different behavior for single- and multi-threaded taskmasters, and, of course, different behavior for lispworks and non-lispworks lisps. [n.b. The reader conditionals for lispworks/non-lispworks seem a bit bogus to me. Is there any reason the bordeaux/usocket code couldn't be run on lispworks? If the bordeaux/usocket can run on lispworks, are there advantages to the lispworks-specific code? Could/should the lispworks specific code be split off into, say, LISPWORKS-ONE-THREAD-PER-CONNECTION-TASKMASTER and the usocket version into USOCKET-ONE-THREAD-PER-CONNECTION-TASKMASTER? Clearly, the lispworks-specific stuff shouldn't be compiled on non-lispworks implementations (and maybe vice versa), but it seems that a more flexible mechanism for deciding to use the lispworks-stuff or the usocket-stuff on lispworks would be a nice feature.] 7b. (with ONE-THREAD-PER-CONNECTION-TASKMASTER taskmasters) TASKMASTER / CREATE-REQUEST-HANDLER-THREAD The ONE-THREAD-PER-CONNECTION-TASKMASTER's CREATE-REQUEST-HANDLER-THREAD spawns a new thread and calls its ACCEPTOR's PROCESS-CONNECTION method. With SINGLE-THREADED-TASKMASTER, HANDLE-INCOMING-CONNECTION calls process-connection directly. 8. ACCEPTOR / PROCESS-CONNECTION Now control switches back to the ACCEPTOR which initializes the connection stream and calls PROCESS-REQUEST on a newly created instance of the ACCEPTOR's REQUEST-CLASS. One important thing that happens in PROCESS-CONNECTION is that INITIALIZE-CONNECTION-STREAM is called with the acceptorand USOCKET:SOCKET-STREAM (as provided by the compatibility function make-socket-stream). One reason that this is important is that this is the "hook" that the SSL-ACCEPTOR uses to make the connection stream a CL+SSL:SSL-SERVER-STREAM. In fact, this (and ACCEPTOR-SSL-P) are the only methods specialized by the SSL-ACCEPTOR. This seems to be a pretty clean place to try to break the SSL functionality off into another class that is not a subclass of ACCEPTOR. Whether or not this is a good idea remains to be seen. Q: Why do we set *CLOSE-HUNCHENTOOT-STREAM* to T inside the loop here? Q: If we created a new class to implement INITIALIZE-CONNECTION-STREAM, what would it be called? 9. REQUEST / PROCESS-REQUEST Having initialized the connection stream (usually a no-op, except for SSL connections), we now move to the next task at hand, processing the actual request. This is done via an unspecialized method on the PROCESS-REQUEST generic function, which happens to live in the request.lisp file, but doesn't actually specialize on request. PROCESS-REQUEST sets up some error handlers, calls HANDLE-REQUEST and then starts (and maybe finishes) sending the HTTP reply. 10. ACCEPTOR / HANDLE-REQUEST The HANDLE-REQUEST method specializes on acceptor and request as the first two arguments and, with the appropriate error handling forms in place, turns around calls the ACCEPTOR's ACCEPTOR-DISPATCH-REQUEST method. 11. ACCEPTOR / ACCEPTOR-DISPATCH-REQUEST The ACCEPTOR-DISPATCH-REQUEST method is the one place in the new hunchentoot source were user-extensible request handling is demonstrated. The new, so-called, easy-handler extends the ACCEPTOR class as EASY-ACCEPTOR and defines an ACCEPTOR-DISPATCH-REQUEST specialized on EASY-ACCEPTOR as its first argument. This method is called in preference to the ACCEPTOR's version (for EASY-ACCEPTOR's anyway) and, in this case, loops over the values in the *dispatch-table* special variable and funcalls each function on the list with the request as the single argument to the function. * Some final comments While this entire process is modularized, somewhat flexible and extensible, in practice it's a bit baroque and a bit challenging to figure out exactly where and how to extend the functionality of the core server. As a case in point, how is one supposed to create an acceptor that is both an easy-acceptor and works over SSL connections? Make a new ACCEPTOR subclass that extends both EASY-ACCEPTOR and SSL-ACCEPTOR? I don't see why that wouldn't work, but it also seems somehow wrong. It seems to me that there should be a way to compose these kinds of functionalities without triggering a combinatorial subclass explosion. The initialization of the connection stream and the actual process of handling the request seem like they might be components for somehow being further modularized. The challenge is to figure out a nice way to do this. * Some ideas for new classes 1. REQUEST-PROCESSOR 2. SOCKET-CONNECTOR

2 1

[hunchentoot-devel] post
by Haris 15 Apr '11

15 Apr '11

Hi. Can Hunchentoot page automaticaly post parameters when fetched from some other page's javascript code to get an ajax effect ? Thanks

1 0

[hunchentoot-devel] recent-ish hunchentoot changes
by Cyrus Harmon 12 Apr '11

12 Apr '11

So, I see that way requests are handled has changed a bit in the last couple months. This leads me to a couple questions: 1. This is still only in the svn (and various mirrors) versions, not yet in a released version, right? Any idea when we should expect a release with these changes? 2. default-dispatcher and *default-handler* seem to have gone away, right? and *dispatch-table* is now an easy-handler-only thing? 3. is the "new" way to handle requests to subclass acceptor, as easy-acceptor does? How is this supposed to play with the ssl-acceptor? Should we define regular (non-ssl) acceptors and ssl-acceptors for our code? I think of http vs. https as more of a transport issue than an application logic issue. It seems a little odd to be using subclassing, along one axis, for http vs. https and for application-logic, along another axis. Am I missing something? In case anyone's curious, I'm trying to bring my hunchentoot-vhost forward to work with the new hunchentoot, and it's requiring a bit of a rethinking of how I handle requests. I'm happy to make a hunchentoot-vhost-acceptor, if that's the way to go, but I'm a little less thrilled about having to make a hunchentoot-vhost-ssl-acceptor too. I think the new design generally looks nice, but it's still taking some getting used to. thanks! Cyrus

2 2

[hunchentoot-devel] ht-simple-ajax
by Haris 07 Apr '11

07 Apr '11

Hi. I hope I can ask questions about ht-simple-ajax here. If not, please tell me where I can. I'm trying out ht-simple-ajax but it doesn't work. I have: (defparameter *ajax-processor* (make-instance 'ajax-processor :server-uri "/ajax")) (defun-ajax say-hi (name) (*ajax-processor*) (concatenate 'string "Hi " name)) (setq *dispatch-table* (list 'dispatch-easy-handlers (create-ajax-dispatcher *ajax-processor*))) And then I go to: http://localhost/ajax/SAY-HI?name=Martin but nothing happens ? Thanks

4 13

Re: [hunchentoot-devel] [cl-plus-ssl-devel] ssl crashing in hunchentoot. A solution?
by Daniel Brunner 04 Apr '11

04 Apr '11

Hi, the patch crashes my CCL on Windows XP (32 Bit) as well. Sometimes with the first request, sometimes after 10 or 20 requests. But it seems not very stable on Windows. Kind regards, Daniel Am 04.04.2011 11:29, schrieb Daniel Brunner: > Hello, > > I started testing the patch. > > On Linux (X8664) it runs without any problems using CCL and SBCL. > > On Windows 7 (64 Bit) CCL crashes my application after a while without > any notice. But I have not figured out what is going wrong. > > Kind regards, Daniel. > > Am 04.04.2011 09:54, schrieb Anton Vodonosov: >> >> Hello. >> >> 03.04.2011, 17:26, "David Lichteblau" <david(a)lichteblau.com>: >> >>> I have committed an updated version of the patch to CL+SSL CVS now. >> >> BTW, from your patch I saw you are using git. Haven't you thought to move CL+SSL >> repository to some git hosting? >> >>>> It's done from Lisp, and should really be done directly in C, similar to >>>> the th-lock.c sample code from OpenSSL. But let's first check whether >>>> it addresses the issue at all. >>> >>> If anyone is interested in developing a C version, let's move to that >>> ultimately. But I've decided to commit the Lisp version for now, >>> because I think that correctness should come before performance in this >>> case. >> >> IMHO the Lisp solution is better, because it frees end user from compiling >> C code. Before developing a C version, we should be sure that performance >> is really somehow significantly affected by the fact that the locking and thread-id >> callbacks are in Lisp. Anyway, cl+ssl is embedded into systems where everything >> is in Lisp. >> >> (I wander, why OpenSSL does not include these locking and thread-id functions >> itself and require users to copy/paste the C code from the examples.) >> >> Best regards, >> - Anton >> >> _______________________________________________ >> cl-plus-ssl-devel mailing list >> cl-plus-ssl-devel(a)common-lisp.net >> http://common-lisp.net/cgi-bin/mailman/listinfo/cl-plus-ssl-devel > >

1 0