December 2007 - tbnl-devel@common-lisp.net

[hunchentoot-devel] Hunchentoot performance
by vseguip＠gmail.com 11 Jan '08

11 Jan '08

Hi all, First, as a disclaimer, I'm a total Lisp noobie and have been playing with it recently poking here and there. I just tried Hunchentoot and was happy to see how easy it is to use. I also tried to make some preformance benchmarks on static files and compared it with lighttpd. Unsurprisingly lighttpd beat Hunchentoot quite badly, but I see there is still a lot of improvement for hunchentoot. I did some profiling and added a specialized handler that would serve static files using the linux sendfile system call which really improved the performance. Anyway I still see plenty room for improvements and I would like to take a stab at it, so I did some profiling. I'm putting the results in the ML in case someone wants to discuss it and help me figure out whether the speed for this use case can be improved. Details about the benchmark setup follow: Intel Dual Core 2 2.0 GHz 1 Gb Ram Ubuntu Gutsy 7.10 SBCL version: Hunchentoot version: Lighttpd version: Benchmark program: ab File: 10 mb bitmap Command: ab -n 1000 -c 20 -k http://127.0.0.1/sendfile/1.jpg Using sendfile system call | consed | calls | sec/call | name ----------------------------------------------------------- 0.644 | 25,410,696 | 1,002 | 0.000643 | HUNCHENTOOT::GET-REQUEST-DATA 0.453 | 197,900,328 | 6,006 | 0.000075 | HUNCHENTOOT::WRITE-HEADER-LINE 0.435 | 19,617,768 | 1,001 | 0.000435 | HUNCHENTOOT::HANDLE-STATIC-SENDFILE 0.367 | 0 | 1,001 | 0.000367 | HUNCHENTOOT::SENDTHEFILE 0.230 | 53,605,640 | 1,001 | 0.000230 | HUNCHENTOOT:LOG-MESSAGE 0.163 | 36,415,056 | 2,002 | 0.000082 | HUNCHENTOOT::START-OUTPUT ... Using hunchentoot static folder dispatcher | consed | calls | sec/call | name ----------------------------------------------------------- 0.678 | 165,618,816 | 5,000 | 0.000136 | HUNCHENTOOT::WRITE-HEADER-LINE 0.556 | 27,725,384 | 1,000 | 0.000556 | HUNCHENTOOT::GET-REQUEST-DATA 0.347 | 45,721,424 | 1,000 | 0.000347 | HUNCHENTOOT::START-OUTPUT 0.266 | 57,833,952 | 1,000 | 0.000266 | HUNCHENTOOT:LOG-MESSAGE 0.247 | 20,982,224 | 1,000 | 0.000247 | HUNCHENTOOT::COMPUTE-LENGTH 0.135 | 12,926,840 | 2,000 | 0.000067 | HUNCHENTOOT::FORMAT-ADDRESS 0.115 | 13,878,936 | 1,000 | 0.000115 | HUNCHENTOOT::ISO-TIME 0.082 | 237,656 | 1,000 | 0.000082 | HUNCHENTOOT::MAKE-SOCKET-STREAM 0.079 | 16,425,808 | 1,000 | 0.000079 | HUNCHENTOOT:RFC-1123-DATE .... This is my very dirty implementation of the dispatcher function which is shameles rip of hunchentoot's with some minor functions cleaned : (declaim (inline sendfile)) (define-alien-routine "sendfile" int (out_fd int :in) (in_fd int :in) (offset unsigned-long :in-out) (count unsigned-long :in)) (in-package :hunchentoot) (defun handle-static-sendfile (path &optional content-type) "A function which acts like a Hunchentoot handler for the file denoted by PATH but uses the sendfile call. Send a content type header corresponding to CONTENT-TYPE or \(if that is NIL) tries to determine the content type via the file's suffix." (unless (or (pathname-name path) (pathname-type path)) ;; not a file (setf (return-code) +http-bad-request+) (throw 'handler-done nil)) (unless (probe-file path) ;; does not exist (setf (return-code) +http-not-found+) (throw 'handler-done nil)) (setf (content-type) (or content-type (mime-type path) "application/octet-stream")) (with-open-file (file path :direction :input :element-type '(unsigned-byte 8) :if-does-not-exist nil) (setf (content-length) (file-length file)) (let ((out (chunga:chunked-stream-stream (flex:flexi-stream-stream *hunchentoot-stream*))) (in (sb-sys:fd-stream-fd file))) (send-headers) (force-output out) (sendthefile (sb-sys:fd-stream-fd out) in 0 (content-length))))) (defun sendthefile (out in off size) (common-lisp-user::sendfile out in off size)) Just for reference, the sendfile dispatcher does some 350 req/s wherea hunchentoots does about 228 req/s on my machine and against 1199.41 req/s that lighttps can serve using the sendfile backend (data for a 993Kb jpg file with 20 concurrency level). Thanks for your time, V. Seguí

8 22

[hunchentoot-devel] clisp port
by Anton Vodonosov 30 Dec '07

30 Dec '07

Hello. Attached is the adjustment of the clisp port suggested by Luis Oliveira ( http://common-lisp.net/pipermail/tbnl-devel/2006-November/000815.html) to hunchentoot 0.14.7. I've dropped clisp-serve-event because didn't find it useful. Just use start-server as you do in case of single-threaded sbcl. Edi, is it possible to commit it? I'm using it from about hunchentoot 0.11.1, it works OK for me (not to say i program for hunchentoot too much, but sometimes i do). Clisp port may be useful in at least two cases: - If you want free lisp on windows. SBCL on windows has problems with sockets (for example http://sourceforge.net/mailarchive/message.php?msg_id=33541185225400%40webm…, also it doesn't close sockets and after few thousands of connections windows runs out of handles). Although it isn't a problem for development, in fact I switch freely between sbcl and clisp all the time. - When small memory amount is available. Windows task manager reports memory usage by sbcl process of about 60 MB when hunchentoot is loaded. For clisp it's less then 25 MB. I've recently deployed small bulletin board I'm writing to a linux VPN with 64 MB of memory. The test is to serve static 200 KB file and request it using ApacheBench tool: ab -n 1000 -c 10 http://localhost:4242/static/file (-n is the number of requests, -c number of threads). In case of SBCL few such commands leads to furious swapping, whole system almost hangs and finally linux kills SBCL with message "out of memory". But clisp works stable. Best regards, -Anton

2 4

[hunchentoot-devel] Sessions not secure?
by Sohail Somani 26 Dec '07

26 Dec '07

Hi, In the documentation for Hunchentoot it says that start-session either uses a cookie or url rewriting to maintain the session state. Then it goes on to say: Hunchentoot makes some reasonable effort to prevent eavesdroppers from hijacking sessions (see below), but this should not be considered really secure. Don't store sensitive data in sessions and rely solely on the session mechanism as a safeguard against malicious users who want to get at this data! What is it referring to when it says "see below?" If we assume malicious users can get at the data, is this different from session hijacking? I am assuming (maybe incorrectly) that the following sequence is as secure as the user name and password. * User connects via https (no session exists) * Since no session exists, user is redirected to some login page * User enters user name and password and submits form * This information is authenticated and a session is created * (setf hunchentoot:*session* (start-session)) * On subsequent requests, since the user has a session, we can assume they are authenticated.: * (unless hunchentoot:*session* (redirect-to-login-page)) I guess it all hinges on start-session and how it generates the session id's. Thanks. -- Sohail Somani http://uint32t.blogspot.com

2 8

[hunchentoot-devel] url-encode doesn't encode apostrophes
by Robert Uhl 22 Dec '07

22 Dec '07

I'm using url-encode to encode strings that I use for keys (see e.g. <http://octopodial-chrome.com/tasting-notes/beer/Fuller%26rsquo%3Bs/2000%20V…>; the brewer and beer name strings are both Unicode/HTML and contain characters meaningful in a URL). While running my pages through the w3c validator, I discovered that url-encode doesn't encode apostrophes ('), so now I run my strings through (cl-who:encode-string (hunchentoot:url-encode string)); is this the appropriate way to do things? It might be: url-encode turns a string into a string suitable for a URL and encode-string turns a string into a string suitable for an HTML attribute value. Still, it seems a bit...complex. -- Robert Uhl <http://public.xdi.org/=ruhl> I'm proud to be an old-fashioned bigoted unixoid. They'll take my keyboard away from me when they pry it from the shattered skulls of my enemies. --Mark Hughes

3 6

Re: [Fwd: [hunchentoot-devel] Having troublewithcreate-folder-dispatcher-and-handler]
by Cunningham, Jeff 17 Dec '07

17 Dec '07

Okay. I'll do that. I've been reading the code more closely and I think I see my misunderstanding. The folder-dispatcher seems to be used to enable file dispatch handlers to access files in directories that differ from the URI-prefix? I notice that if I take it out of the hunchentoot-test then it is unable to "view the code". Thanks for clearing this up for me. --Jeff Hans Hübner wrote: > What I am not sure about is whether the error message is the right > thing, but I am pretty sure that hunchentoot does not do directory > listings by default. Can you please check out whether > http://site.znain.com/dl/lisp/hunchentoot-dir-lister/ helps you out? > > -Hans > > On Dec 17, 2007 7:16 PM, Cunningham, Jeff > <jeffrey.k.cunningham(a)boeing.com> wrote: > >> Hans Hübner wrote: >> The change to enough-url modifies it so that it behaves the same as >> enough-namestring (which was previously used) does. What I am not >> sure about is whether the create-folder-dispatcher-and-handler >> dispatcher and handler can handle an empty path. What you'd propably >> want to see is a directory listing? Can you fetch files if you >> specify the complete URL? >> >> -Hans >> >> >> Yes, I can fetch files with a complete URL. And you are right - I'd like to >> see a directory listing. >> I just re-ran hunchentoot-test. With this url I get the image file: >> >> http://localhost:3001/hunchentoot/code/fz.jpg >> >> But with this URL I get a 400: bad request >> >> http://localhost:3001/hunchentoot/code/ >> >> --Jeff >> >> _______________________________________________ >> tbnl-devel site list >> tbnl-devel(a)common-lisp.net >> http://common-lisp.net/mailman/listinfo/tbnl-devel >> >> > _______________________________________________ > tbnl-devel site list > tbnl-devel(a)common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel >

1 0

Re: [Fwd: [hunchentoot-devel] Having trouble withcreate-folder-dispatcher-and-handler]
by Cunningham, Jeff 17 Dec '07

17 Dec '07

Hans Hübner wrote: > The change to enough-url modifies it so that it behaves the same as > enough-namestring (which was previously used) does. What I am not > sure about is whether the create-folder-dispatcher-and-handler > dispatcher and handler can handle an empty path. What you'd propably > want to see is a directory listing? Can you fetch files if you > specify the complete URL? > > -Hans > > Yes, I can fetch files with a complete URL. And you are right - I'd like to see a directory listing. I just re-ran hunchentoot-test. With this url I get the image file: http://localhost:3001/hunchentoot/code/fz.jpg But with this URL I get a 400: bad request http://localhost:3001/hunchentoot/code/ --Jeff

2 1

[Fwd: [hunchentoot-devel] Having trouble with create-folder-dispatcher-and-handler]
by Cunningham, Jeff 17 Dec '07

17 Dec '07

> Can you please try with the attached patch? > > Thanks, > Hans > /> />

2 1

[hunchentoot-devel] Having trouble with create-folder-dispatcher-and-handler
by Jeff Cunningham 17 Dec '07

17 Dec '07

I haven't used this handler for awhile, but I do recall it worked the last time I did. Now, when I set up a simple test case it fails, generating an Internal Server Error (500) with this error message in the log: [error] [Hunchentoot] The value NIL is not of type (MOD 536870911) I isolated it as far as the enough-url function, where it seems that the mismatch is returning NIL and the subseq doesn't like it. My test case is: (create-folder-dispatcher-and-handler "/test/" #p"/home/hunchentoot/test/" "text/plain") Both folders exist and are world readable. I am running SBCL 1.0.12. I tried checking what the arguments are to enough-url when it fails - they are: "url=/test/ url-prefix=/test/ which, of course, explains the NIL and the failure of the subseq. I don't understand how this handler is supposed to work, which it makes it difficult for me to troubleshoot further. Any idea why this doesn't work? --Jeff

2 1

[hunchentoot-devel] Google search result
by Bob Hutchison 15 Dec '07

15 Dec '07

Hi Edi, Debating whether to mention this or not... Have you noticed the text that shows up when you search for Hunchentoot in google: "HUNCHENTOOT - The Common Lisp web server formerly known as TBNL Web server coded in Common Lisp; supports HTTP/1.1 chunking, persistent connections (keep-alive), and SSL; works only with LispWorks." The 'works only with LispWorks' bit is maybe something you don't want there? Cheers, Bob ---- Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/

4 8

[hunchentoot-devel] how to diagnose output problem?
by Jim Prewett 07 Dec '07

07 Dec '07

Hello, I'm new to Hunchentoot. :) I'm working on an app that uses the Demo Allegro Lisp, the AGraph package, and Hunchentoot for the front-end. When I run on MacOS, everything seems great! When I run on Linux, things are not so good. I get *NO* output from Hunchentoot when I put certain records into the database. I don't have any idea if this is a Hunchentoot bug, a Agraph/Franz bug, or a bug in my understanding :) I don't see any errors at the REPL - I've not hit the limitations of the demo ACL. The function I use, when I put (warn ...) statements in, clearly finds all of the records I've requested and thinks its done all of its work (it believes its output the entire webpage), but I get no output :P Can anyone give me suggestions on how to tell /where/ I'm having the problem? I got hunchentoot and friends from asdf-install this morning on both systems (Mac and Linux). I'm running the latest ACL/AGraph. I've set *SHOW-LISP-BACKTRACES-P* and *SHOW-LISP-ERRORS-P* to t and don't see any errors. Here is a transcript of using telnet to request the page: # telnet localhost 80 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /list?user=download HTTP/1.1 200 OK Content-Type: text/html; charset=iso-8859-1 Date: Thu, 06 Dec 2007 15:41:26 GMT Server: Hunchentoot 0.14.7 Connection: Close Set-Cookie: hunchentoot-session=7%3a355b027548e5bd694b3dd0b7b477b018; path=/ Connection closed by foreign host. Thanks for any ideas, Jim James E. Prewett Jim(a)Prewett.org download(a)hpc.unm.edu Systems Team Leader LoGS: http://www.hpc.unm.edu/~download/LoGS/ Designated Security Officer OpenPGP key: pub 1024D/31816D93 HPC Systems Engineer III UNM HPC 505.277.8210

5 8