tbnl-devel December 2007

tbnl-devel@common-lisp.net

18 participants
12 discussions

[hunchentoot-devel] Hunchentoot performance
by vseguip＠gmail.com 11 Jan '08

11 Jan '08

Hi all, First, as a disclaimer, I'm a total Lisp noobie and have been playing with it recently poking here and there. I just tried Hunchentoot and was happy to see how easy it is to use. I also tried to make some preformance benchmarks on static files and compared it with lighttpd. Unsurprisingly lighttpd beat Hunchentoot quite badly, but I see there is still a lot of improvement for hunchentoot. I did some profiling and added a specialized handler that would serve static files using the linux sendfile system call which really improved the performance. Anyway I still see plenty room for improvements and I would like to take a stab at it, so I did some profiling. I'm putting the results in the ML in case someone wants to discuss it and help me figure out whether the speed for this use case can be improved. Details about the benchmark setup follow: Intel Dual Core 2 2.0 GHz 1 Gb Ram Ubuntu Gutsy 7.10 SBCL version: Hunchentoot version: Lighttpd version: Benchmark program: ab File: 10 mb bitmap Command: ab -n 1000 -c 20 -k http://127.0.0.1/sendfile/1.jpg Using sendfile system call | consed | calls | sec/call | name ----------------------------------------------------------- 0.644 | 25,410,696 | 1,002 | 0.000643 | HUNCHENTOOT::GET-REQUEST-DATA 0.453 | 197,900,328 | 6,006 | 0.000075 | HUNCHENTOOT::WRITE-HEADER-LINE 0.435 | 19,617,768 | 1,001 | 0.000435 | HUNCHENTOOT::HANDLE-STATIC-SENDFILE 0.367 | 0 | 1,001 | 0.000367 | HUNCHENTOOT::SENDTHEFILE 0.230 | 53,605,640 | 1,001 | 0.000230 | HUNCHENTOOT:LOG-MESSAGE 0.163 | 36,415,056 | 2,002 | 0.000082 | HUNCHENTOOT::START-OUTPUT ... Using hunchentoot static folder dispatcher | consed | calls | sec/call | name ----------------------------------------------------------- 0.678 | 165,618,816 | 5,000 | 0.000136 | HUNCHENTOOT::WRITE-HEADER-LINE 0.556 | 27,725,384 | 1,000 | 0.000556 | HUNCHENTOOT::GET-REQUEST-DATA 0.347 | 45,721,424 | 1,000 | 0.000347 | HUNCHENTOOT::START-OUTPUT 0.266 | 57,833,952 | 1,000 | 0.000266 | HUNCHENTOOT:LOG-MESSAGE 0.247 | 20,982,224 | 1,000 | 0.000247 | HUNCHENTOOT::COMPUTE-LENGTH 0.135 | 12,926,840 | 2,000 | 0.000067 | HUNCHENTOOT::FORMAT-ADDRESS 0.115 | 13,878,936 | 1,000 | 0.000115 | HUNCHENTOOT::ISO-TIME 0.082 | 237,656 | 1,000 | 0.000082 | HUNCHENTOOT::MAKE-SOCKET-STREAM 0.079 | 16,425,808 | 1,000 | 0.000079 | HUNCHENTOOT:RFC-1123-DATE .... This is my very dirty implementation of the dispatcher function which is shameles rip of hunchentoot's with some minor functions cleaned : (declaim (inline sendfile)) (define-alien-routine "sendfile" int (out_fd int :in) (in_fd int :in) (offset unsigned-long :in-out) (count unsigned-long :in)) (in-package :hunchentoot) (defun handle-static-sendfile (path &optional content-type) "A function which acts like a Hunchentoot handler for the file denoted by PATH but uses the sendfile call. Send a content type header corresponding to CONTENT-TYPE or \(if that is NIL) tries to determine the content type via the file's suffix." (unless (or (pathname-name path) (pathname-type path)) ;; not a file (setf (return-code) +http-bad-request+) (throw 'handler-done nil)) (unless (probe-file path) ;; does not exist (setf (return-code) +http-not-found+) (throw 'handler-done nil)) (setf (content-type) (or content-type (mime-type path) "application/octet-stream")) (with-open-file (file path :direction :input :element-type '(unsigned-byte 8) :if-does-not-exist nil) (setf (content-length) (file-length file)) (let ((out (chunga:chunked-stream-stream (flex:flexi-stream-stream *hunchentoot-stream*))) (in (sb-sys:fd-stream-fd file))) (send-headers) (force-output out) (sendthefile (sb-sys:fd-stream-fd out) in 0 (content-length))))) (defun sendthefile (out in off size) (common-lisp-user::sendfile out in off size)) Just for reference, the sendfile dispatcher does some 350 req/s wherea hunchentoots does about 228 req/s on my machine and against 1199.41 req/s that lighttps can serve using the sendfile backend (data for a 993Kb jpg file with 20 concurrency level). Thanks for your time, V. Seguí

8 22

[hunchentoot-devel] clisp port
by Anton Vodonosov 30 Dec '07

30 Dec '07

Hello. Attached is the adjustment of the clisp port suggested by Luis Oliveira ( http://common-lisp.net/pipermail/tbnl-devel/2006-November/000815.html) to hunchentoot 0.14.7. I've dropped clisp-serve-event because didn't find it useful. Just use start-server as you do in case of single-threaded sbcl. Edi, is it possible to commit it? I'm using it from about hunchentoot 0.11.1, it works OK for me (not to say i program for hunchentoot too much, but sometimes i do). Clisp port may be useful in at least two cases: - If you want free lisp on windows. SBCL on windows has problems with sockets (for example http://sourceforge.net/mailarchive/message.php?msg_id=33541185225400%40webm…, also it doesn't close sockets and after few thousands of connections windows runs out of handles). Although it isn't a problem for development, in fact I switch freely between sbcl and clisp all the time. - When small memory amount is available. Windows task manager reports memory usage by sbcl process of about 60 MB when hunchentoot is loaded. For clisp it's less then 25 MB. I've recently deployed small bulletin board I'm writing to a linux VPN with 64 MB of memory. The test is to serve static 200 KB file and request it using ApacheBench tool: ab -n 1000 -c 10 http://localhost:4242/static/file (-n is the number of requests, -c number of threads). In case of SBCL few such commands leads to furious swapping, whole system almost hangs and finally linux kills SBCL with message "out of memory". But clisp works stable. Best regards, -Anton

2 4

[hunchentoot-devel] Sessions not secure?
by Sohail Somani 26 Dec '07

26 Dec '07

Hi, In the documentation for Hunchentoot it says that start-session either uses a cookie or url rewriting to maintain the session state. Then it goes on to say: Hunchentoot makes some reasonable effort to prevent eavesdroppers from hijacking sessions (see below), but this should not be considered really secure. Don't store sensitive data in sessions and rely solely on the session mechanism as a safeguard against malicious users who want to get at this data! What is it referring to when it says "see below?" If we assume malicious users can get at the data, is this different from session hijacking? I am assuming (maybe incorrectly) that the following sequence is as secure as the user name and password. * User connects via https (no session exists) * Since no session exists, user is redirected to some login page * User enters user name and password and submits form * This information is authenticated and a session is created * (setf hunchentoot:*session* (start-session)) * On subsequent requests, since the user has a session, we can assume they are authenticated.: * (unless hunchentoot:*session* (redirect-to-login-page)) I guess it all hinges on start-session and how it generates the session id's. Thanks. -- Sohail Somani http://uint32t.blogspot.com

2 8

[hunchentoot-devel] url-encode doesn't encode apostrophes
by Robert Uhl 22 Dec '07

22 Dec '07

I'm using url-encode to encode strings that I use for keys (see e.g. <http://octopodial-chrome.com/tasting-notes/beer/Fuller%26rsquo%3Bs/2000%20V…>; the brewer and beer name strings are both Unicode/HTML and contain characters meaningful in a URL). While running my pages through the w3c validator, I discovered that url-encode doesn't encode apostrophes ('), so now I run my strings through (cl-who:encode-string (hunchentoot:url-encode string)); is this the appropriate way to do things? It might be: url-encode turns a string into a string suitable for a URL and encode-string turns a string into a string suitable for an HTML attribute value. Still, it seems a bit...complex. -- Robert Uhl <http://public.xdi.org/=ruhl> I'm proud to be an old-fashioned bigoted unixoid. They'll take my keyboard away from me when they pry it from the shattered skulls of my enemies. --Mark Hughes

3 6

Re: [Fwd: [hunchentoot-devel] Having troublewithcreate-folder-dispatcher-and-handler]
by Cunningham, Jeff 17 Dec '07

17 Dec '07

Okay. I'll do that. I've been reading the code more closely and I think I see my misunderstanding. The folder-dispatcher seems to be used to enable file dispatch handlers to access files in directories that differ from the URI-prefix? I notice that if I take it out of the hunchentoot-test then it is unable to "view the code". Thanks for clearing this up for me. --Jeff Hans Hübner wrote: > What I am not sure about is whether the error message is the right > thing, but I am pretty sure that hunchentoot does not do directory > listings by default. Can you please check out whether > http://site.znain.com/dl/lisp/hunchentoot-dir-lister/ helps you out? > > -Hans > > On Dec 17, 2007 7:16 PM, Cunningham, Jeff > <jeffrey.k.cunningham(a)boeing.com> wrote: > >> Hans Hübner wrote: >> The change to enough-url modifies it so that it behaves the same as >> enough-namestring (which was previously used) does. What I am not >> sure about is whether the create-folder-dispatcher-and-handler >> dispatcher and handler can handle an empty path. What you'd propably >> want to see is a directory listing? Can you fetch files if you >> specify the complete URL? >> >> -Hans >> >> >> Yes, I can fetch files with a complete URL. And you are right - I'd like to >> see a directory listing. >> I just re-ran hunchentoot-test. With this url I get the image file: >> >> http://localhost:3001/hunchentoot/code/fz.jpg >> >> But with this URL I get a 400: bad request >> >> http://localhost:3001/hunchentoot/code/ >> >> --Jeff >> >> _______________________________________________ >> tbnl-devel site list >> tbnl-devel(a)common-lisp.net >> http://common-lisp.net/mailman/listinfo/tbnl-devel >> >> > _______________________________________________ > tbnl-devel site list > tbnl-devel(a)common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel >

1 0

Re: [Fwd: [hunchentoot-devel] Having trouble withcreate-folder-dispatcher-and-handler]
by Cunningham, Jeff 17 Dec '07

17 Dec '07

Hans Hübner wrote: > The change to enough-url modifies it so that it behaves the same as > enough-namestring (which was previously used) does. What I am not > sure about is whether the create-folder-dispatcher-and-handler > dispatcher and handler can handle an empty path. What you'd propably > want to see is a directory listing? Can you fetch files if you > specify the complete URL? > > -Hans > > Yes, I can fetch files with a complete URL. And you are right - I'd like to see a directory listing. I just re-ran hunchentoot-test. With this url I get the image file: http://localhost:3001/hunchentoot/code/fz.jpg But with this URL I get a 400: bad request http://localhost:3001/hunchentoot/code/ --Jeff

2 1

[Fwd: [hunchentoot-devel] Having trouble with create-folder-dispatcher-and-handler]
by Cunningham, Jeff 17 Dec '07

17 Dec '07

> Can you please try with the attached patch? > > Thanks, > Hans > /> />

2 1

[hunchentoot-devel] Having trouble with create-folder-dispatcher-and-handler
by Jeff Cunningham 17 Dec '07

17 Dec '07

I haven't used this handler for awhile, but I do recall it worked the last time I did. Now, when I set up a simple test case it fails, generating an Internal Server Error (500) with this error message in the log: [error] [Hunchentoot] The value NIL is not of type (MOD 536870911) I isolated it as far as the enough-url function, where it seems that the mismatch is returning NIL and the subseq doesn't like it. My test case is: (create-folder-dispatcher-and-handler "/test/" #p"/home/hunchentoot/test/" "text/plain") Both folders exist and are world readable. I am running SBCL 1.0.12. I tried checking what the arguments are to enough-url when it fails - they are: "url=/test/ url-prefix=/test/ which, of course, explains the NIL and the failure of the subseq. I don't understand how this handler is supposed to work, which it makes it difficult for me to troubleshoot further. Any idea why this doesn't work? --Jeff

2 1

[hunchentoot-devel] Google search result
by Bob Hutchison 15 Dec '07

15 Dec '07

Hi Edi, Debating whether to mention this or not... Have you noticed the text that shows up when you search for Hunchentoot in google: "HUNCHENTOOT - The Common Lisp web server formerly known as TBNL Web server coded in Common Lisp; supports HTTP/1.1 chunking, persistent connections (keep-alive), and SSL; works only with LispWorks." The 'works only with LispWorks' bit is maybe something you don't want there? Cheers, Bob ---- Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/

4 8

[hunchentoot-devel] how to diagnose output problem?
by Jim Prewett 07 Dec '07

07 Dec '07

Hello, I'm new to Hunchentoot. :) I'm working on an app that uses the Demo Allegro Lisp, the AGraph package, and Hunchentoot for the front-end. When I run on MacOS, everything seems great! When I run on Linux, things are not so good. I get *NO* output from Hunchentoot when I put certain records into the database. I don't have any idea if this is a Hunchentoot bug, a Agraph/Franz bug, or a bug in my understanding :) I don't see any errors at the REPL - I've not hit the limitations of the demo ACL. The function I use, when I put (warn ...) statements in, clearly finds all of the records I've requested and thinks its done all of its work (it believes its output the entire webpage), but I get no output :P Can anyone give me suggestions on how to tell /where/ I'm having the problem? I got hunchentoot and friends from asdf-install this morning on both systems (Mac and Linux). I'm running the latest ACL/AGraph. I've set *SHOW-LISP-BACKTRACES-P* and *SHOW-LISP-ERRORS-P* to t and don't see any errors. Here is a transcript of using telnet to request the page: # telnet localhost 80 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /list?user=download HTTP/1.1 200 OK Content-Type: text/html; charset=iso-8859-1 Date: Thu, 06 Dec 2007 15:41:26 GMT Server: Hunchentoot 0.14.7 Connection: Close Set-Cookie: hunchentoot-session=7%3a355b027548e5bd694b3dd0b7b477b018; path=/ Connection closed by foreign host. Thanks for any ideas, Jim James E. Prewett Jim(a)Prewett.org download(a)hpc.unm.edu Systems Team Leader LoGS: http://www.hpc.unm.edu/~download/LoGS/ Designated Security Officer OpenPGP key: pub 1024D/31816D93 HPC Systems Engineer III UNM HPC 505.277.8210

5 8

[hunchentoot-devel] Problem with flexi-stream-bound on openmcl
by Denis Mashkevich 05 Dec '07

05 Dec '07

Hello, I have a problem running hunchentoot on openmcl (OpenMCL Version 1.1-pre-070512 (DarwinPPC32), hunchentoot 0.14.2, flexi-streams 0.13.0) The error is: Undefined function SETF::|HUNCHENTOOT::FLEXI-STREAM-BOUND| called with arguments (NIL #<FLEXI-STREAMS::FLEXI-LATIN-1-IO-STREAM #x92CD646>) . The error is signaled on every request. Doing this as a way of test from REPL seems to be working ok: TBNL> (make-instance 'flex::flexi-latin-1-io-stream ) #<FLEXI-STREAMS::FLEXI-LATIN-1-IO-STREAM #x9374906> TBNL> (setf (flexi-stream-bound *) nil) NIL TBNL> (symbol-package 'flexi-stream-bound) #<Package "FLEXI-STREAMS"> Under lispworks it runs ok on the same system. Backtrace follows: Undefined function SETF::|HUNCHENTOOT::FLEXI-STREAM-BOUND| called with arguments (NIL #<FLEXI-STREAMS::FLEXI-LATIN-1-IO-STREAM #x92CD646>) . [Condition of type CCL::UNDEFINED-FUNCTION-CALL] Restarts: 0: [CONTINUE] Retry applying SETF::|HUNCHENTOOT::FLEXI-STREAM-BOUND| to (NIL #<FLEXI-STREAMS::FLEXI-LATIN-1-IO-STREAM #x92CD646>). 1: [USE-VALUE] Apply specified function to (NIL #<FLEXI- STREAMS::FLEXI-LATIN-1-IO-STREAM #x92CD646>) this time. 2: [STORE-VALUE] Specify a function to use as the definition of SETF::|HUNCHENTOOT::FLEXI-STREAM-BOUND|. 3: [ABORT-BREAK] Reset this process 4: [ABORT] Kill this process Backtrace: 0: (#<Anonymous Function #x85ED03E> 0 20) 1: (SWANK:BACKTRACE 1 41) 2: (CCL::CALL-CHECK-REGS 'SWANK:BACKTRACE) 3: (#<Anonymous Function #x861931E> '(SWANK:BACKTRACE 1 41) 36 'NIL) 4: (#<Anonymous Function #x8574DE6> #<Compiled-function SWANK:SWANK-DEBUGGER-HOOK #x863212E> #<COMPILED-LEXICAL-CLOSURE #x92E6D86>) 5: (FUNCALL 'SWANK::EVAL-FOR-EMACS) 6: (SWANK::SLDB-LOOP #<CONNECTION #x8656776>) 7: (SWANK::CALL-WITH-BINDINGS '((*PRINT-PRETTY* . T) (*PRINT- LEVEL* . 4) (*PRINT-LENGTH* . 10) (*PRINT-CIRCLE* . T) (*PRINT-READABLY*) (*PRINT-PPRINT-DISPATCH* . #S(CCL::PPRINT-DISPATCH-TABLE :CONSES-WITH-CARS #<HASH- TABLE :TEST EQ size 60/60 #x85FEDAE> :STRUCTURES NIL :OTHERS (# #))) (*PRINT-GENSYM* . T) (*PRINT-BASE* . 10) (*PRINT-RADIX*) (*PRINT-ARRAY* . T) ...) #<Anonymous Function #x8631096>) 8: (#<Anonymous Function #x85EDEF6> #<Anonymous Function #x86310E6>) 9: (SWANK::DEBUG-IN-EMACS #<CCL::UNDEFINED-FUNCTION-CALL #x92CCB66>) 10: (#<Anonymous Function #x8574DE6> #<Compiled-function SWANK:SWANK- DEBUGGER-HOOK #x863212E> #<COMPILED-LEXICAL-CLOSURE #x92CC4C6>) 11: (SWANK::CALL-WITH-REDIRECTED-IO #<COMPILED-LEXICAL-CLOSURE #x92CC4AE> #<Compiled-function SWANK:SWANK-DEBUGGER-HOOK #x863212E>) 12: (SWANK::CALL-WITH-CONNECTION #<CONNECTION #x8656776> #<COMPILED- LEXICAL-CLOSURE #x92CC4C6>) 13: (INVOKE-DEBUGGER #<CCL::UNDEFINED-FUNCTION-CALL #x92CCB66>) 14: (SIGNAL #<CCL::UNDEFINED-FUNCTION-CALL #x92CCB66>) 15: (CCL::%ERROR 'NIL #<CCL::FAKE-STACK-FRAME #x233AAE6>) 16: (CCL::DEFAULT-UNDEFINED-FUNCTION-CALL-RESTARTS #<CCL::FAKE-STACK- FRAME #x233AAE6> #<A Foreign Pointer [stack-allocated] #xF0734B40> 'SETF::|CCL::XP-GPR-LISP|) 17: (CCL::HANDLE-UDF-CALL #<CCL::FAKE-STACK-FRAME #x233AAE6> 'CCL::% ERROR) 18: (#<Anonymous Function #x8216916> 67108868 'NIL '(NIL) 'CCL::%ERR- DISP-INTERNAL #<CCL::FAKE-STACK-FRAME #x233AAE6> 2 0 #<A Foreign Pointer [stack-allocated] #xF0734B40>) 19: (CCL::FUNCALL-WITH-ERROR-REENTRY-DETECTION #<COMPILED-LEXICAL- CLOSURE #x233AAA6>) 20: (#<Anonymous Function #x821699E> 67108868 'NIL '(NIL) 'CCL::%ERR- DISP-INTERNAL 2 0 #<A Foreign Pointer [stack-allocated] #xF0734B40> #<CCL::FAKE-STACK-FRAME #x233AAE6>) 21: (CCL::FUNCALL-WITH-XP-STACK-FRAMES 0 0 0) 22: (CCL::%XERR-DISP -65220390) 23: (CCL::%PASCAL-FUNCTIONS% 2 -65220390) 24: (HUNCHENTOOT::PROCESS-CONNECTION #<HUNCHENTOOT::SERVER #x928A32E> #<BASIC-TCP-STREAM ISO-8859-1 (SOCKET/12) #x92BF386>) 25: (CCL::RUN-PROCESS-INITIAL-FORM '(#<COMPILED-LEXICAL-CLOSURE #x92BEA66>) #<PROCESS hunchentoot-worker-1(9) [Active] #x92BEB0E>) 26: (#<Anonymous Function #x8174C16> '(#<COMPILED-LEXICAL-CLOSURE #x92BEA66>) 0) 27: (#<Anonymous Function #x815EF56> 788848 #<LISP-THREAD hunchentoot- worker-1 [tcr @ #x3025C0] #x92BED5E>)

3 7

[hunchentoot-devel] Re: Hunchentoot's query parameter seperator
by Edi Weitz 01 Dec '07

01 Dec '07

Hi, On Tue, 20 Nov 2007 09:31:01 -0500, "Kyle R. Burton" <kyle.burton(a)gmail.com> wrote: > I'm not sure if there is a mailing list that would be more > appropriate. Really? http://weitz.de/hunchentoot/#mail I'd appreciate if we could continue this discussion on the list, see Cc. (You have to subscribe first, of course.) > I'd like to start by saying thanks for Hunchentoot! (and all the > other CL libraries you've developed). You're welcome... :) > I am just starting with Hunchentoot, but I noticed that the query > string parser supports ampersand (&) as a pair separator and I > rememberd semi-colon (;) also being a valid separator for query > strings. I only remember this from having worked with Perl's CGI. > A bit of quick searching on the net lead me to: > > http://en.wikipedia.org/wiki/Query_string > http://www.w3.org/TR/1999/REC-html401-19991224/appendix/notes.html#h-B.2.2 > > to confirm that the semi-colon is mentioned as a separator. I have to admit that this is the first time I've heard of this recommendation. A quick test with the vastly popular PHP shows that they don't implement it either: <? print_r($_GET); ?> Try these: http://zappa.agharta.de/test.html?foo=bar&baz=quux http://zappa.agharta.de/test.html?foo=bar;baz=quux So, to everyone on the list - what's your experience with semicolons as query string separators? Is this normal or esoteric? And, even more interesting, does any client out there actually use this convention? > I looked into the sources and I think the minor change is in > defmethod initialize-instance :after, to just update the split from: > > (form-url-encoded-list-to-alist (split "&" query-string)) > > to > > (form-url-encoded-list-to-alist (split "[;&]" query-string)) > > A quick empirical test with a browser shows symptoms of working: > > /lisp/test?foo=bar&qux=baz&blarf=qbozzle > /lisp/test?foo=bar;qux=baz;blarf=qbozzle > > I'm new to CL so I'm not entirely confident of my suggestion... Seems OK, but it might break existing applications. > Also, is there a place in the source tree for any kind of unit tests > or assertions? I'd have added one for this test, but I didn't > recognize where it would have been appropriate to put it. No, only the demo in the "test/" directory. > Is this something you'd consider enhancing Hunchentoot with? Lots of stuff... Cheers, Edi.

5 7

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

tbnl-devel December 2007