- tbnl-devel@lists.common-lisp.net - mailman3.common-lisp.net

[hunchentoot-devel] hunchentoot cookie-values - url-encode them or not
by Ralf Stoye 02 Feb '12

02 Feb '12

As you can see on git, future hunchentoot versions will not url-encode/decode cookie-values anymore. Since this encoding is an de facto standard regarding php, java and perl, and it might break existing code, i would like to hear your opinion about that. At a minimum i want to suggest to introduce a sanity-check on #'cookie-values throwing an error if someone tries to set the value to a nonconforming value (similar to the check on valid cookie-names). The following code is an example predicate: (defun http-cookie-value-p (value) "Tests whether VALUE is a string which is a valid cookie-value according to RFC 6265" (and (stringp value) (not (some (lambda (char) (let ((cc (char-code char))) (or (< cc #x21) (= #x22 cc) (= #x2c cc) (= #x3b cc) (= #x5c cc) (> cc #x7e)))) value)))) the decision is a matter of performance versus simplicity: we could decide to always encode cookie values (eg. per base64 or url-encode, or both) or leave the decision and responsability to the application. Regards, Ralf Stoye

2 3

[hunchentoot-devel] Patch Submission: Custom Reason Phrases
by Kari Lentz 01 Feb '12

01 Feb '12

Included is a patch that incorporates the ability to have a request specific reason phrase that allows one to use code such along the lines of (setf (return-code*) (make-custom-status 500 "some database query error"))) This would override the templates and the cooked messages but only for that request and is intended to be an answer to the "StatusDescription" member of the HTTPResponse class in the ASP.NET library. Essentially, a custom-status-t class has been added that links a code with a reason phrase. The "make-custom-status" function above is just a constructor to this new class. The "reason-phrase" function is now a generic method that dispatches off both the custom-status-t class and the previous numeric status codes to be backward compatible. I also ended up changing the error code dispatch to be CLOS based using eql dispatch instead of a case statement for the various http-status-code values. It seems to be a trend to use generic functions instead of a case statement in this situation, probably because of a hash table type dispatch is used behind the scenes rather than going from start to finish as in a case statement. Cheers, Kari

3 2

[hunchentoot-devel] GET parameters and request external format
by Dmitriy Ivanov 24 Jan '12

24 Jan '12

Hello folks, OT1H, the charset and external format of a request are extracted from the Content-Type header and used to percent-decode post parameters in the function maybe-read-post-parameters. OTOH, get parameters are always 1) initialized in the method #<initialize-instance :after (request) >, 2) percent-decoded using *hunchentoot-default-external-format*. That works fine when all pages of the site use a the same external format - just setting *hunchentoot-default-external-format* globally is enough. But if one wanted various pages to be in different formats, that could be inappropriate. For now, it is even buggy because within the current flexi-streams version, the decode error is not suppressed by following code: (let ((*substitution-char* #\?)) (form-url-encoded-list-to-alist (split "&" query-string))) I have bumped into all this while trying to process forms in the :windows-1251 external format. AFAIK, user agents percent-encode get parameters using in the charset specified in the previous server response. Without post parameters, user agents do not supply the charset in Content-Type as a rule. Thus only a Hunchentoot request handler can definitely know what the charset is actually. All that means that we should postpone retrieval of get parameters as far as possible. Maybe it is worth not to specify the :initform for the get-parameters slot and leave it unbound? Then we could initialize this slot with the following: (defmethod slot-unbound (class (request request) (slot-name (eql 'get-parameters))) (setf (slot-value request 'get-parameters) (form-url-encoded-list-to-alist (split "&" (query-string request)) (request-external-format request)))) The only question is how to compute such a request-external-format. Any takers? -- Sincerely, Dmitriy Ivanov lisp.ystok.ru

1 0

[hunchentoot-devel] Error on LWW 4.4.6: -1 cannot be converted to foreign type (:UNSIGNED-INTEGER-TYPE 32).
by Dmitriy Ivanov 19 Jan '12

19 Jan '12

Hello folks, I have bumped into the error while playing around: 1. (hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4242 :write-timeout nil)) ; for lispworks4 2. Browsing results from test-handlers.lisp. It worked fine until I made a break for half an hour. When I come back to my computer and click on some link, LispWorks signaled. The odd thing is that in call to COMM::LISTEN-AND-ATTACH-STREAM-FROM-SOCKET, COMM::SOCKET : 464 But in call to (METHOD HUNCHENTOOT:CREATE-REQUEST-HANDLER-THREAD (HUNCHENTOOT:ONE-THREAD-PER-CONNECTION-TASKMASTER T)) HUNCHENTOOT::SOCKET : -1 Looks like LispWorks (or even Windows) error. Has anybody observed something like this? The backtrace excerpt is below. --------------------------------------------------- Condition: -1 cannot be converted to foreign type (:UNSIGNED-INTEGER-TYPE 32). Call to (SUBFUNCTION 2 COMM:START-UP-SERVER) (offset 43) COMM::SOCKET : NIL CONDITION : #<FLI:FOREIGN-TYPE-ERROR 206AB0CC> Call to (SUBFUNCTION 1 COMM::LISTEN-AND-ATTACH-STREAM-NO-ERROR) (offset 41) CONDITION : #<FLI:FOREIGN-TYPE-ERROR 206AB0CC> Binding frame: CONDITIONS::*IN-SIGNAL-CATCH* : T Call to SIGNAL (offset 699) CONDITIONS::DATUM : #<FLI:FOREIGN-TYPE-ERROR 206AB0CC> CONDITIONS::ARGUMENTS : NIL Binding frame: CONDITIONS::*IN-SIGNAL-CATCH* : NIL Catch frame: CONDITIONS::SIGNAL-CATCH Binding frame: CONDITIONS::*BROKEN-ON-SIGNALS* : NIL Catch frame: #<unbound> Binding frame: FLI::*FOREIGN-OBJECT-SCOPE-STACK-DETAILS* : #(16 #1=(8519856 . 4096) (#1#)) Call to COMM::GETPEERNAME (offset 227) COMM::S : -1 COMM::NAME : #<Pointer to type (:STRUCT COMM::SOCKADDR) = #x008200B0> COMM::NAMELEN : 16 Catch frame: #<unbound> Binding frame: FLI::*FOREIGN-OBJECT-SCOPE-STACK-DETAILS* : #(0 NIL NIL) Call to COMM::GET-SOCKET-ADDRESS-AUX (offset 370) COMM::SOCKET : -1 FUNCTION : COMM::GETPEERNAME Call to HUNCHENTOOT::GET-PEER-ADDRESS-AND-PORT (offset 43) HUNCHENTOOT::SOCKET : -1 Call to (METHOD HUNCHENTOOT:CREATE-REQUEST-HANDLER-THREAD (HUNCHENTOOT:ONE-THREAD-PER-CONNECTION-TASKMASTER T)) (offset 97) HUNCHENTOOT:TASKMASTER : #<HUNCHENTOOT:ONE-THREAD-PER-CONNECTION-TASKMASTER 22087454> HUNCHENTOOT::SOCKET : -1 CLOS::.ISL. : :DONT-KNOW Catch frame: #<unbound> Call to COMM::LISTEN-AND-ATTACH-STREAM-FROM-SOCKET (offset 133) COMM::SOCKET : 464 FUNCTION : #<closure (SUBFUNCTION 3 (METHOD HUNCHENTOOT:START-LISTENING (HUNCHENTOOT:ACCEPTOR))) 22087F32> Catch frame: (NIL) Call to COMM::LISTEN-AND-ATTACH-STREAM (offset 619) FUNCTION : #<closure (SUBFUNCTION 3 (METHOD HUNCHENTOOT:START-LISTENING (HUNCHENTOOT:ACCEPTOR))) 22087F32> COMM::SERVICE : 4242 COMM::ANNOUNCE : #<closure (SUBFUNCTION 2 COMM:START-UP-SERVER) 22087F62> COMM::BACKLOG : 5 COMM::ADDRESS : NIL Catch frame: (NIL) Call to COMM::LISTEN-AND-ATTACH-STREAM-NO-ERROR (offset 284) FUNCTION : #<closure (SUBFUNCTION 3 (METHOD HUNCHENTOOT:START-LISTENING (HUNCHENTOOT:ACCEPTOR))) 22087F32> COMM::SERVICE : 4242 COMM::ANNOUNCE : #<closure (SUBFUNCTION 2 COMM:START-UP-SERVER) 22087F62> COMM::BACKLOG : 5 COMM::ADDRESS : NIL Catch frame: MP::PROCESS-TAG ... -- Sincerely, Dmitriy Ivanov lisp.ystok.ru

2 1

[hunchentoot-devel] What does +latin-1+ means ?
by z_axis 13 Jan '12

13 Jan '12

Can we use +utf-8+ instead of +latin-1+ in handler ? It is a newbie question but the following code doesnot work using +latin-1+: (let ((out (flexi-streams:make-flexi-stream (tbnl:send-headers) :external-format tbnl::+utf-8+))) ; tbnl::+;latin-1+ doesnot work (copy-stream in out 'character)) where, "in" holds utf-8 data. In hunchentoot-1.2.2, it says that +latin-1+ is "A FLEXI-STREAMS external format used for `faithful' input and output of binary data.", which i cannot understand very well. Sincerely!

1 0

[hunchentoot-devel] Why can't the python script get the post parameters from hunchentoot-cgi?
by z_axis 11 Jan '12

11 Jan '12

The hunchentoot-cgi basically works with hunchentoot-1.2.2 now with a few modification: 1. In #'handle-cgi-script, replacing :external-format tbnl::+latin-1+ with :external-format tbnl::+utf-8+ 2. As WhiteCat suggested, use #'make-pathname as below: (pushnew (hunchentoot-cgi::create-cgi-dispatcher-and-handler "/cgi-bin/" (make-pathname :directory '(:absolute "media" "E" "myapp" "cgi-bin"))) dispatch-table :test #'equal) However, I do not understand why the python script cannot get the POSTed parameter, i.e., when visiting http://127.0.0:8000/cgi-bin/login.py?cmd=view, login.py can get the parameter cmd's value sucessfully, but when posted by the following form, login.py cannot get all the posted values (no matter hidden or not): <html><body> <form method='POST' action='cgi-bin/login.py'> <input type='text' name='userid'> <input type='password' name='userpwd'> <input type='submit' value='Login'> <input type='hidden' name='cmd' value='view'> </form> </body></html> I guess hunchentoot-cgi does not pass the posted parameter(s) to the python script according to the environment: GET http://127.0.0.1:8000/cgi-bin/nav.py?userid=xyz&cmd=view (SERVER_SOFTWARE=hunchentoot/1.2.2 SERVER_NAME=127.0.0.1 GATEWAY_INTERFACE=CGI/1.1 SERVER_PROTOCOL=HTTP/1.1 SERVER_PORT=8000 REQUEST_METHOD=GET CONTENT_TYPE=text/html CONTENT_LENGTH=NIL SCRIPT_NAME=/cgi-bin/nav.py QUERY_STRING=userid=xyz&cmd=view REMOTE_ADDR=127.0.0.1 HTTP_HOST=NIL REQUEST_URI=/cgi-bin/nav.py?userid=xyz&cmd=view SERVER_ADDR=NIL HTTP_USER_AGENT=Mozilla/5.0 (X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1 HTTP_REFERER=http://127.0.0.1:8000/) POST to /cgi-bin/nav.py (SERVER_SOFTWARE=hunchentoot/1.2.2 SERVER_NAME=127.0.0.1 GATEWAY_INTERFACE=CGI/1.1 SERVER_PROTOCOL=HTTP/1.1 SERVER_PORT=8000 REQUEST_METHOD=POST CONTENT_TYPE=text/html CONTENT_LENGTH=NIL POST_PARAMETERS=((userid . xyz) (userpwd . 123) (cmd . view)) SCRIPT_NAME=/cgi-bin/nav.py QUERY_STRING=NIL REMOTE_ADDR=127.0.0.1 HTTP_HOST=NIL REQUEST_URI=/cgi-bin/nav.py SERVER_ADDR=NIL HTTP_USER_AGENT=Mozilla/5.0 (X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1 Please shed a light on me, thanks!

4 9

[hunchentoot-devel] Problem with (setf (hunchentoot:reply-external-format*) ...)
by Raymond Wiker 10 Jan '12

10 Jan '12

I have (finally!) updated my local installation of hunchentoot. Testing indicates that things work as expected, except for one thing: calling (setf (hunchentoot:reply-external-format*) ...) in a request handler causes the request to abort. This is probably not a big issue, since I probably don't need to set the external format (it appears to get the right value automatically). I'm reporting it here because it may actually be caused by a bug; alternatively, it could be that I'm doing something stupid, which would then be useful to know. The code snippet below works (after making an easy-acceptor). Removing the #+nil causes it to break. (hunchentoot:define-easy-handler (test :uri "/test") (foo) (setf (hunchentoot:content-type*) "text/plain") #+nil (setf (hunchentoot:reply-external-format*) :utf-8) (format nil "Host: ~a~%Proto: ~a (~a)~%Script: ~a~%requrl: ~a~%foo: ~a" (hunchentoot:host) (hunchentoot:server-protocol*) (type-of (hunchentoot:server-protocol*)) (hunchentoot:script-name*) (hunchentoot:request-uri*) foo))

2 5

[hunchentoot-devel] Problems with big uploads
by Stefan Nobis 09 Jan '12

09 Jan '12

Hi. I have written a small and very simple web-service to which I want to upload files. With the current version 1.2.1 of hunchentoot I have problems uploading big files, say 1GB. As far as I remeber, with older versions (1.0 and 1.1) big uploads worked well. Basically the following code is in use: --8<---------------cut here---------------start------------->8--- (hunchentoot:define-easy-handler (handle-upload :uri "/path/to/upload-service") () (let ((uploaded (when (and (boundp 'hunchentoot:*request*) (hunchentoot:post-parameter "filename")) (handle-file (hunchentoot:post-parameter "filename"))))) (generate-html-code))) --8<---------------cut here---------------end--------------->8--- And handle-file looks like this: --8<---------------cut here---------------start------------->8--- (defun handle-file (post-parameter) (ht-log :info "Handling file upload with params: '~A'." post-parameter) (when (and post-parameter (listp post-parameter)) (destructuring-bind (path filename content-type) post-parameter (declare (ignore content-type)) ;; strip directory info send by Windows browsers (when (search "Windows" (hunchentoot:user-agent) :test #'char-equal) (setf filename (ppcre:regex-replace ".*\\\\" filename ""))) (fad:copy-file path (ensure-directories-exist (merge-pathnames filename *unsecure-upload-dir*)) :overwrite t) filename))) --8<---------------cut here---------------end--------------->8--- It seems that hunchentoot tries to read the whole stream into memory and that the heap is too small (the server has only 1GB RAM and the heap of the sbcl process is limited to about 600MB). Is there any (easy) way to let hunchentoot load the data in small chunks in order to limit the maximal amount of memory used, independent of the file size? -- Until the next mail..., Stefan.

2 3

Re: [hunchentoot-devel] easy-handler outputs to REPL instead of client
by Tyler Smith 23 Dec '11

23 Dec '11

Thank you! I didn't realize that with-html-output was returning a string, I thought it was emitting the html to the output as a side effect. In retrospect, that's silly. The following works, although it looks a little unwieldy. Any suggestions for improvement are welcome. Otherwise, thanks for your help and for all your code. Cheers, Tyler (define-easy-handler (tyler :uri "/tyler") () (multiple-value-bind (records fields) (select [genus] [species] :from "specimens") (loop for rec in records collect (loop for label in fields for val in rec collect (with-html-output-to-string (*standard-output*) (:dt :class label (str label)) (:dd :class label (str val))) into inner-output finally (return (format nil "~{~a~&~}" inner-output))) into outer-output finally (return (format nil "~{~a~&~}" outer-output))))) On Fri, Dec 23, 2011 at 3:05 AM, Edi Weitz <edi(a)agharta.de> wrote: > Your loop doesn't return anything, does it? > > On Thu, Dec 22, 2011 at 11:51 PM, Tyler Smith > <tyler.smith(a)mail.mcgill.ca> wrote: >> Hi, >> >> I'm trying to set up a simple sql-based website using cl-who, cl-sql >> and hunchentoot. I've got the basic cl-sql and cl-who parts working, >> but I'm having a trouble serving the pages with hunchentoot. >> >> The following works as expected: >> >> (ql:quickload :hunchentoot) >> (use-package :hunchentoot) >> >> (setf my-acceptor (start (make-instance 'hunchentoot:easy-acceptor :port 4242))) >> >> (define-easy-handler (tyler :uri "/tyler") () >> (with-html-output (*standard-output*) >> (:ul (:li "hi there again")))) >> >> >> After this, if I open localhost:4242/tyler in a browser I see my >> unordered list. However, with this: >> >> (define-easy-handler (tyler :uri "/tyler") () >> (multiple-value-bind (records fields) >> (select [genus] [species] :from "specimens") >> (loop for rec in records do >> (loop for label in fields >> for val in rec do >> (with-html-output (*standard-output*) >> (:dt :class label (str label)) >> (:dd :class label (str val))))))) >> >> When I browse to localhost:4242/tyler nothing appears in the browser, >> instead the html is printed in the REPL. If I evaluate just the (M-V-B >> ...) form without the enclosing (D-E-H ...) it outputs the html to the >> REPL as I expect. >> >> I have been following the examples in the hunchentoot docs, but I'm >> stumped as to why this doesn't work. What have I missed? >> >> Thanks, >> >> Tyler >> >> _______________________________________________ >> tbnl-devel site list >> tbnl-devel(a)common-lisp.net >> http://common-lisp.net/mailman/listinfo/tbnl-devel >> > > _______________________________________________ > tbnl-devel site list > tbnl-devel(a)common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel

2 1

[hunchentoot-devel] easy-handler outputs to REPL instead of client
by Tyler Smith 23 Dec '11

23 Dec '11

Hi, I'm trying to set up a simple sql-based website using cl-who, cl-sql and hunchentoot. I've got the basic cl-sql and cl-who parts working, but I'm having a trouble serving the pages with hunchentoot. The following works as expected: (ql:quickload :hunchentoot) (use-package :hunchentoot) (setf my-acceptor (start (make-instance 'hunchentoot:easy-acceptor :port 4242))) (define-easy-handler (tyler :uri "/tyler") () (with-html-output (*standard-output*) (:ul (:li "hi there again")))) After this, if I open localhost:4242/tyler in a browser I see my unordered list. However, with this: (define-easy-handler (tyler :uri "/tyler") () (multiple-value-bind (records fields) (select [genus] [species] :from "specimens") (loop for rec in records do (loop for label in fields for val in rec do (with-html-output (*standard-output*) (:dt :class label (str label)) (:dd :class label (str val))))))) When I browse to localhost:4242/tyler nothing appears in the browser, instead the html is printed in the REPL. If I evaluate just the (M-V-B ...) form without the enclosing (D-E-H ...) it outputs the html to the REPL as I expect. I have been following the examples in the hunchentoot docs, but I'm stumped as to why this doesn't work. What have I missed? Thanks, Tyler

5 4