[hunchentoot-devel] hunchentoot-cgi and the new hunchentoot regime
So... I'm trying to convert my hunchentoot-cgi over to the new hunchentoot and running into some problems. My old approach was to do: (progn (let* ((return-code (tbnl::return-code)) (reason-phrase (reason-phrase return-code)) (first-line (format nil "HTTP/1.1 ~D ~A" return-code reason-phrase))) (write-sequence (map 'list #'char-code first-line) out) (write-sequence tbnl::+crlf+ out) (tbnl::maybe-write-to-header-stream first-line)) (setf tbnl::*headers-sent* t) (setf (tbnl::content-type) nil) (sb-ext::run-program path nil :output out :environment env) nil) but this doesn't seem to work anymore. My new approach is to try to use send-headers, but with some hackery to make it optionally more cgi-friendly: Index: headers.lisp =================================================================== --- headers.lisp (revision 3423) +++ headers.lisp (working copy) @@ -70,6 +70,8 @@ (:method (key value) (write-header-line key (princ-to-string value)))) +(defparameter *cgi-hack* nil) + (defun start-output (&key (content nil content-provided-p) (request *request*)) "Sends all headers and maybe the content body to @@ -84,7 +86,8 @@ ;; Read post data to clear stream - Force binary mode to avoid OCTETS-TO-STRING overhead. (raw-post-data :force-binary t) (let* ((return-code (return-code)) - (chunkedp (and (server-output-chunking-p *server*) + (chunkedp (and (not *cgi-hack*) + (server-output-chunking-p *server*) (eq (server-protocol request) :http/1.1) ;; only turn chunking on if the content ;; length is unknown at this point... @@ -196,12 +199,14 @@ ;; write all headers from the REPLY object (loop for (key . value) in (headers-out) when value - do (write-header-line (as-capitalized-string key) value)) + do (unless (and *cgi-hack* (equal key :content-type)) + (write-header-line (as-capitalized-string key) value))) ;; now the cookies (loop for (nil . cookie) in (cookies-out) do (write-header-line "Set-Cookie" (stringify-cookie cookie))) ;; all headers sent - (write-sequence +crlf+ *hunchentoot-stream*) + (unless *cgi-hack* + (write-sequence +crlf+ *hunchentoot-stream*)) (maybe-write-to-header-stream "") ;; access log message (when-let (access-logger (server-access-logger *server*)) The motivation for this hack being that the CGI script wants to send the content-type header, so we can't finish the headers here when running a CGI script. Then I can do: (let* ((tbnl::*cgi-hack* t) (stream (flexi-streams:make-flexi-stream (tbnl:send-headers) :external-format tbnl::+latin-1+))) (sb-ext::run-program path nil :output stream :environment env)) and this sort of works, but, 1) it's kind of a hack and 2) it either exacerbates the timeout situation or there's some other problem where the first request is handled promptly, but subsequent requests to 10 seconds or so while something times out. Not sure why this would be. any thoughts on a good way to handle this? thanks, Cyrus
So... any input/suggestions on how to hack start-output to deal with CGI-style content? thanks, Cyrus On Jul 11, 2008, at 12:20 AM, Cyrus Harmon wrote:
So... I'm trying to convert my hunchentoot-cgi over to the new hunchentoot and running into some problems. My old approach was to do:
(progn (let* ((return-code (tbnl::return-code)) (reason-phrase (reason-phrase return-code)) (first-line (format nil "HTTP/1.1 ~D ~A" return-code reason- phrase))) (write-sequence (map 'list #'char-code first-line) out) (write-sequence tbnl::+crlf+ out) (tbnl::maybe-write-to-header-stream first-line)) (setf tbnl::*headers-sent* t) (setf (tbnl::content-type) nil) (sb-ext::run-program path nil :output out :environment env) nil)
but this doesn't seem to work anymore. My new approach is to try to use send-headers, but with some hackery to make it optionally more cgi-friendly: Index: headers.lisp =================================================================== --- headers.lisp (revision 3423) +++ headers.lisp (working copy) @@ -70,6 +70,8 @@ (:method (key value) (write-header-line key (princ-to-string value)))) +(defparameter *cgi-hack* nil) + (defun start-output (&key (content nil content-provided-p) (request *request*)) "Sends all headers and maybe the content body to @@ -84,7 +86,8 @@ ;; Read post data to clear stream - Force binary mode to avoid OCTETS-TO-STRING overhead. (raw-post-data :force-binary t) (let* ((return-code (return-code)) - (chunkedp (and (server-output-chunking-p *server*) + (chunkedp (and (not *cgi-hack*) + (server-output-chunking-p *server*) (eq (server-protocol request) :http/1.1) ;; only turn chunking on if the content ;; length is unknown at this point... @@ -196,12 +199,14 @@ ;; write all headers from the REPLY object (loop for (key . value) in (headers-out) when value - do (write-header-line (as-capitalized-string key) value)) + do (unless (and *cgi-hack* (equal key :content-type)) + (write-header-line (as-capitalized-string key) value))) ;; now the cookies (loop for (nil . cookie) in (cookies-out) do (write-header-line "Set-Cookie" (stringify-cookie cookie))) ;; all headers sent - (write-sequence +crlf+ *hunchentoot-stream*) + (unless *cgi-hack* + (write-sequence +crlf+ *hunchentoot-stream*)) (maybe-write-to-header-stream "") ;; access log message (when-let (access-logger (server-access-logger *server*))
The motivation for this hack being that the CGI script wants to send the content-type header, so we can't finish the headers here when running a CGI script.
Then I can do:
(let* ((tbnl::*cgi-hack* t) (stream (flexi- streams:make-flexi-stream (tbnl:send-headers) :external-format tbnl::+latin-1+))) (sb-ext::run-program path nil :output stream :environment env))
and this sort of works, but, 1) it's kind of a hack and 2) it either exacerbates the timeout situation or there's some other problem where the first request is handled promptly, but subsequent requests to 10 seconds or so while something times out. Not sure why this would be.
any thoughts on a good way to handle this?
thanks,
Cyrus
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
On Sat, 19 Jul 2008 13:40:20 -0700, Cyrus Harmon <ch-tbnl@bobobeach.com> wrote:
So... any input/suggestions on how to hack start-output to deal with CGI-style content?
Hi Cyrus, Sorry, too busy right now to give a good reply. Otherwise I would have done so already. My very general answer would be that I'd be very happy if we could find a clean (for some value of "clean) solution which doesn't involve "hacks" and/or :: access. More later (I don't know when, though), Edi.
well, one easy thing to do is to rename *cgi-hack* to *start-output- send-content-type-and-crlf*, and export it. That would remove the "hack" name and the :: access :) Cyrus On Jul 19, 2008, at 2:17 PM, Edi Weitz wrote:
On Sat, 19 Jul 2008 13:40:20 -0700, Cyrus Harmon <ch-tbnl@bobobeach.com
wrote:
So... any input/suggestions on how to hack start-output to deal with CGI-style content?
Hi Cyrus,
Sorry, too busy right now to give a good reply. Otherwise I would have done so already.
My very general answer would be that I'd be very happy if we could find a clean (for some value of "clean) solution which doesn't involve "hacks" and/or :: access.
More later (I don't know when, though), Edi. _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
On Mon, Jul 21, 2008 at 06:40, Cyrus Harmon <ch-tbnl@bobobeach.com> wrote:
well, one easy thing to do is to rename *cgi-hack* to *start-output-send-content-type-and-crlf*, and export it. That would remove the "hack" name and the :: access :)
Why not make the flag to suppress sending the content type and terminating the header be an argument to START-OUTPUT? I don't think that adding more special variables for no good reason would be so nice. It may just be me, though. -Hans
On Mon, 21 Jul 2008 07:54:51 +0200, "Hans Hübner" <hans@huebner.org> wrote:
Why not make the flag to suppress sending the content type and terminating the header be an argument to START-OUTPUT? I don't think that adding more special variables for no good reason would be so nice. It may just be me, though.
Fine with me. What's so special about the content type, though? Do you actually want /any/ headers to be sent or any processing to be done?
Falling squarely in the more than one way to skin a cat, perhaps it's easier to just read off the CGI headers and DTRT WRT to those headers and then send the output of the CGI to the tbnl output-stream. (let* ((process (sb-ext::run-program path nil :output :stream :environment env)) (in (sb-ext:process-output process))) (let ((headers (loop for line = (chunga:read-line* in) until (equal line "") collect (destructuring-bind (key val) (ppcre:split ": " line) (cons (chunga:as-keyword key) val))))) (let ((type-cons (assoc :content-type headers))) (when type-cons (setf (tbnl:content-type) (cdr type-cons))))) (let ((out (flexi-streams:make-flexi-stream (tbnl:send-headers) :external-format tbnl::+latin-1+))) (do ((c (read-char in) (read-char in))) ((eq c 'eof)) (write-char c out)))) This works well enough for the git CGI interface to sit behind (an unhacked) hunchentoot and hunchentoot-cgi as can be seen here: http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=hunchentoot-cgi.git thanks for prodding me to think about this some more... cyrus On Jul 20, 2008, at 11:44 PM, Edi Weitz wrote:
On Mon, 21 Jul 2008 07:54:51 +0200, "Hans Hübner" <hans@huebner.org> wrote:
Why not make the flag to suppress sending the content type and terminating the header be an argument to START-OUTPUT? I don't think that adding more special variables for no good reason would be so nice. It may just be me, though.
Fine with me.
What's so special about the content type, though? Do you actually want /any/ headers to be sent or any processing to be done? _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
Cyrus, in the code sample below, is it your intention to discard all headers sent by the CGI program? If not, I'd suggest that you just add them all to the list of outgoing headers. Also, you may want to use a COPY-STREAM function (like http://bknr.net/trac/browser/trunk/bknr/datastore/src/utils/utils.lisp#L189) instead of copying the data character wise for efficiency and readability. (let* ((process (sb-ext::run-program path nil :output :stream :environment env)) (in (sb-ext:process-output process))) (loop for line = (chung:read-line* in) until (equal line "") do (destructuring-bind (key val) (ppcre:split ":\\s*" line) (setf (hunchentoot:header-out key) val))) (copy-stream in (flexi-streams:make-flexi-stream (tbnl:send-headers) :external-format tbnl::+latin-1+) :element-type 'character)) -Hans On Tue, Jul 22, 2008 at 04:04, Cyrus Harmon <ch-tbnl@bobobeach.com> wrote:
Falling squarely in the more than one way to skin a cat, perhaps it's easier to just read off the CGI headers and DTRT WRT to those headers and then send the output of the CGI to the tbnl output-stream.
(let* ((process (sb-ext::run-program path nil :output :stream :environment env)) (in (sb-ext:process-output process))) (let ((headers (loop for line = (chunga:read-line* in) until (equal line "") collect (destructuring-bind (key val) (ppcre:split ": " line) (cons (chunga:as-keyword key) val))))) (let ((type-cons (assoc :content-type headers))) (when type-cons (setf (tbnl:content-type) (cdr type-cons))))) (let ((out (flexi-streams:make-flexi-stream (tbnl:send-headers) :external-format tbnl::+latin-1+))) (do ((c (read-char in) (read-char in))) ((eq c 'eof)) (write-char c out))))
This works well enough for the git CGI interface to sit behind (an unhacked) hunchentoot and hunchentoot-cgi as can be seen here:
http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=hunchentoot-cgi.git
thanks for prodding me to think about this some more...
cyrus
On Jul 20, 2008, at 11:44 PM, Edi Weitz wrote:
On Mon, 21 Jul 2008 07:54:51 +0200, "Hans Hübner" <hans@huebner.org> wrote:
Why not make the flag to suppress sending the content type and terminating the header be an argument to START-OUTPUT? I don't think that adding more special variables for no good reason would be so nice. It may just be me, though.
Fine with me.
What's so special about the content type, though? Do you actually want /any/ headers to be sent or any processing to be done? _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
Hans, Yes, my original code was a "proof of concept" that I didn't need to hack up send-output to make this is work. Your version is much better. thanks! Cyrus On Jul 21, 2008, at 10:33 PM, Hans Hübner wrote:
Cyrus,
in the code sample below, is it your intention to discard all headers sent by the CGI program? If not, I'd suggest that you just add them all to the list of outgoing headers. Also, you may want to use a COPY-STREAM function (like http://bknr.net/trac/browser/trunk/bknr/datastore/src/utils/utils.lisp#L189) instead of copying the data character wise for efficiency and readability.
(let* ((process (sb-ext::run-program path nil :output :stream :environment env)) (in (sb-ext:process-output process))) (loop for line = (chung:read-line* in) until (equal line "") do (destructuring-bind (key val) (ppcre:split ":\\s*" line) (setf (hunchentoot:header-out key) val))) (copy-stream in (flexi-streams:make-flexi-stream (tbnl:send-headers) :external-format tbnl::+latin-1+) :element-type 'character))
-Hans
On Tue, Jul 22, 2008 at 04:04, Cyrus Harmon <ch-tbnl@bobobeach.com> wrote:
Falling squarely in the more than one way to skin a cat, perhaps it's easier to just read off the CGI headers and DTRT WRT to those headers and then send the output of the CGI to the tbnl output-stream.
(let* ((process (sb-ext::run-program path nil :output :stream :environment env)) (in (sb-ext:process-output process))) (let ((headers (loop for line = (chunga:read-line* in) until (equal line "") collect (destructuring-bind (key val) (ppcre:split ": " line) (cons (chunga:as-keyword key) val))))) (let ((type-cons (assoc :content-type headers))) (when type-cons (setf (tbnl:content-type) (cdr type-cons))))) (let ((out (flexi-streams:make-flexi-stream (tbnl:send-headers) :external-format tbnl::+latin-1+))) (do ((c (read-char in) (read-char in))) ((eq c 'eof)) (write-char c out))))
This works well enough for the git CGI interface to sit behind (an unhacked) hunchentoot and hunchentoot-cgi as can be seen here:
http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=hunchentoot-cgi.git
thanks for prodding me to think about this some more...
cyrus
On Jul 20, 2008, at 11:44 PM, Edi Weitz wrote:
On Mon, 21 Jul 2008 07:54:51 +0200, "Hans Hübner" <hans@huebner.org> wrote:
Why not make the flag to suppress sending the content type and terminating the header be an argument to START-OUTPUT? I don't think that adding more special variables for no good reason would be so nice. It may just be me, though.
Fine with me.
What's so special about the content type, though? Do you actually want /any/ headers to be sent or any processing to be done? _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
participants (3)
-
Cyrus Harmon -
Edi Weitz -
Hans Hübner