- tbnl-devel@common-lisp.net - mailman3.common-lisp.net

[tbnl-devel] New version 0.9.0
by Edi Weitz 24 Dec '05

24 Dec '05

ChangeLog: Version 0.9.0 2005-12-24 Experimental support for writing directly to the front-end (see SEND-HEADERS) Added HANDLE-STATIC-FILE Changed CREATE-STATIC-FILE-DISPATCHER-AND-HANDLER to use new facilities Added CREATE-FOLDER-DISPATCHER-AND-HANDLER Added link to Travis Cross' message w.r.t. SBCL Download: <http://weitz.de/files/tbnl.tar.gz> Happy Holidays, Edi.

2 1

[tbnl-devel] fixed: tbnl / sbcl binary upload / download issues
by Travis Cross 20 Dec '05

20 Dec '05

TBNL has played poorly with SBCL up to this point with regard to binary data. Uploading binary files never worked at all, and only 1/2 of the binary download examples appeared to work out of the box. As it turns out, these problems are not related to code in current versions of TBNL itself, but in TBNL's dependencies: KMRCL and rfc2388. Attached are the necessary patches. *Changes* -- kmrcl: Prior to this patch, kmrcl opened streams in SBCL with :element-type 'base-char. This effectively prevented streams from writing character values above 127 or operating in a bivalent fashion. SBCL will open bivalent streams when :element-type :default is specified. -- rfc2388 rfc2388 opens a file stream to save attachments. If the :external-format option is not given to OPEN, SBCL will default to :ascii and only write characters up to 127. The :external-format :latin-1 option needs to be passed to OPEN. *Testing* The "Binary data, delivered from file", "Binary data, delivered from RAM", and "File Upload" examples in the tbnl-test package all work correctly now with SBCL-0.9.7, both when using TBNL directly (without a front-end) and with mod_lisp2 / Apache2. For binary upload testing, I uploaded a 165MB video. I then downloaded the video through the tbnl interface (with Firefox) and ensured that the hashes of the two files matched. I was a bit surprised, actually, that TBNL / SBCL didn't choke on the size of the file ;) To make sure these changes were production-ready, I ran ApacheBench2 against the binary image downloads: /usr/sbin/ab2 -c 50 -n 1000 http://server/tbnl/test/image-ram.jpg Complete requests: 1000 Failed requests: 0 Requests per second: 343.94 [#/sec] (mean) Time per request: 145.373 [ms] (mean) /usr/sbin/ab2 -c 50 -n 1000 http://server/tbnl/test/image.jpg Complete requests: 1000 Failed requests: 0 Requests per second: 237.63 [#/sec] (mean) Time per request: 210.412 [ms] (mean) As far as I can tell, these changes make SBCL a fully supported platform for TBNL. Cheers, -- Travis PS - I should note a few things here for reference: -- :tbnl-bivalent-streams needs to be enabled in *features*. -- The unicode tests in the TBNL suite appear to work correctly. -- Contrary to our prior expectations, the underlying issue was not related to handling sb-unicode appropriately, and in fact, binary handling does not seem to rely on sb-unicode at all. diff -u kmrcl-1.84/processes.lisp kmrcl/processes.lisp --- kmrcl-1.84/processes.lisp 2005-11-03 12:43:52.000000000 -0500 +++ kmrcl/processes.lisp 2005-12-02 19:33:15.000000000 -0500 @@ -17,7 +17,7 @@ #+allegro (mp:process-run-function name func) #+cmu (mp:make-process func :name name) #+lispworks (mp:process-run-function name nil func) - #+sb-thread (sb-thread:make-thread func) + #+sb-thread (sb-thread:make-thread func :name name) #+openmcl (ccl:process-run-function name func) #-(or allegro cmu lispworks sb-thread openmcl) (funcall func) ) diff -u kmrcl-1.84/sockets.lisp kmrcl/sockets.lisp --- kmrcl-1.84/sockets.lisp 2005-11-03 12:43:52.000000000 -0500 +++ kmrcl/sockets.lisp 2005-12-19 20:20:27.000000000 -0500 @@ -84,7 +84,7 @@ (let ((sock (sb-bsd-sockets:socket-accept listener))) (values (sb-bsd-sockets:socket-make-stream - sock :element-type 'base-char :input t :output t) + sock :element-type :default :input t :output t) sock))) #+openmcl (let ((sock (ccl:accept-connection listener :wait t))) @@ -115,7 +115,8 @@ (defun close-active-socket (socket) - (close socket)) + #+sbcl (sb-bsd-sockets:socket-close socket) + #-sbcl (close socket)) (defun ipaddr-to-dotted (ipaddr &key values) "Convert from 32-bit integer to dotted string." @@ -178,7 +179,7 @@ (sb-bsd-sockets:socket-connect sock (lookup-hostname server) port) (values (sb-bsd-sockets:socket-make-stream - sock :input t :output t :element-type 'base-char) + sock :input t :output t :element-type :default) sock)) #+cmu (let ((sock (ext:connect-to-inet-socket server port))) diff -u rfc2388.orig/rfc2388.lisp rfc2388/rfc2388.lisp --- rfc2388.orig/rfc2388.lisp 2005-12-19 20:12:42.000000000 -0500 +++ rfc2388/rfc2388.lisp 2005-12-19 20:29:50.000000000 -0500 @@ -441,8 +441,9 @@ :direction :output ;; external format for faithful I/O ;; see <http://cl-cookbook.sourceforge.net/io.html#faith> - #+(or :lispworks :allegro) + #+(or :sbcl :lispworks :allegro) :external-format + #+:sbcl :latin-1 #+:lispworks '(:latin-1 :eol-style :lf) #+:allegro (excl:crlf-base-ef :latin1)) (read-until-next-boundary input boundary nil out-file))

2 1

[tbnl-devel] Fixing binary uploads in SBCL / TBNL / mod_lisp2
by Travis Cross 20 Dec '05

20 Dec '05

Running SBCL 0.9.6 with TBNL 0.8.4 and Apache 2.0.54 with mod_lisp2, the image download examples work as expected, but uploading any binary file results in a 500 Internal Server Error being throw by Apache. No error is reported in Lisp, but the Apache log file error_log shows the following entry: [error] (70014)End of file found: error reading from Lisp Setting apache to log in debug mode doesn't provide any additional insight, but some testing and modification of the mod_lisp2.c code showed that the error is starting at line 588 where mod_lisp attempts to read the first of the headers. This leads back to read_lisp_line(), then deeper to fill_input_buffer(), where the failure seems to originate at the call to: RELAY_ERROR (((length = (sizeof (buffer->data))), (apr_recv (socket, (buffer->data), (&length))))); With *DEBUG-MODE* set to t, *COMMAND* contains seemingly plausible headers (shown below). I am not familiar enough with the internals to have much of an idea of what is causing the breakdown, though the C code seems to suggest that lisp is sending a different amount of data than is promised. I'm happy to dig into either the TBNL, KMRCL, or mod_lisp2.c code and come up with a patch for this, but it would be tremendously helpful if someone has a good idea of where to start and what the problem might be. Cheers, -- Travis *COMMAND* after failed file upload: (("content-stream" . #<SB-SYS:FD-STREAM for "a constant string" {9CEF2D9}>) ("server-protocol" . "HTTP/1.1") ("method" . "POST") ("url" . "/tbnl/test/upload.html") ("content-type" . "multipart/form-data; boundary=---------------------------130786897319483305451172113725") ("content-length" . "281039") ("server-ip-addr" . "127.0.0.1") ("server-ip-port" . "80") ("remote-ip-addr" . "127.0.0.1") ("script-filename" . "/usr/htdocs") ("remote-ip-port" . "48073") ("server-id" . "tbnl") ("server-baseversion" . "Apache/2.0.54") ("modlisp-version" . "1.2") ("modlisp-major-version" . "2") ("Host" . "localhost") ("User-Agent" . "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050829 Firefox/1.0.6") ("Accept" . "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5") ("Accept-Language" . "en-us,en;q=0.5") ("Accept-Encoding" . "gzip,deflate") ("Accept-Charset" . "ISO-8859-1,utf-8;q=0.7,*;q=0.7") ("Keep-Alive" . "300") ("Connection" . "keep-alive") ("Referer" . "http://localhost/tbnl/test/upload.html") ("Cookie" . "pumpkin=barking") ("Authorization" . "Basic bmFub29rOmlnbG9v") ("Content-Type" . "multipart/form-data; boundary=---------------------------130786897319483305451172113725") ("Content-Length" . "281039"))

2 2

[tbnl-devel] ajax / tbnl
by Tolstoy 16 Dec '05

16 Dec '05

Folks-- I'm working on an ajax kind of thing and when my html page loads, I attempt to init the page's data structures by firing off many XmlHttpRequests. When I fire off two of them, things work fine. (One populates the main data I want to display on the page, another populates a form with some drop down list.) When I add a third asynchronous request, things break. Alas, I'm not exactly sure how to explain it. The way I'm seeing it at the moment is that the Oracle OCI shared lib throws an exception ORA-21500 which takes down SBCL, I think, or at least slime. So, just to rule things out: there's no possible way this can be some problem with tbnl (0.8.9 on SBCL), right? Each of those asynchronous requests should be coming in on a separate thread and things should work accordingly. Right? ;) Surely it can't be the clsql oracle interface which can't handle simultaneous requests? Do I need to block on clsql requests? Anyway, curious to know if any of you have encountered this sort of problem before. Keith

2 2

[tbnl-devel] New version 0.8.9
by Edi Weitz 16 Dec '05

16 Dec '05

ChangeLog: Version 0.8.9 2005-12-16 Also use :TBNL-BIVALENT-STREAMS if :SB-UNICODE is present Download: <http://weitz.de/files/tbnl.tar.gz>

1 0

[tbnl-devel] sbcl 0.9.7, tbnl 0.8.7.0, serving images
by Tolstoy 14 Dec '05

14 Dec '05

Folks-- Is there still no workaround (aside from using apache) for serving images via tbnl standalone and sbcl? Here's the error I get: #<SB-SYS:FD-STREAM for "file /home/keith/projex/cl-marmoset/client/arrowdown.gif" {B2FAFA9}> (:EXTERNAL-FORMAT :UTF-8): the octet sequence (145) cannot be decoded. Does the newer TBNL fix this? Doesn't sbcl support bi-valent streams, now? Curious.... --T

3 3

[tbnl-devel] New version 0.8.8
by Edi Weitz 08 Dec '05

08 Dec '05

ChangeLog: Version 0.8.8 2005-12-08 Made RAW-POST-DATA more useful Updated docs w.r.t. Araneida (thanks to Alan Shields) Download: <http://weitz.de/files/tbnl.tar.gz> Cheers, Edi.

2 2

[tbnl-devel] Content length with multibyte character encodings
by Ignas Mikalajunas 29 Nov '05

29 Nov '05

Content length is calculated by calling (length content) which produces wrong results with unicode characters in the string. Piso on #lisp proposed a solution - using (length (string-to-octets string :external-format :utf-8)) which translates to just (length (string-to-octets string :external-format)) in the code. The true way to solve this would be using (file-string-length), but the function is not working properly on sbcl yet. So could you please fix the (send-output), because with current setup browsers that strictly adhere to the content-lenght (IE 6.0, Opera) would trim 1 character of the responses body for each UTF-8 character in it. Ignas

2 3

[tbnl-devel] New version 0.8.6
by Edi Weitz 18 Nov '05

18 Nov '05

ChangeLog: Version 0.8.6 2005-11-18 Restored original stream-based code for multipart/form-data parsing (got lost somehow) Wrap REMOTE-ADDR with IGNORE-ERRORS (just in case) Download: <http://weitz.de/files/tbnl.tar.gz> Cheers, Edi.

1 0

[tbnl-devel] tbnl patch
by Jeff Caldwell 14 Nov '05

14 Nov '05

Hello, Here is a tiny patch for tbnl. The motivation is that I'm working with blocks of dispatch table entries that are added and withdrawn as modules without restarting tbnl. I'd also like reflective access to the dispatch table e.g. to show a web page of installed dispatch handlers. To affect the base distrubution as little as possible, I took the loop through the dispatch table that funcalls each entry and moved that loop into a defmethod. Now I can specialize on the method and have my own dispatch mechanisms. Jeff Caldwell __________________________________ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com

2 1