TBNL has played poorly with SBCL up to this point with regard to binary data. Uploading binary files never worked at all, and only 1/2 of the binary download examples appeared to work out of the box.
As it turns out, these problems are not related to code in current versions of TBNL itself, but in TBNL's dependencies: KMRCL and rfc2388.
Attached are the necessary patches.
*Changes*
-- kmrcl: Prior to this patch, kmrcl opened streams in SBCL with :element-type 'base-char. This effectively prevented streams from writing character values above 127 or operating in a bivalent fashion. SBCL will open bivalent streams when :element-type :default is specified.
-- rfc2388 rfc2388 opens a file stream to save attachments. If the :external-format option is not given to OPEN, SBCL will default to :ascii and only write characters up to 127. The :external-format :latin-1 option needs to be passed to OPEN.
*Testing*
The "Binary data, delivered from file", "Binary data, delivered from RAM", and "File Upload" examples in the tbnl-test package all work correctly now with SBCL-0.9.7, both when using TBNL directly (without a front-end) and with mod_lisp2 / Apache2.
For binary upload testing, I uploaded a 165MB video. I then downloaded the video through the tbnl interface (with Firefox) and ensured that the hashes of the two files matched. I was a bit surprised, actually, that TBNL / SBCL didn't choke on the size of the file ;)
To make sure these changes were production-ready, I ran ApacheBench2 against the binary image downloads:
/usr/sbin/ab2 -c 50 -n 1000 http://server/tbnl/test/image-ram.jpg Complete requests: 1000 Failed requests: 0 Requests per second: 343.94 [#/sec] (mean) Time per request: 145.373 [ms] (mean)
/usr/sbin/ab2 -c 50 -n 1000 http://server/tbnl/test/image.jpg Complete requests: 1000 Failed requests: 0 Requests per second: 237.63 [#/sec] (mean) Time per request: 210.412 [ms] (mean)
As far as I can tell, these changes make SBCL a fully supported platform for TBNL.
Cheers,
-- Travis
PS - I should note a few things here for reference: -- :tbnl-bivalent-streams needs to be enabled in *features*. -- The unicode tests in the TBNL suite appear to work correctly. -- Contrary to our prior expectations, the underlying issue was not related to handling sb-unicode appropriately, and in fact, binary handling does not seem to rely on sb-unicode at all.
diff -u kmrcl-1.84/processes.lisp kmrcl/processes.lisp --- kmrcl-1.84/processes.lisp 2005-11-03 12:43:52.000000000 -0500 +++ kmrcl/processes.lisp 2005-12-02 19:33:15.000000000 -0500 @@ -17,7 +17,7 @@ #+allegro (mp:process-run-function name func) #+cmu (mp:make-process func :name name) #+lispworks (mp:process-run-function name nil func) - #+sb-thread (sb-thread:make-thread func) + #+sb-thread (sb-thread:make-thread func :name name) #+openmcl (ccl:process-run-function name func) #-(or allegro cmu lispworks sb-thread openmcl) (funcall func) ) diff -u kmrcl-1.84/sockets.lisp kmrcl/sockets.lisp --- kmrcl-1.84/sockets.lisp 2005-11-03 12:43:52.000000000 -0500 +++ kmrcl/sockets.lisp 2005-12-19 20:20:27.000000000 -0500 @@ -84,7 +84,7 @@ (let ((sock (sb-bsd-sockets:socket-accept listener))) (values (sb-bsd-sockets:socket-make-stream - sock :element-type 'base-char :input t :output t) + sock :element-type :default :input t :output t) sock))) #+openmcl (let ((sock (ccl:accept-connection listener :wait t))) @@ -115,7 +115,8 @@
(defun close-active-socket (socket) - (close socket)) + #+sbcl (sb-bsd-sockets:socket-close socket) + #-sbcl (close socket))
(defun ipaddr-to-dotted (ipaddr &key values) "Convert from 32-bit integer to dotted string." @@ -178,7 +179,7 @@ (sb-bsd-sockets:socket-connect sock (lookup-hostname server) port) (values (sb-bsd-sockets:socket-make-stream - sock :input t :output t :element-type 'base-char) + sock :input t :output t :element-type :default) sock)) #+cmu (let ((sock (ext:connect-to-inet-socket server port)))
diff -u rfc2388.orig/rfc2388.lisp rfc2388/rfc2388.lisp --- rfc2388.orig/rfc2388.lisp 2005-12-19 20:12:42.000000000 -0500 +++ rfc2388/rfc2388.lisp 2005-12-19 20:29:50.000000000 -0500 @@ -441,8 +441,9 @@ :direction :output ;; external format for faithful I/O ;; see http://cl-cookbook.sourceforge.net/io.html#faith - #+(or :lispworks :allegro) + #+(or :sbcl :lispworks :allegro) :external-format + #+:sbcl :latin-1 #+:lispworks '(:latin-1 :eol-style :lf) #+:allegro (excl:crlf-base-ef :latin1)) (read-until-next-boundary input boundary nil out-file))