Hi
I have a fingerprint scanning application, that uploads fingerprint images using Drakma to a Hunchentoot server. Everything works great as long as I don't use a secure connection. With https, I get a CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE condition on the server, and "Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM >" on the client. Sometimes the uploads are successful with https, though.
The server runs LispWorks 5.0.2 on Debian Etch. The client runs on LispWorks 5.0.2 on Windows XP SP2. Part of the application pulls in the trivial-ldap module which depends on cl+ssl - not sure if this can be an issue as cl+ssl is not needed on LispWorks for Drakma and Hunchentoot.
I am sure I am doing something stupid. The handler that does the actual upload on the client (which also uses Hunchentoot) is: (define-easy-handler (upload-fp-image.action :uri "/hunchentoot/patient-study-db/upload-fp-image.action") () (let ((cookie-jar (make-instance 'drakma:cookie-jar :cookies (list (make-instance 'drakma:cookie :name (session-value :hs-name) :value (session-value :hs-id) :path (session-value :hs-path) :domain (session-value :hs-domain)))))) (handler-case (let ((html-result (drakma:http-request (session-value :upload-url) :method :post :parameters (list (cons "finger" (session-value :finger)) (cons "img-file" (pathname *scanned-file*)) (cons "hs-id" (session-value :hs-id))) :user-agent (session-value :user-agent) :cookie-jar cookie-jar))) (delete-file *scanned-file*) (with-html-output-to-string (*standard-output*) (str html-result))) (condition (con) (delete-file *scanned-file*) (show-condition-html con (session-value :back-caption) (session-value :back-url))))))
The Hunchentoot server log spits out: [2007-12-12 11:35:39 [ERROR]] While parsing multipart/form-data parameters: The condition #<CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE 200B000C> occurred [2007-12-12 11:35:39] 10.38.1.45 - "POST /hunchentoot/patient-study-db/identify-patient.action HTTP/1.1" 200 1058 "-" "Opera/9.10 (Windows NT 5.1; U; en)" [2007-12-12 11:35:39 [ERROR]] Error while processing connection: Not all chunks from #<CHUNGA:CHUNKED-IO-STREAM 223A7893> have been read completely.
Backtraces on the server and client are included below.
Thanks. Nico
Backtrace on server: Error: The condition #<CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE 223AB36C> occurred 1 (abort) Quit process.
Type :b for backtrace, :c <option number> to proceed, or :? for other options
CL-USER 1 : 1 > :b Call to SIGNAL Call to (METHOD CHUNGA::FILL-BUFFER (CHUNGA:CHUNKED-INPUT-STREAM)) Call to (METHOD STREAM:STREAM-READ-BYTE (CHUNGA:CHUNKED-INPUT-STREAM)) Call to READ-BYTE Call to (METHOD FLEXI-STREAMS::READ-BYTE* (FLEXI-STREAMS::FLEXI-BINARY-INPUT-STREAM)) Call to (METHOD STREAM:STREAM-READ-CHAR (FLEXI-STREAMS::FLEXI-LATIN-1-INPUT-STREAM)) Call to READ-CHAR Call to (SUBFUNCTION RFC2388::RUN RFC2388::READ-UNTIL-NEXT-BOUNDARY) Call to RFC2388::READ-UNTIL-NEXT-BOUNDARY Call to (METHOD RFC2388:PARSE-MIME (STREAM T)) Call to HUNCHENTOOT::PARSE-RFC2388-FORM-DATA Call to (METHOD INITIALIZE-INSTANCE :AFTER (HUNCHENTOOT::REQUEST)) Call to CLOS::METHOD-COMBINATION-TEMPLATE Call to CLOS::MAKE-INSTANCE-FROM-CLASS-1 Call to HUNCHENTOOT::PROCESS-REQUEST Call to HUNCHENTOOT::PROCESS-CONNECTION Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK)
Backtrace on client: Error: Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM 2009C0E7>. 1 (abort) Quit process.
Type :b for backtrace, :c <option number> to proceed, or :? for other options
CL-USER 1 : 1 > :b Call to SIGNAL Call to (METHOD STREAM:STREAM-FLUSH-BUFFER (STREAM:BUFFERED-STREAM)) Call to (METHOD STREAM:STREAM-WRITE-SEQUENCE (STREAM:BUFFERED-STREAM T T T)) Call to CLOS::METHOD-COMBINATION-TEMPLATE Call to WRITE-SEQUENCE Call to WRITE-SEQUENCE Call to (METHOD CHUNGA::WRITE-CHUNK (CHUNGA:CHUNKED-OUTPUT-STREAM T)) Call to (METHOD CHUNGA::FLUSH-BUFFER (CHUNGA:CHUNKED-OUTPUT-STREAM)) Call to (METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (CHUNGA:CHUNKED-OUTPUT-STREAM T T T)) Call to WRITE-SEQUENCE Call to WRITE-SEQUENCE Call to DRAKMA::SEND-CONTENT Call to DRAKMA::SEND-CONTENT Call to (SUBFUNCTION 1 DRAKMA::MAKE-FORM-DATA-FUNCTION) Call to (SUBFUNCTION (LABELS DRAKMA::FINISH-REQUEST) DRAKMA:HTTP-REQUEST) Call to DRAKMA:HTTP-REQUEST Interpreted call to PATIENT-STUDY-DB-CLIENT::UPLOAD-FP-IMAGE.ACTION Call to HUNCHENTOOT::PROCESS-REQUEST Call to HUNCHENTOOT::PROCESS-CONNECTION Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK)
Nico de Jager ndj@hivsa.com writes:
Hi
I have a fingerprint scanning application, that uploads fingerprint images using Drakma to a Hunchentoot server. Everything works great as long as I don't use a secure connection. With https, I get a CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE condition on the server, and "Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM >" on the client. Sometimes the uploads are successful with https, though.
The server runs LispWorks 5.0.2 on Debian Etch. The client runs on LispWorks 5.0.2 on Windows XP SP2. Part of the application pulls in the trivial-ldap module which depends on cl+ssl - not sure if this can be an issue as cl+ssl is not needed on LispWorks for Drakma and Hunchentoot.
Trying to isolate the problem, I set up a general upload easy handlers in a separate application. All https uploads work fine from Linux clients but not from Windows, so I guess it is my version of OpenSSL on the Windows box. I use a binary version which I downloaded here (Win32 OpenSSL v0.9.8g): http://www.slproweb.com/products/Win32OpenSSL.html. I followed this link from the FAQ on www.openssl.org.
Does anyone have advice on easily getting a reliable/working copy of OpenSSL on Windows (preferably a binary package), before I compile my own? Or is it something else?
Thanks. Nico
I am sure I am doing something stupid. The handler that does the actual upload on the client (which also uses Hunchentoot) is: (define-easy-handler (upload-fp-image.action :uri "/hunchentoot/patient-study-db/upload-fp-image.action") () (let ((cookie-jar (make-instance 'drakma:cookie-jar :cookies (list (make-instance 'drakma:cookie :name (session-value :hs-name) :value (session-value :hs-id) :path (session-value :hs-path) :domain (session-value :hs-domain)))))) (handler-case (let ((html-result (drakma:http-request (session-value :upload-url) :method :post :parameters (list (cons "finger" (session-value :finger)) (cons "img-file" (pathname *scanned-file*)) (cons "hs-id" (session-value :hs-id))) :user-agent (session-value :user-agent) :cookie-jar cookie-jar))) (delete-file *scanned-file*) (with-html-output-to-string (*standard-output*) (str html-result))) (condition (con) (delete-file *scanned-file*) (show-condition-html con (session-value :back-caption) (session-value :back-url))))))
The Hunchentoot server log spits out: [2007-12-12 11:35:39 [ERROR]] While parsing multipart/form-data parameters: The condition #<CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE 200B000C> occurred [2007-12-12 11:35:39] 10.38.1.45 - "POST /hunchentoot/patient-study-db/identify-patient.action HTTP/1.1" 200 1058 "-" "Opera/9.10 (Windows NT 5.1; U; en)" [2007-12-12 11:35:39 [ERROR]] Error while processing connection: Not all chunks from #<CHUNGA:CHUNKED-IO-STREAM 223A7893> have been read completely.
Backtraces on the server and client are included below.
Thanks. Nico
Backtrace on server: Error: The condition #<CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE 223AB36C> occurred 1 (abort) Quit process.
Type :b for backtrace, :c <option number> to proceed, or :? for other options
CL-USER 1 : 1 > :b Call to SIGNAL Call to (METHOD CHUNGA::FILL-BUFFER (CHUNGA:CHUNKED-INPUT-STREAM)) Call to (METHOD STREAM:STREAM-READ-BYTE (CHUNGA:CHUNKED-INPUT-STREAM)) Call to READ-BYTE Call to (METHOD FLEXI-STREAMS::READ-BYTE* (FLEXI-STREAMS::FLEXI-BINARY-INPUT-STREAM)) Call to (METHOD STREAM:STREAM-READ-CHAR (FLEXI-STREAMS::FLEXI-LATIN-1-INPUT-STREAM)) Call to READ-CHAR Call to (SUBFUNCTION RFC2388::RUN RFC2388::READ-UNTIL-NEXT-BOUNDARY) Call to RFC2388::READ-UNTIL-NEXT-BOUNDARY Call to (METHOD RFC2388:PARSE-MIME (STREAM T)) Call to HUNCHENTOOT::PARSE-RFC2388-FORM-DATA Call to (METHOD INITIALIZE-INSTANCE :AFTER (HUNCHENTOOT::REQUEST)) Call to CLOS::METHOD-COMBINATION-TEMPLATE Call to CLOS::MAKE-INSTANCE-FROM-CLASS-1 Call to HUNCHENTOOT::PROCESS-REQUEST Call to HUNCHENTOOT::PROCESS-CONNECTION Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK)
Backtrace on client: Error: Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM 2009C0E7>. 1 (abort) Quit process.
Type :b for backtrace, :c <option number> to proceed, or :? for other options
CL-USER 1 : 1 > :b Call to SIGNAL Call to (METHOD STREAM:STREAM-FLUSH-BUFFER (STREAM:BUFFERED-STREAM)) Call to (METHOD STREAM:STREAM-WRITE-SEQUENCE (STREAM:BUFFERED-STREAM T T T)) Call to CLOS::METHOD-COMBINATION-TEMPLATE Call to WRITE-SEQUENCE Call to WRITE-SEQUENCE Call to (METHOD CHUNGA::WRITE-CHUNK (CHUNGA:CHUNKED-OUTPUT-STREAM T)) Call to (METHOD CHUNGA::FLUSH-BUFFER (CHUNGA:CHUNKED-OUTPUT-STREAM)) Call to (METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (CHUNGA:CHUNKED-OUTPUT-STREAM T T T)) Call to WRITE-SEQUENCE Call to WRITE-SEQUENCE Call to DRAKMA::SEND-CONTENT Call to DRAKMA::SEND-CONTENT Call to (SUBFUNCTION 1 DRAKMA::MAKE-FORM-DATA-FUNCTION) Call to (SUBFUNCTION (LABELS DRAKMA::FINISH-REQUEST) DRAKMA:HTTP-REQUEST) Call to DRAKMA:HTTP-REQUEST Interpreted call to PATIENT-STUDY-DB-CLIENT::UPLOAD-FP-IMAGE.ACTION Call to HUNCHENTOOT::PROCESS-REQUEST Call to HUNCHENTOOT::PROCESS-CONNECTION Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK) _______________________________________________ drakma-devel mailing list drakma-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel
On Wed, 12 Dec 2007 15:59:20 +0200, Nico de Jager ndj@hivsa.com wrote:
Trying to isolate the problem, I set up a general upload easy handlers in a separate application. All https uploads work fine from Linux clients but not from Windows, so I guess it is my version of OpenSSL on the Windows box. I use a binary version which I downloaded here (Win32 OpenSSL v0.9.8g): http://www.slproweb.com/products/Win32OpenSSL.html. I followed this link from the FAQ on www.openssl.org.
Does anyone have advice on easily getting a reliable/working copy of OpenSSL on Windows (preferably a binary package), before I compile my own? Or is it something else?
I use the one you also have. They are the ones recommended by LispWorks:
http://www.lispworks.com/documentation/lw50/LWUG/html/lwuser-344.htm
On Wed, 12 Dec 2007 13:34:14 +0200, Nico de Jager ndj@hivsa.com wrote:
I have a fingerprint scanning application, that uploads fingerprint images using Drakma to a Hunchentoot server. Everything works great as long as I don't use a secure connection. With https, I get a CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE condition on the server, and "Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM >" on the client. Sometimes the uploads are successful with https, though.
It's hard to say something about this from here. Are you able to generate a small, self-contained example that exposes the problem? What did you do to isolate the error? Does it work without the trivial-ldap module being loaded? Does it work with another Lisp implementation on the client or on the server side? Does it work if you use :CONTENT-LENGTH T in your Drakma request?
Edi.
Edi Weitz edi@agharta.de writes:
On Wed, 12 Dec 2007 13:34:14 +0200, Nico de Jager ndj@hivsa.com wrote:
I have a fingerprint scanning application, that uploads fingerprint images using Drakma to a Hunchentoot server. Everything works great as long as I don't use a secure connection. With https, I get a CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE condition on the server, and "Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM >" on the client. Sometimes the uploads are successful with https, though.
Sorry for only replying now, I was away. Let me know if need to repost my original message.
It's hard to say something about this from here. Are you able to generate a small, self-contained example that exposes the problem?
Yes, see the attached program. I also included a couple of images that fails _most_ of the time.
What did you do to isolate the error?
Uhm, the attached example simply tries to upload a few image files (evaluate the function test-upload). *Important*: From Linux to XP/Linux all uploads work for both secure and non-secure attempts. From XP only non-secure uploads work. Secure uploads from XP will work about 5% of the time. I even tried it on different machines.
Does it work without the trivial-ldap module being loaded?
No, I removed trivial-ldap completely in the example.
Does it work with another Lisp implementation on the client or on the server side?
Uploads from LispWorks (Win32) to LispWorks or SBCL on Linux fails. LispWorks (Win32) to LispWorks on Win32 fails. SBCL (Linux) to Win32 LispWorks works. SBCL (Linux) to LispWorks and SBCL on Linux works. LispWorks (Linux) to LispWorks and SBCL on Linux works. I don't have another CL for Win32. LispWorks 5.0.2 was used in all instances.
Does it work if you use :CONTENT-LENGTH T in your Drakma request?
No, it does not seem to make a difference.
Thanks in advance. Nico
On Wed, 09 Jan 2008 14:57:37 +0200, Nico de Jager ndj@hivsa.com wrote:
Uhm, the attached example simply tries to upload a few image files
I could reproduce the example and it seems this is a bug in LispWorks (which I've reported to LW support). The new release doesn't really fix the problem (I can't do that), but it tries to deal with it in a more or less reasonable way. Please try it.
Note that the new release is only relevant for LispWorks users. Users of other Lisps can as well keep 0.11.1.
Thanks for the report, Edi.
Edi Weitz edi@agharta.de writes:
On Wed, 09 Jan 2008 14:57:37 +0200, Nico de Jager ndj@hivsa.com wrote: Note that the new release is only relevant for LispWorks users. Users of other Lisps can as well keep 0.11.1.
Many thanks, Edi. But it appears that you forgot to upload the new version to your site (no, that's impossible - maybe you just decided to take a break at 3am and did not do it yet :-). Both asdf-install and a manual download still gives me version 0.11.1 (Just the change log on your site was updated).
Thanks. Nico
On Mon, 14 Jan 2008 11:29:50 +0200, Nico de Jager ndj@hivsa.com wrote:
But it appears that you forgot to upload the new version to your site (no, that's impossible - maybe you just decided to take a break at 3am and did not do it yet :-). Both asdf-install and a manual download still gives me version 0.11.1
That's strange - I get the new one. See below.
(Just the change log on your site was updated).
That's actually a good sign that the new tarball is online. The change log is automatically unpacked from the tarball and not uploaded separately.
Edi.
edi@groucho:/tmp$ wget http://weitz.de/files/drakma.tar.gz --17:45:21-- http://weitz.de/files/drakma.tar.gz => `drakma.tar.gz' Resolving weitz.de... 81.169.156.108 Connecting to weitz.de|81.169.156.108|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 42,464 (41K) [application/x-tar]
100%[=============================================================================================================>] 42,464 --.--K/s
17:45:21 (344.32 KB/s) - `drakma.tar.gz' saved [42464/42464]
edi@groucho:/tmp$ tar vxzf drakma.tar.gz drakma-0.11.2/ drakma-0.11.2/CHANGELOG.txt drakma-0.11.2/cookies.lisp drakma-0.11.2/doc/ drakma-0.11.2/doc/index.html drakma-0.11.2/drakma.asd drakma-0.11.2/packages.lisp drakma-0.11.2/read.lisp drakma-0.11.2/request.lisp drakma-0.11.2/specials.lisp drakma-0.11.2/util.lisp edi@groucho:/tmp$ head drakma-0.11.2/CHANGELOG.txt Version 0.11.2 2008-01-14 Disable WRITE-TIMEOUT for LW5.0 if SSL is used (reported by Nico de Jager)
Version 0.11.1 2007-10-11 Make Drakma work with AllegroCL's "modern" mode (patch by Ross Jekel) Needs at least Chunga 0.4.1 and FLEXI-STREAMS 0.13.1
Version 0.11.0
On Mon, 14 Jan 2008 17:47:58 +0100, Edi Weitz edi@agharta.de wrote:
That's strange - I get the new one. See below.
But I've just released a new version anyway which "refines" the previous change a bit. Again, only relevant for LispWorks users.
Edi Weitz edi@agharta.de writes:
On Mon, 14 Jan 2008 17:47:58 +0100, Edi Weitz edi@agharta.de wrote:
That's strange - I get the new one. See below.
It appears that my ISP caches the binary file (see below). I get the new version if I dial-up with my mobile phone to another ISP.
But I've just released a new version anyway which "refines" the previous change a bit. Again, only relevant for LispWorks users.
It works!! If I were religious, you would be my God!
What will the trade-off of the patch be in a production environment? Did LispWorks indicate that they will release a patch for this bug in their next release? Do you have a ticket number that I can track? I bought the Windows version specifically for this app, so I am a bit disappointed.
Thanks again. Nico
ndj@davinci:~/tmp$ date Tue Jan 15 08:09:58 SAST 2008 ndj@davinci:~/tmp$ wget http://weitz.de/files/drakma.tar.gz --08:10:14-- http://weitz.de/files/drakma.tar.gz => `drakma.tar.gz' Resolving weitz.de... 81.169.156.108 Connecting to weitz.de|81.169.156.108|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 42,282 (41K) [application/x-tar]
100%[=====================================================================================================================================================================>] 42,282 60.92K/s
08:10:14 (60.80 KB/s) - `drakma.tar.gz' saved [42282/42282]
ndj@davinci:~/tmp$ tar xvzf drakma.tar.gz drakma-0.11.1/ drakma-0.11.1/CHANGELOG.txt drakma-0.11.1/cookies.lisp drakma-0.11.1/doc/ drakma-0.11.1/doc/index.html drakma-0.11.1/drakma.asd drakma-0.11.1/packages.lisp drakma-0.11.1/read.lisp drakma-0.11.1/request.lisp drakma-0.11.1/specials.lisp drakma-0.11.1/util.lisp ndj@davinci:~/tmp$ head drakma-0.11.1/CHANGELOG.txt Version 0.11.1 2007-10-11 Make Drakma work with AllegroCL's "modern" mode (patch by Ross Jekel) Needs at least Chunga 0.4.1 and FLEXI-STREAMS 0.13.1
Version 0.11.0 2007-10-01 Added *TEXT-CONTENT-TYPES* and *BODY-FORMAT-FUNCTION* (suggested by Peter Eddy)
Version 0.10.2
On Tue, 15 Jan 2008 08:36:23 +0200, Nico de Jager ndj@hivsa.com wrote:
It appears that my ISP caches the binary file (see below).
Strange. And they don't update the cache if the file changes upstream? Time to switch...
It works!! If I were religious, you would be my God!
You're overdoing it a bit... :)
What will the trade-off of the patch be in a production environment? Did LispWorks indicate that they will release a patch for this bug in their next release? Do you have a ticket number that I can track?
I'm going to answer that in a private email.
Edi.
On Tue, 15 Jan 2008 08:36:23 +0200, Nico de Jager ndj@hivsa.com wrote:
What will the trade-off of the patch be in a production environment?
Oh, I forgot - you can of course always implement a write timeout manually if you depend on it. See for example the WITH-TIMEOUT macro in Hunchentoot's port-lw.lisp.
-
Edi Weitz -
Nico de Jager