[tbnl-devel] Problem with file uploads
Hi-- I'm using tbnl to run a site (educational in purpose), which will be rich in texts and images. I want people with the appropriate privileges to be able to upload images to the server, where various incantations will be performed to ensure that the images are formatted correctly for the site. However, I seem to be running into a problem with either mod lisp or tbnl, for images >=~1MB. When I try to upload an image of this site, an internal server error is returned after a few seconds, and the converted image (using imagemagick) appears in the thumbnail gallery with a portion of the bottom shown in grey, as if the image did not upload completely to the server. Is there some configurable upper limit to the size of file uploads, or can someone point me to the likely source of the problem? regards -- Tiarnán
"Tiarnán Ó Corráin" <ocorrain@yahoo.com> wrote: [...]
However, I seem to be running into a problem with either mod lisp or tbnl, for images >=~1MB. When I try to upload an image of this site, an internal server error is returned after a few seconds, and the converted image (using imagemagick) appears in the thumbnail gallery with a portion of the bottom shown in grey, as if the image did not upload completely to the server.
Is there some configurable upper limit to the size of file uploads, or can someone point me to the likely source of the problem?
What Lisp are you using ? For instance in Lispworks the array-total-size-limit is very small: 1048448 so you can't have strings greater than that. :-( Marc
"Marc Battyani" <marc.battyani@fractalconcept.com> writes:
What Lisp are you using ?
OpenMCL on Mac OS X (for development), and CMUCL on FreeBSD (deployment). Just checked, and CMUCL seems to be more forgiving, with an array-total-size-limit of 536870911, as opposed to 16777216 in OpenMCL. Is there any way of getting around these limitations, perhaps by rewriting the code that handles uploads? If so, could you point me in the direction of what needs to change? regards Tiarnán
On Tue, 08 Mar 2005 15:58:05 +0000, ocorrain@yahoo.com (Tiarnán Ó Corráin) wrote:
Is there any way of getting around these limitations, perhaps by rewriting the code that handles uploads? If so, could you point me in the direction of what needs to change?
I'm not aware of any limitations - I specifically rewrote Janis Dzerins' to read from a stream and write to a stream. FWIW, I just uploaded a 126MB MP3 file via TBNL/LispWorks (using the upload test that comes with the distribution), downloaded it again, and got a result which is identical to the original file according to diff. Cheers, Edi.
Edi Weitz <edi@agharta.de> writes:
On Tue, 08 Mar 2005 15:58:05 +0000, ocorrain@yahoo.com (Tiarnán Ó Corráin) wrote:
Is there any way of getting around these limitations, perhaps by rewriting the code that handles uploads? If so, could you point me in the direction of what needs to change?
I'm not aware of any limitations - I specifically rewrote Janis Dzerins' to read from a stream and write to a stream.
What version of tbnl? I'm using 0.3.11 at the mo, and am definitely running into problems that seem to be related to array-size-limit. I'll see if I can come up with a reproducible bug report. -- Tiarnán
On Tue, 08 Mar 2005 17:35:19 +0000, ocorrain@yahoo.com (Tiarnán Ó Corráin) wrote:
What version of tbnl?
I tested with 0.3.12 and 0.3.10, both worked.
I'm using 0.3.11 at the mo, and am definitely running into problems that seem to be related to array-size-limit.
Are you sure you have problems at file upload time and not when delivering the uploaded file back to the user? Delivering large files through TBNL won't work, of course, because all the content is sent to mod_lisp at once. This is a deliberate design decision of TBNL. If you want to serve large static files you can always bypass TBNL and serve them directly through Apache.
I'll see if I can come up with a reproducible bug report.
That would be nice. Cheers, Edi.
Edi Weitz <edi@agharta.de> writes:
On Tue, 08 Mar 2005 17:35:19 +0000, ocorrain@yahoo.com (Tiarnán Ó Corráin) wrote: I tested with 0.3.12 and 0.3.10, both worked.
I'll upgrade and see if it's still a problem.
Are you sure you have problems at file upload time and not when delivering the uploaded file back to the user? Delivering large files through TBNL won't work, of course, because all the content is sent to mod_lisp at once. This is a deliberate design decision of TBNL.
Fair enough. I'm sure that it isn't a problem with serving the files, since I do bypass TBNL for serving images. The flow goes like this: 1. Upload file to server 2. File is changed using imagemagick to a sensible size 3. Thumbnail is produced. 4. User is given gallery of thumbnails As I described earlier in the thread, the thumbnail/processed image is grey at the bottom, leading me to believe that imagemagick is processing an incomplete jpeg. Is there any reason at all that array-max-size-limit would restrict upload size? -- Tiarnán
Tiarnán Ó Corráin writes:
However, I seem to be running into a problem with either mod lisp or tbnl, for images >=~1MB. When I try to upload an image of this site, an internal server error is returned after a few seconds, and the converted image (using imagemagick) appears in the thumbnail gallery with a portion of the bottom shown in grey, as if the image did not upload completely to the server.
I had the same problem trying to upload a 23MB TIFF file to a PIII/733MHz server running Linux, Apache 1.3 and CMUCL. The cause was the "excessive" speed of Apache: the Lisp server is not so fast in handling POST body, its socket buffer is filled up by mod_lisp, which gets a write error and closes the connection. To solve the problem, I added some declarations in rfc2388.lisp and modified the inner loop to read the POST request body in chunks of 4096 byte, using the "Content-length" HTTP header. The result was a 25% increase in file transfer speed, I was able to handle about 3.8MB/sec. Not bad, but not enough for data coming from a 100Mbit/sec LAN. So I had also to "enlarge" the socket buffer with a call to the setsockopt Unix system call. If someone is interested, I could post my changes. With a caveat: request body is read using READ-SEQUENCE; if the connection is closed during transfer, the corresponding server thread hangs indefinitely. Regards, Aurelio
On Fri, 11 Mar 2005 22:03:10 +0100, Aurelio Bignoli <tbnl@bignoli.it> wrote:
I had the same problem trying to upload a 23MB TIFF file to a PIII/733MHz server running Linux, Apache 1.3 and CMUCL. The cause was the "excessive" speed of Apache: the Lisp server is not so fast in handling POST body, its socket buffer is filled up by mod_lisp, which gets a write error and closes the connection.
I wonder why I haven't seen this when I tested with large (>100MB) files on my machine - see previous posting. I tested with server and client on the same machine so the speed should have been high enough. My combination was LW/WinXP/Apache2 so your results /might/ show problems specific to CMUCL or to Linux or to Apache 1.x... Cheers, Edi.
Edi Weitz <edi@agharta.de> writes:
On Fri, 11 Mar 2005 22:03:10 +0100, Aurelio Bignoli <tbnl@bignoli.it> wrote:
I had the same problem trying to upload a 23MB TIFF file to a PIII/733MHz server running Linux, Apache 1.3 and CMUCL. The cause was the "excessive" speed of Apache: the Lisp server is not so fast in handling POST body, its socket buffer is filled up by mod_lisp, which gets a write error and closes the connection.
My combination was LW/WinXP/Apache2 so your results /might/ show problems specific to CMUCL or to Linux or to Apache 1.x...
I have the problem (or a similar one) with OpenMCL/MacOSX/Apache 1.3.31. Haven't experienced it yet on the server, which runs CMUCL/FreeBSD/Apache 1.3.31 -- Tiarnán
Edi Weitz writes:
My combination was LW/WinXP/Apache2 so your results /might/ show problems specific to CMUCL or to Linux or to Apache 1.x...
well, I tried also with LWL Personal and the result was the same. If I have time I'll install Apache 2 and I'll give it a try. Regards, Aurelio
Just started newest tbnl 0.4 and uploaded 180mb file. Yet the name of the file is "[en] Shrek 3D [2004].avi" so i am geting "http://localhost/tbnl/test/files/%5Ben%5D+Shrek+3D+%5B2004%5D.avi?path=%2Ftm..." as a download link which leads to an error when trying to download it ... Error in function LISP::FD-STREAM-READ-N-BYTES: Error reading #<Stream for file "/tmp/tbnl/test/tbnl-test-1">: Bad address though the file in that place exists ... The backtrace is attached in a file. Ignas Mikalajūnas
On Tue, 15 Mar 2005 02:12:13 +0200, Ignas Mikalajunas <ignas.mikalajunas@gmail.com> wrote:
Just started newest tbnl 0.4 and uploaded 180mb file. Yet the name of the file is "[en] Shrek 3D [2004].avi" so i am geting "http://localhost/tbnl/test/files/%5Ben%5D+Shrek+3D+%5B2004%5D.avi?path=%2Ftm..." as a download link which leads to an error when trying to download it ...
Error in function LISP::FD-STREAM-READ-N-BYTES: Error reading #<Stream for file "/tmp/tbnl/test/tbnl-test-1">: Bad address
though the file in that place exists ...
The backtrace is attached in a file.
You didn't say which Lisp and OS you were using but I guess it's CMUCL, right? I don't think the name of the file matters, I guess the file is simply too big to be served by TBNL. See this message: <http://common-lisp.net/pipermail/tbnl-devel/2005-March/000221.html> If you think this is not the cause for the error try this (with or without TBNL loaded): (with-open-file (file "/path/to/tbnl-test-1" :direction :input :element-type '(unsigned-byte 8)) (let* ((len (file-length file)) (buf (make-array len :element-type '(unsigned-byte 8)))) (read-sequence buf file) 'OK)) Does that work? Can you check (with diff) whether tbnl-test-1 and the file you uploaded are identical? Cheers, Edi.
participants (5)
-
Aurelio Bignoli -
Edi Weitz -
Ignas Mikalajunas -
Marc Battyani -
ocorrain＠yahoo.com