Hi,
I'm trying to see if it's feasible to use lisp + tbnl to build an internal documentation system that allows user to upload huge files.
Typically users might upload files with size ranges from 20mb to 600mb.
I'm using lispworks as the development platform, but intend to deploy the site in freebsd/linux + sbcl/cmucl.
I notice that by uploading a 600mb file on win32 + lispworks + tbnl mod-lisp, the memory usage remains constant (ie. Lispworks is streaming the content directly to disk).
However, with either freebsd/cmucl or linux/sbcl, top shows that all of the uploaded contents are first stored in memory and then get written to disk. This is a major scalability problem, I'm not sure if this is a design issue that can be fixed.
Since uploading huge files take a bit of time, displaying a progress is a must; otherwise user might think that the browser hangs. It is also good to let them know how long will it take to upload the file so they can check back in couple minutes instead of staring at a stalled screen.
There exist some open source implementations for php and ruby. http://www.raditha.com/php/progress.php http://sean.treadway.info/demo/upload/
I quickly went through the tbnl request post initialization routine and the rfc2388 module. It looks like what we need to do is to change rfc2388 to continuously write a report of the current upload progress to a file, and then add an entry point in tbnl to return a html page showing the current progress. This page can be launched by javascript when the upload form is submitted.
Am I missing any important piece of information in this picture? If anyone can give me more pointers I would try my hands on this as an exercise. I'm not sure if anyone would find this useful though.
Thanks in advance. -- Mac
I'm not sure if anyone would find this useful though.
I for one would find this useful. It would also serve to make TBNL a more compelling system, as this is a hard problem to solve at the application level in many frameworks.
If anyone can give me more pointers I would try my hands on this
You seem to have identified the key points. I will take a look through the code and see if I can add anything that might be useful to you.
Cheers,
-- Travis
Mac Chan wrote:
Hi,
I'm trying to see if it's feasible to use lisp + tbnl to build an internal documentation system that allows user to upload huge files.
Typically users might upload files with size ranges from 20mb to 600mb.
I'm using lispworks as the development platform, but intend to deploy the site in freebsd/linux + sbcl/cmucl.
I notice that by uploading a 600mb file on win32 + lispworks + tbnl mod-lisp, the memory usage remains constant (ie. Lispworks is streaming the content directly to disk).
However, with either freebsd/cmucl or linux/sbcl, top shows that all of the uploaded contents are first stored in memory and then get written to disk. This is a major scalability problem, I'm not sure if this is a design issue that can be fixed.
Since uploading huge files take a bit of time, displaying a progress is a must; otherwise user might think that the browser hangs. It is also good to let them know how long will it take to upload the file so they can check back in couple minutes instead of staring at a stalled screen.
There exist some open source implementations for php and ruby. http://www.raditha.com/php/progress.php http://sean.treadway.info/demo/upload/
I quickly went through the tbnl request post initialization routine and the rfc2388 module. It looks like what we need to do is to change rfc2388 to continuously write a report of the current upload progress to a file, and then add an entry point in tbnl to return a html page showing the current progress. This page can be launched by javascript when the upload form is submitted.
Am I missing any important piece of information in this picture? If anyone can give me more pointers I would try my hands on this as an exercise. I'm not sure if anyone would find this useful though.
Thanks in advance. -- Mac _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
Hi!
On Thu, 9 Feb 2006 01:38:13 -0800, Mac Chan emailmac@gmail.com wrote:
I notice that by uploading a 600mb file on win32 + lispworks + tbnl mod-lisp, the memory usage remains constant (ie. Lispworks is streaming the content directly to disk).
However, with either freebsd/cmucl or linux/sbcl, top shows that all of the uploaded contents are first stored in memory and then get written to disk. This is a major scalability problem, I'm not sure if this is a design issue that can be fixed.
This should not happen. Maybe you're using older versions of TBNL here? The code which is responsible for the streaming was inadvertently left out from some released versions.
I quickly went through the tbnl request post initialization routine and the rfc2388 module. It looks like what we need to do is to change rfc2388 to continuously write a report of the current upload progress to a file, and then add an entry point in tbnl to return a html page showing the current progress. This page can be launched by javascript when the upload form is submitted.
Am I missing any important piece of information in this picture? If anyone can give me more pointers I would try my hands on this as an exercise. I'm not sure if anyone would find this useful though.
What you describe seems to be the usual way to do it. I also did something similar once for a mod_perl application.
As far as changing RFC2388 is concerned, it would be nice if we could convince Janis to add the changes to his distributions (and therefore make them unobtrusive so they won't bother non-TBNL users).
Thanks, Edi.
On 2/9/06, Edi Weitz edi@agharta.de wrote:
This should not happen. Maybe you're using older versions of TBNL here? The code which is responsible for the streaming was inadvertently left out from some released versions.
I forget what version of tbnl I was testing with, but it was the latest one when I tried it. I'll report back when I have time to do this test again.
What you describe seems to be the usual way to do it. I also did something similar once for a mod_perl application.
As far as changing RFC2388 is concerned, it would be nice if we could convince Janis to add the changes to his distributions (and therefore make them unobtrusive so they won't bother non-TBNL users).
I spent sometime reading the perl code by Raditha Dissanayake (http://www.raditha.com/php/progress.php) and implementede the same idea in tbnl.
Basically, before the user submits a file, we need to create a unique identifier (at least unique for each user) and associate this id with the form post request.
We then pass this id to a popup page so it can poll the uploading progress.
Since the file upload will be part of a multipart/form-data stream, it's easier to pass the id in the query string and rearrange the initialization order of the get-parameters and post-parameters. This way by the time we are saving the file (post-parameters) we'll have the id ready (get-parameters).
There might be a cleaner way to do this but I don't know better. Please take a look and feel free to change it or make it sucks less (I'm not trying to create more work for Edi or anyone though).
TBNL is really a lot more fun than other web frameworks I've used.
Cheers, -- Mac
On Mon, 13 Feb 2006 16:15:33 -0800, Mac Chan emailmac@gmail.com wrote:
Please take a look and feel free to change it or make it sucks less (I'm not trying to create more work for Edi or anyone though).
Thanks for the patch. We're in the process of moving to a new flat, so I probably won't have much time to review this in the next days. Maybe at the end of the month.
TBNL is really a lot more fun than other web frameworks I've used.
Thanks... :)
Cheers, Edi.
-
Edi Weitz -
Mac Chan -
Travis Cross