[drakma-devel] Sending content without loading it all in memory?
Is there any way to send a non-chunked request without having the entire content in memory at once? I'm working on an Amazon S3 client based on Drakma, but sometimes I want to upload (via PUT) very large files and I'd prefer to determine the content-length via FILE-LENGTH and upload the body buffer-by-buffer instead of all at once. Zach
On Wed, 10 Sep 2008 09:20:30 -0400, Zach Beane <xach@xach.com> wrote:
Is there any way to send a non-chunked request without having the entire content in memory at once?
You can set the content length header yourself. In that case Drakma shouldn't use chunked encoding IIRC. http://weitz.de/drakma/#content-length
I'm working on an Amazon S3 client based on Drakma, but sometimes I want to upload (via PUT) very large files and I'd prefer to determine the content-length via FILE-LENGTH and upload the body buffer-by-buffer instead of all at once.
That should be no problem if you provide a function as the :content keyword argument or use the "continuation" facility. Cheers, Edi.
On Wed, Sep 10, 2008 at 05:03:33PM +0200, Edi Weitz wrote:
On Wed, 10 Sep 2008 09:20:30 -0400, Zach Beane <xach@xach.com> wrote:
Is there any way to send a non-chunked request without having the entire content in memory at once?
You can set the content length header yourself. In that case Drakma shouldn't use chunked encoding IIRC.
Unfortunately that's not the case, as the documentation says: | A non-NIL content-length argument means that Drakma must build the | request body in RAM and compute the content length even if it would | have otherwise used chunked encoding - for example in the case of | file uploads. The code says it too: (when (and content-length (not (or (arrayp content) (listp content) (eq content :continuation)))) ;; CONTENT-LENGTH forces us to compute request body ;; in RAM ...)
I'm working on an Amazon S3 client based on Drakma, but sometimes I want to upload (via PUT) very large files and I'd prefer to determine the content-length via FILE-LENGTH and upload the body buffer-by-buffer instead of all at once.
That should be no problem if you provide a function as the :content keyword argument or use the "continuation" facility.
I tried that too, but: (when (and (eq content :continuation) content-length) (error "CONTENT-LENGTH must be NIL if CONTENT is :CONTINUATION.")) I think I might be out of luck. Zach
On Wed, Sep 10, 2008 at 11:07:49AM -0400, Zach Beane wrote:
On Wed, Sep 10, 2008 at 05:03:33PM +0200, Edi Weitz wrote:
On Wed, 10 Sep 2008 09:20:30 -0400, Zach Beane <xach@xach.com> wrote:
Is there any way to send a non-chunked request without having the entire content in memory at once?
You can set the content length header yourself. In that case Drakma shouldn't use chunked encoding IIRC.
Unfortunately that's not the case, as the documentation says: [snip]
Or rather, you're right, but not chunking is only half the battle. I'd like to avoid loading the whole content into memory too... Zach
On Wed, 10 Sep 2008 11:11:15 -0400, Zach Beane <xach@xach.com> wrote:
Or rather, you're right, but not chunking is only half the battle. I'd like to avoid loading the whole content into memory too...
From a brief look it seems to me that some of the tests in the Drakma code are unnecessary and could be removed so that what you want can actually be achieved. I don't have the time to look at it right now, though. (Actually, one of the many half-finished things I'm working on is a test suite for Drakma and Hunchentoot and I'd rather finish the test suite first before I modify the internals. RSN...)
"Edi Weitz" <edi@agharta.de> wrote in message news:umyig6l5a.fsf@agharta.de...
On Wed, 10 Sep 2008 11:11:15 -0400, Zach Beane <xach@xach.com> wrote:
Or rather, you're right, but not chunking is only half the battle. I'd like to avoid loading the whole content into memory too...
From a brief look it seems to me that some of the tests in the Drakma code are unnecessary and could be removed so that what you want can actually be achieved. I don't have the time to look at it right now, though. [...]
Is the attached patch in the general direction of what you're getting at? (patch avoids computing request-body in RAM if content-length is provided and numeric.) I'm fairly certain the behavior is correct: * drakma::send-content already handles sending streams and files using a buffer (what Zach wanted) * content-length is always trusted if provided, even when the request-body is computed in RAM--which makes the in-memory computation superfluous unless (eq content-length t) The only change in behavior I am aware of is the error "Don't know how to send content ~S to server." (e.g., for a non-octet stream) moves from being before drakma sends the 'Content-length' header to after, when content-length is numeric. However, there is always the very real possibility I missed something and it will blow up when somebody else uses it. -pix begin 666 content-length.patch M+2TM(&]L9"UD<F%K;6$O<F5Q=65S="YL:7-P"3(P,#@M,3 M,C@@,#4Z,#8Z M,#DN,# P,# P,# P("LP,# P"BLK*R!N97<M9')A:VUA+W)E<75E<W0N;&ES M< DR,# X+3$P+3(X(# U.C V.C$P+C P,# P,# P," K,# P, I 0" M,C@Q M+#$R("LR.#$L,3(@0$ *('-E;F0@=&AE(&-O;G1E;G0@8F]D>2X@($YO=&4@ M=&AA="!T:&ES('=I;&P@;F]T('=O<FL@=VET:"!O;&1E<B!W96(*('-E<G9E M<G,N"B *+4$@;F]N+4Y)3"!#3TY414Y4+4Q%3D=42"!A<F=U;65N="!M96%N M<R!T:&%T($1R86MM82 O;75S="\@8G5I;&0@=&AE"BUR97%U97-T(&)O9'D@ M:6X@4D%-(&%N9"!C;VUP=71E('1H92!C;VYT96YT(&QE;F=T:"!E=F5N(&EF M(&ET('=O=6QD"BUH879E(&]T:&5R=VES92!U<V5D(&-H=6YK960@96YC;V1I M;F<L(&9O<B!E>&%M<&QE(&EN('1H92!C87-E(&]F(&9I;&4*+75P;&]A9',N M("!!('-P96-I86P@8V%S92!I<R!T:&4@=F%L=64@5"!F;W(@0T].5$5.5"U, M14Y'5$@@=VAI8V@@;65A;G,*+71H870@1')A:VUA('-H;W5L9"!C;VUP=71E M('1H92!C;VYT96YT(&QE;F=T:"!A9G1E<B!B=6EL9&EN9R!T:&4*+7)E<75E M<W0@8F]D>2X**T-/3E1%3E0M3$5.1U1(+"!S970@=&\@5"P@;65A;G,@=&AA M="!$<F%K;6$@+VUU<W0O(&)U:6QD('1H92!R97%U97-T"BMB;V1Y(&EN(%)! M32!A;F0@8V]M<'5T92!T:&4@8V]N=&5N="!L96YG=&@@979E;B!I9B!I="!W M;W5L9"!H879E"BMO=&AE<G=I<V4@=7-E9"!C:'5N:V5D(&5N8V]D:6YG+"!F M;W(@97AA;7!L92!I;B!T:&4@8V%S92!O9B!F:6QE"BMU<&QO861S+B @4V5T M=&EN9R!#3TY414Y4+4Q%3D=42"!T;R!A(&YU;65R:6,@=F%L=64@=VEL;"!B M92!A<W-U;65D"BMC;W)R96-T+"!A;&QO=VEN9R!T:&4@<F5Q=65S="!B;V1Y M('1O(&)E('-T<F5A;65D(&9R;VT@0T].5$5.5"!W:&5R90HK87!P;&EC86)L M92X*( H@0T].5$5.5"U465!%(&ES('1H92!C;W)R97-P;VYD:6YG(&!#;VYT M96YT+51Y<&4G(&AE861E<B!T;R!B92!S96YT(&%N9 H@=VEL;"!B92!I9VYO M<F5D('5N;&5S<R!#3TY414Y4(&ES('!R;W9I9&5D(&%S('=E;&PN"D! ("TU M,C@L-R K-3(X+#@@0$ *(" @(" @(" @(" @(" @(" H=VAE;B!C;VYT96YT M+71Y<&4*(" @(" @(" @(" @(" @(" @("AW<FET92UH96%D97(@(D-O;G1E M;G0M5'EP92(@(GY!(B!C;VYT96YT+71Y<&4I*0H@(" @(" @(" @(" @(" @ M("AW:&5N("AA;F0@8V]N=&5N="UL96YG=&@*+2 @(" @(" @(" @(" @(" @ M(" @(" @(" @("AN;W0@*&]R("AA<G)A>7 @8V]N=&5N="D**R @(" @(" @ M(" @(" @(" @(" @(" @(" @("AN;W0@*&]R("AN=6UB97)P(&-O;G1E;G0M M;&5N9W1H*0HK(" @(" @(" @(" @(" @(" @(" @(" @(" @(" @(" @(" @ M*&%R<F%Y<"!C;VYT96YT*0H@(" @(" @(" @(" @(" @(" @(" @(" @(" @ M(" @(" @(" @*&QI<W1P(&-O;G1E;G0I"B @(" @(" @(" @(" @(" @(" @ M(" @(" @(" @(" @(" @(" H97$@8V]N=&5N=" Z8V]N=&EN=6%T:6]N*2DI M*0H@(" @(" @(" @(" @(" @(" @.SL@0T].5$5.5"U,14Y'5$@@9F]R8V5S =('5S('1O(&-O;7!U=&4@<F5Q=65S="!B;V1Y"@H` ` end
On Wed, Sep 10, 2008 at 11:07:49AM -0400, Zach Beane wrote:
On Wed, Sep 10, 2008 at 05:03:33PM +0200, Edi Weitz wrote:
On Wed, 10 Sep 2008 09:20:30 -0400, Zach Beane <xach@xach.com> wrote:
Is there any way to send a non-chunked request without having the entire content in memory at once?
You can set the content length header yourself. In that case Drakma shouldn't use chunked encoding IIRC.
Attached is a small patch that updates Drakma so it doesn't reject a content-length if content is :continuation. It has allowed me to upload big files without loading them into memory first. Zach
participants (3)
-
Edi Weitz -
Pixel // pinterface -
Zach Beane