February 2007 - drakma-devel@common-lisp.net

[drakma-devel] Re: Portability of Drakma
by Edi Weitz 07 Apr '07

07 Apr '07

On Sun, 25 Feb 2007 14:38:02 +0100, "Erik Huelsmann" <ehuels(a)gmail.com> wrote: >> - The code should have been tested successfully on at least the >> Lisp/OS combinations that are currently supported by Drakma. > > Is there a list somewhere as a reference to what I'm getting into? No, unfortunately not. I myself use mostly LispWorks (Windows and Linux/x86) and SBCL (Linux/x86). (LispWorks shouldn't be a problem anyway as it's not affected by the switch.) I think that at least LispWorks, SBCL, AllegroCL, CMUCL, CLISP, and OpenMCL should be supported, everything else being a bonus. Operating systems: Windows (where applicable), Linux, OS X. If you don't want to test all of this for yourself, how about offering a tarball of Drakma which uses usocket for download? Send an announcement to this mailing list including an overview of what you've tested and what not. We can ask "interested parties" to try it out and we'll switch to the new version in four weeks, say, unless someone objects. Does that sound OK? Cheers, Edi.

3 20

[drakma-devel] Fwd: Portability of Drakma
by Erik Huelsmann 27 Feb '07

27 Feb '07

Forwarding rejected message. I wasn't subscribed yet. Sorry. bye, Erik. ---------- Forwarded message ---------- From: Erik Huelsmann <ehuels(a)gmail.com> Date: Feb 25, 2007 2:38 PM Subject: Re: Portability of Drakma To: Edi Weitz <edi(a)agharta.de> Cc: drakma-devel(a)common-lisp.net On 2/24/07, Edi Weitz <edi(a)agharta.de> wrote: > Hi Erik, > > I'm sending a copy of this to the mailing list where I think we should > continue this discussion. Ah. Sorry about that, I wasn't aware of this list. > > I've been working on a very portable library for sockets code. This > > library is now more portable than trivial-sockets and supports more > > functionality on all of its supported lisp implementations. > > I'm aware of usockets' existence because Andreas Fuchs pointed it out > to me shortly after I had released the portable version of Drakma > (using trivial-sockets). At that point I tried to switch to usocket > and immediately ran into problems - IIRC it didn't even load on > LispWorks on Windows (although ISTR the website claimed that LispWorks > was a supported implementation), and it couldn't provide binary socket > streams for all supported implementations. So, I dismissed it for the > time being. That's both great and bad news: It's great you're aware of the usocket project, it's too bad you tried and failed. > It might well be the case that both of these issues have been fixed > since, but I currently don't have the time to test again. I generally > think it's better to rely on a maintained and documented library than > on obscure and old code, but of course the new code should work at > least as good as the old one. Absolutely. New code shouldn't be a step backward. With that requirement, a chicken-and-egg problem is introduced though: to develop well-tested code, it needs to be (widely) used. But to address your findings: you probably used one of the very first releases: With 0.3.0, binary streams are supported on all implementations. Next to that, I just downloaded and used LW5.0 to test a simple GET request: all seems to work well. Indeed have there been win32 related fixes to many backends. > I'd be happy to accept patches to switch Drakma from trivial-sockets > to usocket, but the following criteria should be met: > > - The LispWorks code should remain untouched (i.e. not use usocket). > > - The code should have been tested successfully on at least the > Lisp/OS combinations that are currently supported by Drakma. Is there a list somewhere as a reference to what I'm getting into? > The actual patch itself should be a piece of cake, but I guess the > testing will take some time. Yes. Not having a Mac, I won't be able to test OpenMCL myself, but maybe others can assist there? Thanks for your time. bye, Erik.

2 1

[drakma-devel] Bug handling bad html?
by Jeffrey Cunningham 25 Feb '07

25 Feb '07

I was playing with drakma and had it drop into the debugger when retrieving a commercial page. It looks like it might be a bug in flexi-streams, but I don't know how to isolate the input more specifically than what came up here: Unexpected value #xA0 at start of UTF-8 sequence. [Condition of type FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR] Restarts: 0: [ABORT] Abort SLIME compilation. 1: [ABORT] Return to SLIME's top level. 2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker" {CDD6631}>) Backtrace: 0: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR #<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {12CF8379}> "Unexpected value #x~X at start of UTF-8 sequence." 160) 1: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR #<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {12CF8379}> "Unexpected value #x~X at start of UTF-8 sequence.") 2: ((FLET #:BODY-FN327)) 3: ((SB-PCL::FAST-METHOD STREAM-READ-CHAR (FLEXI-STREAMS::FLEXI-UTF-8-INPUT-STREAM)) #<unavailable argument> #<unavailable argument> #<unavailable argument>) 4: ((SB-PCL::FAST-METHOD TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE (FLEXI-STREAMS:FLEXI-INPUT-STREAM #1="#<...>" . #1#)) #<unused argument> #<unused argument> #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument>) 5: (READ-SEQUENCE "y make a difference this holiday season. Our gift ideas are unique and of high quality. <a href=\"http://www.1giftidea.com/\" target=\"_blank\" title=\"Christmas Gift Ideas\">Gift ideas for every occasion, Christmas, Birthday, Mother's day...</a> Gift ideas for every occasion, Christmas, Birthday, Mothers day, Graduation, Fathers day, Anniversary, Wedding, & Baby Shower. <a href=\"http://www.mixedblessing.com/\" target=\"_blank\" title=\"Mixed Blessing\">Hanukkah card, Christmas gift idea and Holiday greeting cards from MixedBlessing</a> Greeting Cards for Interfaith and Multicultures from MixedBlesing. Hanukkah cards, Holiday cards, Christmas Gift Ideas, Holiday Gifts and more.. Find great gifts now! ..) 6: (DRAKMA::READ-BODY #<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {12CF8379}> ((:DATE . "Sat, 24 Feb 2007 06:30:03 GMT") (:SERVER . "Apache/2.0.46 (Red Hat)") (:SET-COOKIE . "GS_UUID=24.18.193.65.1172298603635841; path=/,PHPSESSID=e009a521cb2bf134a00df925e4f4d510; path=/,cart_hash=e009a521cb2bf134a00df925e4f4d510; expires=Tuesday, 27-Feb-07 06:30:03 GMT; path=/") (:X-POWERED-BY . "PHP/4.4.0") (:EXPIRES . "Thu, 19 Nov 1981 08:52:00 GMT") (:CACHE-CONTROL . "no-store, no-cache, must-revalidate, post-check=0, pre-check=0") ..)) 7: ((LABELS DRAKMA::FINISH-REQUEST) NIL NIL) 8: (HTTP-REQUEST #<URI http://www.gifttree.com/Christmas/Christmas-gift-idea.html> :PROXY NIL) 9: (RETRIEVE-URI "http://www.gifttree.com/Christmas/Christmas-gift-idea.html" NIL) 10: (WALK-SITE "http://www.gifttree.com/Christmas/Christmas-gift-idea.html" #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument>) 11: (SB-FASL::FOP-FUNCALL) 12: (SB-FASL::LOAD-FASL-GROUP #<SB-SYS:FD-STREAM for "file /tmp/fileIQGlqR.fasl" {CDF1089}>) 13: (SB-FASL::LOAD-AS-FASL #<SB-SYS:FD-STREAM for "file /tmp/fileIQGlqR.fasl" {CDF1089}> NIL #<unavailable argument>) 14: (SB-FASL::INTERNAL-LOAD #P"/tmp/fileIQGlqR.fasl" #P"/tmp/fileIQGlqR.fasl" :ERROR NIL NIL :BINARY NIL) 15: (SB-FASL::INTERNAL-LOAD #P"/tmp/fileIQGlqR.fasl" #P"/tmp/fileIQGlqR.fasl" :ERROR NIL NIL NIL :DEFAULT) 16: (LOAD #P"/tmp/fileIQGlqR.fasl") 17: ((LAMBDA (STRING &KEY #1="#<...>" . #1#)) "(print (walk-site \"http://www.gifttree.com\")) " :BUFFER "seo.lisp" :POSITION 27060 :DIRECTORY #<unused argument>) 18: ((LAMBDA ())) --more-- --Jeff

4 13

[drakma-devel] Re: Portability of Drakma
by Edi Weitz 24 Feb '07

24 Feb '07

Hi Erik, I'm sending a copy of this to the mailing list where I think we should continue this discussion. On Sat, 24 Feb 2007 22:35:10 +0100, "Erik Huelsmann" <ehuels(a)gmail.com> wrote: > I've been working on a very portable library for sockets code. This > library is now more portable than trivial-sockets and supports more > functionality on all of its supported lisp implementations. > > If you want to support the same platforms (and all the ones I'll be > adding), you could switch from the -unmaintained- trivial-sockets to > usocket (http://common-lisp.net/project/usocket/) > > I'm merely sending this mail to point out the existence of the > library, in case you didn't know. Thanks for your time, attention > and continued support for Common Lisp libraries. I'm aware of usockets' existence because Andreas Fuchs pointed it out to me shortly after I had released the portable version of Drakma (using trivial-sockets). At that point I tried to switch to usocket and immediately ran into problems - IIRC it didn't even load on LispWorks on Windows (although ISTR the website claimed that LispWorks was a supported implementation), and it couldn't provide binary socket streams for all supported implementations. So, I dismissed it for the time being. It might well be the case that both of these issues have been fixed since, but I currently don't have the time to test again. I generally think it's better to rely on a maintained and documented library than on obscure and old code, but of course the new code should work at least as good as the old one. I'd be happy to accept patches to switch Drakma from trivial-sockets to usocket, but the following criteria should be met: - The LispWorks code should remain untouched (i.e. not use usocket). - The code should have been tested successfully on at least the Lisp/OS combinations that are currently supported by Drakma. The actual patch itself should be a piece of cake, but I guess the testing will take some time. Thanks, Edi.

1 0

[drakma-devel] Darcs repositories
by Edi Weitz 22 Feb '07

22 Feb '07

[My apologies if you get this more than once.] Several people have asked for Darcs repositories of my software. These do exists now: http://common-lisp.net/~loliveira/ediware/ Special thanks to Luís Oliveira who made this possible and who maintains the repositories. Cheers, Edi.

1 0

[drakma-devel] Problem with file uploading
by Andrei Stebakov 14 Feb '07

14 Feb '07

This is exaclty what I need. The GET method works just fine, but I have trouble with the POST method uploading the files. Edi, here is a question (I am not sure if it's the right mailing list to ask it...) When I say (this is part of a function, so I use back-quote for parameters): (drakma:http-request "/some/uri" :method :post :form-data t :parameters `(("Name1" . ,name1) ("Name2" . ,name2) ("File" . ,file-name)))) I got an "unknown error" from the remote host. Looks like there is problem with streaming file contents. I did a little of debugging printing the content of file buffer (in send-content function) looks like the file is being open and read, but something happens at the receiving end. I wonder how can I debug it more. When I do the same request from the FORM in Firefox everything works. What debuggind techniques I can try here? (Sorry I am still very new to Lisp) Thank you, Andrew

3 6

[drakma-devel] Not following redirects and conditions
by Saurabh Nanda 09 Feb '07

09 Feb '07

Hi, I'm trying to write some tests for an HTTP based API. I need to check whether a server responds with a 302 status code and then need to check the referred location as well, without actually visiting that link. How is it possible with drakma? If I use :redirect 0 then the http-request function throws up an error. TIA Nandz. -- http://nandz.blogspot.com http://foodieforlife.blogspot.com

2 4

[drakma-devel] Closing streams with :want-stream t
by Chris Dean 08 Feb '07

08 Feb '07

I have a file handle leak in my code (that is, I run out of file handles after running for a while) and I suspect that I'm not using drakma correctly. I wish to use the :want-stream t parameter and read the resulting stream directly, so I have this code: (defun simple-get (url) "Download the url using GET and return the body as a string." (handler-case (multiple-value-bind (stream code headers dummy-uri dummy-stream must-close?) (drakma:http-request url :want-stream t :keep-alive nil :method :get) (declare (ignore headers dummy-stream dummy-uri)) (unwind-protect (and stream code (= code 200) (with-output-to-string (out) (do ((ch (read-char stream nil :eof) (read-char stream nil :eof))) ((not (characterp ch))) (princ ch out)))) (when (and stream must-close?) (ignore-errors (close stream))))) (error (condition) (format t "Error ~s: ~a~%" url condition) nil))) Is there anything extra I need to do to make sure that all the streams opened by drakma are closed? My production code is much more complex, but the simple stub above will generate the out of file handles problem. Besides the actual error I can use lsof on Linux and Mac OS and see many sockets stuck in CLOSED or CLOSE_WAIT states. This is all under LispWorks 5.0.1 Also, when using :want-stream nil I never encounter the problem. Cheers, Chris Dean

3 12

[drakma-devel] New Drakma release 0.6.0
by Edi Weitz 08 Feb '07

08 Feb '07

ChangeLog: Version 0.6.0 2006-02-08 Make sure stream is closed in case of early errors (thanks to Chris Dean for test data) Robustified cookie parsing Send all outgoing cookies in one fell swoop (for Sun's buggy web server) Deal with empty Location headers Deal with corrupted Content-Type headers Download: http://weitz.de/files/drakma.tar.gz Have fun, Edi.

1 0

[drakma-devel] New Chunga release 0.2.4
by Edi Weitz 08 Feb '07

08 Feb '07

ChangeLog: Version 0.2.4 2007-02-08 Allow more characters in cookie names/values according to original Netscape spec Robustified READ-COOKIE-VALUE Download: http://weitz.de/files/chunga.tar.gz Cheers, Edi.

1 0