- drakma-devel@lists.common-lisp.net - mailman3.common-lisp.net

[drakma-devel] New release 0.5.4
by Edi Weitz 30 Nov '06

30 Nov '06

ChangeLog: Version 0.5.4 2006-12-01 Workaround for servers which send headers after 100 status line (provided by Donavon Keithley) Download: http://weitz.de/files/drakma.tar.gz Cheers, Edi.

1 0

[drakma-devel] Error when chunked transfer encoding is used on clisp
by Anton Vodonosov 26 Oct '06

26 Oct '06

Hi! I've decided to repost this messsage here, because the message I've posted to lispweb mailing list contains stack backtrace produced by slime. But, as it turned out, slime debugger doesn't work well with clisp. The problem is that the http-request function fails when it's used for server with chunker transfer encoding. For example, all the following calls fail: (http-request "http://www.phpsecurepages.com/test/test.php") (http-request "http://bcheck.scanit.be/bcheck/") (http-request "http://google.com") (http-request "http://weitz.de/drakma/") whereas following cals work fine: (http-request "http://zappa.com/favicon.ico") (http-request "http://www.jalat.com/blogs/lisp?id=3") (http-request "http://www.lispworks.com/") (http-request "http://meme.b9.com/login.html") All the failed calls use cunked transfer encoding; all successful - don't use chunked transfer encoding. The error is always the same: SYSTEM::STORE: #\< does not fit into #(0 0 0 0 0 0 0 0 0 0 ...), bad type See backtrace bellow. Also, consider an example: ;; in order to use the ;; chunked-stream-input-stream ;; function (in-package :chunga) (setq *stream* (drakma:http-request "http://weitz.de/drakma/" :want-stream t)) *stream* => #<FLEXI-IO-STREAM #x1A223C11> (flexi-streams:flexi-stream-stream *stream*) => #<CHUNKED-IO-STREAM #x1A221BBD> (chunked-stream-input-stream (flexi-streams:flexi-stream-stream *stream*)) => #<FLEXI-IO-STREAM #x1A223C4D> (flexi-streams:flexi-stream-stream (chunked-stream-input-stream (flexi-streams:flexi-stream-stream *stream*))) => #<IO UNBUFFERED SOCKET-STREAM (UNSIGNED-BYTE 8) weitz.de:80> As far as I understand, chunked streams expect underlying stream to be binary. But in this case, chunked input stream has flexi stream as underlying stream. ----------------------------------------- Backtrace for the error signaled by (http-request "http://weitz.de/drakma/") ----------------------------------------- The error itself: #<SIMPLE-TYPE-ERROR #x1A22990D>: standard object type: SIMPLE-TYPE-ERROR 0 [$DATUM]: #\< 1 [$EXPECTED-TYPE]: (UNSIGNED-BYTE 8) 2 [$FORMAT-CONTROL]: "~S: ~S does not fit into ~S, bad type " 3 [$FORMAT-ARGUMENTS]: (SYSTEM::STORE #\< #<ARRAY (UNSIGNED-BYTE 8) (3566) #x1A228AE5>) ------------------------------------------- Backtrace: ... <11> #<SYSTEM-FUNCTION INVOKE-DEBUGGER> 1 frame binding variables (~ = dynamically): | ~ SYSTEM::*PRIN-STREAM* <--> #<UNBOUND> frame binding variables (~ = dynamically): | ~ *PRINT-READABLY* <--> NIL frame binding variables (~ = dynamically): | ~ *PRINT-ESCAPE* <--> T - #<SIMPLE-TYPE-ERROR #x1A230399> - #(0 0 0 0 0 ...) <12> #<SYSTEM-FUNCTION SYSTEM::STORE> 3 - #\< - 0 - #(8 #<SYSTEM-FUNCTION SYSTEM::VECTOR-INIT> #<SYSTEM-FUNCTION SYSTEM::VECTOR-UPD> #<SYSTEM-FUNCTION SYSTEM::VECTOR-ENDTEST> #<SYSTEM-FUNCTION SYSTEM::VECTOR-FE-INIT> #<SYSTEM-FUNCTION SYSTEM::VECTOR-FE-UPD> #<SYSTEM-FUNCTION SYSTEM::VECTOR-FE-ENDTEST> #<SYSTEM-FUNCTION AREF> #<SYSTEM-FUNCTION SYSTEM::STORE> #<SYSTEM-FUNCTION IDENTITY> #<SYSTEM-FUNCTION SYSTEM::VECTOR-LENGTH> #<COMPILED-FUNCTION #:|67 91 (MAPC #'(LAMBDA # #) '(1 2 4 ...))-5-1|> #<SYSTEM-FUNCTION AREF> #<SYSTEM-FUNCTION SYSTEM::STORE> #<SYSTEM-FUNCTION SYSTEM::VECTOR-INIT-START> #<SYSTEM-FUNCTION SYSTEM::VECTOR-FE-INIT-END>) - 3566 - 0 - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22CB5D> - #(0 0 0 0 0 ...) <13> #<SYSTEM-FUNCTION EXT:READ-CHAR-SEQUENCE> - #<SYSTEM-FUNCTION EXT:READ-CHAR-SEQUENCE> - (UNSIGNED-BYTE 8) - CHARACTER - #<COMPILED-FUNCTION READ-SEQUENCE> - 3566 - 0 - (:START 0 :END 3566) - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22CB5D> - #(0 0 0 0 0 ...) <14> #<COMPILED-FUNCTION READ-SEQUENCE> - #<CHUNGA:CHUNKED-IO-STREAM #x1A22A811> - 3566 - #<COMPILED-FUNCTION CHUNGA::GET-CHUNK-SIZE> - #(#(#(NIL #<CHUNGA:CHUNKED-IO-STREAM #x1A22A811>) #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22CB5D>) #<COMPILED-FUNCTION CHUNGA::ADD-EXTENSIONS>) frame binding variables (~ = dynamically): | ~ CHUNGA::*CURRENT-ERROR-FUNCTION* <--> #<COMPILED-FUNCTION #:|86 143 (DEFMETHOD FILL-BUFFER (#) ...)-9-1-1-1|> - #(#(NIL #<CHUNGA:CHUNKED-IO-STREAM #x1A22A811>) #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22CB5D>) - #(NIL #<CHUNGA:CHUNKED-IO-STREAM #x1A22A811>) - #<COMPILED-FUNCTION #:|86 143 (DEFMETHOD FILL-BUFFER (#) ...)-9-1-1|> - #<CHUNGA:CHUNKED-IO-STREAM #x1A22A811> <15> #<COMPILED-FUNCTION #:|86 143 (DEFMETHOD FILL-BUFFER (#) ...)-9-1-1|> <16> #<STANDARD-GENERIC-FUNCTION CHUNGA::FILL-BUFFER> - #<COMPILED-FUNCTION #:|145 156 (DEFMETHOD STREAM-READ-BYTE (#) ...)-10-1-1|> - #<CHUNGA:CHUNKED-IO-STREAM #x1A22A811> <17> #<COMPILED-FUNCTION #:|145 156 (DEFMETHOD STREAM-READ-BYTE (#) ...)-10-1-1|> <18> #<STANDARD-GENERIC-FUNCTION GRAY:STREAM-READ-BYTE> - NIL - NIL - #<CHUNGA:CHUNKED-IO-STREAM #x1A22A811> <19> #<SYSTEM-FUNCTION READ-BYTE> - #<COMPILED-FUNCTION FLEXI-STREAMS::READ-BYTE*> - NIL - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> <20> #<COMPILED-FUNCTION FLEXI-STREAMS::READ-BYTE*> - #<COMPILED-FUNCTION FLEXI-STREAMS::READ-CHAR-8-BIT> - #(0 1 2 3 4 5 ... 253 254 255) - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> <21> #<COMPILED-FUNCTION FLEXI-STREAMS::READ-CHAR-8-BIT> - #<COMPILED-FUNCTION FLEXI-STREAMS::GET-CHAR-CODE> <22> #<COMPILED-FUNCTION FLEXI-STREAMS::GET-CHAR-CODE> - #<COMPILED-FUNCTION FLEXI-STREAMS::GET-CHAR-CODE> - :LF - #(#(NIL #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865>) #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) #x1A22CB99> :ISO-8859-1) - #(NIL #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865>) - #<COMPILED-FUNCTION #:|221 275 (DEFMETHOD STREAM-READ-CHAR (#) ...)-11-1-1|> - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> <23> #<COMPILED-FUNCTION #:|221 275 (DEFMETHOD STREAM-READ-CHAR (#) ...)-11-1-1|> <24> #<STANDARD-GENERIC-FUNCTION GRAY:STREAM-READ-CHAR> - NIL - 0 - #<COMPILED-FUNCTION #:|285 311 (DEFMETHOD STREAM-READ-SEQUENCE (# SEQUENCE START ...) ...)-13-1-1-1|> - 8192 - 0 - "^@^@^@^@^@..." - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> <25> #<COMPILED-FUNCTION #:|285 311 (DEFMETHOD STREAM-READ-SEQUENCE (# SEQUENCE START ...) ...)-13-1-1-1|> - #<COMPILED-FUNCTION #:|285 311 (DEFMETHOD STREAM-READ-SEQUENCE (# SEQUENCE START ...) ...)-13-1-1-1|> - #<COMPILED-FUNCTION #:|285 311 (DEFMETHOD STREAM-READ-SEQUENCE (# SEQUENCE START ...) ...)-13-1-1|> - NIL - 8192 - 0 - "^@^@^@^@^@..." - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> <26> #<COMPILED-FUNCTION #:|285 311 (DEFMETHOD STREAM-READ-SEQUENCE (# SEQUENCE START ...) ...)-13-1-1|> <27> #<STANDARD-GENERIC-FUNCTION TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE> - 8192 - 0 - #<COMPILED-FUNCTION #:|30 105 (PROGN (DEFMETHOD GRAY:STREAM-READ-BYTE-SEQUENCE # ...) (DEFMETHOD GRAY:STREAM-WRITE-BYTE-SEQUENCE # ...) ...)-11-3-1-1|> - (0 8192) - "^@^@^@^@^@..." - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> <28> #<COMPILED-FUNCTION #:|30 105 (PROGN (DEFMETHOD GRAY:STREAM-READ-BYTE-SEQUENCE # ...) (DEFMETHOD GRAY:STREAM-WRITE-BYTE-SEQUENCE # ...) ...)-11-3-1-1|> <29> #<STANDARD-GENERIC-FUNCTION GRAY:STREAM-READ-CHAR-SEQUENCE> - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> - "^@^@^@^@^@..." - 8192 - 0 - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> - "^@^@^@^@^@..." <30> #<SYSTEM-FUNCTION EXT:READ-CHAR-SEQUENCE> - #<SYSTEM-FUNCTION EXT:READ-CHAR-SEQUENCE> - CHARACTER - CHARACTER - #<COMPILED-FUNCTION READ-SEQUENCE> - #<UNBOUND> - #<UNBOUND> - NIL - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> - "^@^@^@^@^@ <31> #<COMPILED-FUNCTION READ-SEQUENCE> - NIL - 0 - "" - "^@^@^@^@^@ - T - CHARACTER - NIL - #(NIL ((:DATE . "Tue, 24 Oct 2006 02:51:03 GMT") (:SERVER . "Apache") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/html; charset=iso-8859-1"))) - #<COMPILED-FUNCTION DRAKMA::READ-BODY> - #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) #x1A22CB99> - T - ((:DATE . "Tue, 24 Oct 2006 02:51:03 GMT") (:SERVER . "Apache") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/html; charset=iso-8859-1")) - #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> <32> #<COMPILED-FUNCTION DRAKMA::READ-BODY> - NIL - #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) #x1A22CB99> - NIL - ((:DATE . "Tue, 24 Oct 2006 02:51:03 GMT") (:SERVER . "Apache") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/html; charset=iso-8859-1")) - 200 - :HTTP/1.1 - #<COMPILED-FUNCTION DRAKMA::HTTP-REQUEST-FINISH-REQUEST> - NIL - NIL <33> #<COMPILED-FUNCTION DRAKMA::HTTP-REQUEST-FINISH-REQUEST> - #<COMPILED-FUNCTION DRAKMA::HTTP-REQUEST-FINISH-REQUEST> - #<COMPILED-FUNCTION DRAKMA::HTTP-REQUEST-WRITE-HEADER> - #(#(#(NIL #<URI http://weitz.de/drakma/> NIL :HTTP/1.1 :GET NIL NIL NIL 5 (:GET :HEAD) NIL :LATIN-1 :LATIN-1 NIL NIL) #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> T) #<COMPILED-FUNCTION DRAKMA::HTTP-REQUEST-WRITE-HTTP-LINE>) unwind-protect frame - #(#(NIL #<URI http://weitz.de/drakma/> NIL :HTTP/1.1 :GET NIL NIL NIL 5 (:GET :HEAD) NIL :LATIN-1 :LATIN-1 NIL NIL) #<FLEXI-STREAMS:FLEXI-IO-STREAM #x1A22C865> T) frame binding variables (~ = dynamically): | ~ FLEXI-STREAMS:*DEFAULT-EOL-STYLE* <--> :LF - NIL - :LF compiled block frame for HTTP-REQUEST - NIL - #(NIL #<URI http://weitz.de/drakma/> NIL :HTTP/1.1 :GET NIL NIL NIL 5 (:GET :HEAD) NIL :LATIN-1 :LATIN-1 NIL NIL) - #<COMPILED-FUNCTION HTTP-REQUEST> - NIL - #<UNBOUND> - #<UNBOUND> - #<UNBOUND> - #<UNBOUND> - T - NIL - #<UNBOUND> - #<UNBOUND> - #<UNBOUND> - #<UNBOUND> - NIL - NIL - "*/*" - :DRAKMA - NIL - #<UNBOUND> - NIL - NIL - "application/x-www-form-urlencoded" - NIL - NIL - #<UNBOUND> - #<UNBOUND> - #<UNBOUND> - NIL - "http://weitz.de/drakma/" <34> #<COMPILED-FUNCTION HTTP-REQUEST> - #<COMPILED-FUNCTION HTTP-REQUEST> EVAL frame for form (HTTP-REQUEST "http://weitz.de/drakma/") frame binding environments VAR_ENV <--> NIL FUN_ENV <--> NIL BLOCK_ENV <--> NIL GO_ENV <--> NIL DECL_ENV <--> ((DECLARATION OPTIMIZE DECLARATION)) <35> ... Regards, -Anton

2 2

[drakma-devel] New version 0.5.3 (Was: [Lispweb] Announcement: Drakma)
by Edi Weitz 11 Oct '06

11 Oct '06

On Wed, 11 Oct 2006 14:17:59 +0200, Magnus Henoch <mange(a)freemail.hu> wrote: > I tested it with CLISP (CVS version, slightly post-2.40). I used > the examples on the web page; it worked fine until the binary > example. In line 121 of request.lisp, READ-SEQUENCE for some reason > extracts characters from STREAM, which it then fails to put into the > byte array RESULT: > > SYSTEM::STORE: #\G does not fit into #(0 0 0 0 0 0 0 0 0 0 ...), bad type > [Condition of type SIMPLE-TYPE-ERROR] > > The TEXTP parameter to READ-BODY is NIL, and > (flexi-streams:flexi-stream-element-type stream) returns CHARACTER. Hmm, CLISP certainly tries hard to be different from the pack... :) Anyway, please try the new version 0.5.3 and see if it makes a difference. Thanks for the report, Edi.

1 0

[drakma-devel] New version 0.5.2
by Edi Weitz 08 Oct '06

08 Oct '06

ChangeLog: Version 0.5.2 2006-10-08 Adher to user-provided content length if FORM-DATA is true Download: http://weitz.de/files/drakma.tar.gz Cheers, Edi.

1 0

[drakma-devel] New version 0.5.1
by Edi Weitz 06 Oct '06

06 Oct '06

ChangeLog: Version 0.5.1 2006-10-07 Take Content-Encoding header into account (due to a bug report by Gregory Tod) Download: http://weitz.de/files/drakma.tar.gz Cheers, Edi.

1 0

[drakma-devel] New Chunga version 0.2.0
by Edi Weitz 06 Oct '06

06 Oct '06

ChangeLog: Version 0.2.0 2006-10-06 Only wrap inner stream with flexi stream if really needed Download: http://weitz.de/files/chunga.tar.gz Cheers, Edi.

1 0

[drakma-devel] Couple of problems
by Ivan Toshkov 25 Sep '06

25 Sep '06

Hi, First of all, thank you Edi Weitz for your excellent libraries. I find them simple to use, well thought out, and with excellent documentation. Again, a big thank you. And thanks to whoever is working on porting Drakma. I found a couple of problems: Drakma cannot receive cookies from localhost. After a short investigation I found that the problem was with the method VALID-COOKIE-DOMAIN-P, which insist to have at least 2 dots in the normalized domain name. As a workaround I used 127.0.0.1 instead of localhost. The other problem is with an incomplete implementation of flexi-streams under CMUCL. The call: > (drakma:http-request "http://www.google.com/" :parameters '(("q" . "something"))) works normally, but > (drakma:http-request "http://www.google.com/search" :parameters '(("q" . "something"))) gives: No matching method for the generic function #<STANDARD-GENERIC-FUNCTION INPUT-STREAM-P (2) {58E17501}>, when called with arguments (#<FLEXI-STREAMS::VECTOR-OUTPUT-STREAM {5834F8FD}>). [Condition of type PCL::NO-APPLICABLE-METHOD-ERROR] 0: ("DEFMETHOD NO-APPLICABLE-METHOD (T)" #<#1=unused-arg> #<#1#> #<STANDARD-GENERIC-FUNCTION INPUT-STREAM-P (2) {58E17501}> (#<FLEXI-STREAMS::VECTOR-OUTPUT-STREAM {5834F8FD}>)) 1: (FLEXI-STREAMS:MAKE-FLEXI-STREAM #<FLEXI-STREAMS::VECTOR-OUTPUT-STREAM {5834F8FD}> :EXTERNAL-FORMAT :LATIN-1) 2: (FLEXI-STREAMS:STRING-TO-OCTETS "q" :EXTERNAL-FORMAT :LATIN-1 :START ...) 3: (DRAKMA::URL-ENCODE "q" :LATIN-1) 4: (DRAKMA::ALIST-TO-URL-ENCODED-STRING (("q" . "something")) :LATIN-1) 5: (DRAKMA:HTTP-REQUEST #<URI http://www.google.com/search> :PARAMETERS (("q" . "something"))) I don't have time at the moment to see if anything else is missing and to give patches. For this case though the workaround seems to be as simple as: (defmethod input-stream-p ((s flexi-streams::vector-output-stream)) nil) Cheers, Ivan P.S. I did a clean ASDF-install on Drakma and it's dependencies. I can give the precise versions if needed.

2 3

[drakma-devel] Patch for autoreferer support
by Colin Simmonds 25 Sep '06

25 Sep '06

I'd previously been using Curl through FFI bindings for a simple Lisp web client project of mine, but have now switched to Drakma as it's much easier to use. However, there was one feature in Curl that I needed which was missing, and was easily added in the attached patch. Curl has an autoreferer option, which when following redirects, automatically adds a Referer header containing the URI of the page that is being redirected from. I've added an equivalent option to the http-request function. Colin Simmonds

2 1

[drakma-devel] New version 0.4.3
by Edi Weitz 23 Sep '06

23 Sep '06

ChangeLog: Version 0.4.3 2006-09-24 Circumvent CL+SSL for AllegroCL (suggested by David Lichteblau) Download: http://weitz.de/files/drakma.tar.gz Cheers, Edi.

1 0

[drakma-devel] Compile-time bug on OpenMCL
by Robert Synnott 23 Sep '06

23 Sep '06

Compiling on OpenMCL, (both drakma itself and chunga) an error is given: "No MAKE-LOAD-FORM method is defined for [external-format stuff]". Apparently, this is something to do with OpenMCL's handling of constants: http://www.clozure.com/pipermail/openmcl-devel/2002-December/000823.html . The constant in question is +latin-1+. I found replacing all references to the constant with its value to work, but I'm sure there's a neater way. Rob

2 2