October 2009 - drakma-devel@common-lisp.net

[drakma-devel] GET :PARAMETERS added again during redirects
by Eugene Ossintsev 12 Nov '09

12 Nov '09

I use DRAKMA 1.0, Clozure CL 1.4-dev (Ubuntu Linux x86). According to the DRAKMA documentation, parameters is an alist of name/value pairs (the car and the cdr each being a string) which denotes the parameters which are added to the query part of the URI or (in the case of a POST request) comprise the request body. In my example QUOTES1 sends a GET request passing the request parameters in the URI, and QUOTES2 uses the :PARAMETERS instead. The results, unfortunately, are not the same. (defun quotes1 () (multiple-value-bind (body code) (drakma:http-request "http://finance.yahoo.com/d/quotes.csv?s=XIC.TO&f=sxl1d1t1") (when (= code 200) (split-sequence:split-sequence #\, body)))) ("\"XIC.TO\"" "\"Toronto\"" "18.03" "\"10/9/2009\"" "\"3:59pm\" ^M ") <=== It's fine :-) (defun quotes2 () (multiple-value-bind (body code) (drakma:http-request "http://finance.yahoo.com/d/quotes.csv" :method :get :parameters '(("s" . "XIC.TO") ("f" . "sxl1d1t1"))) (when (= code 200) (split-sequence:split-sequence #\, body)))) ("\"XIC.TO\"" "\"Toronto\"" "18.03" "\"10/9/2009\"" "\"3:59pm\"" "\"XIC.TO\"" "\"XIC.TO\"" "\"XIC.TO\"" "\"Toronto\"" "18.03" "\"10/9/2009\"" "\"3:59pm\" ^M \"XIC.TO\"" "\"Toronto\"" "18.03" "\"10/9/2009\"" "\"3:59pm\"" "\"XIC.TO\"" "\"XIC.TO\"" "\"XIC.TO\"" "\"Toronto\"" "18.03" "\"10/9/2009\"" "\"3:59pm\" ^M ") <=== It's a mess :-( TRACE DRAKMA:HTTP-REQUEST shows that in case of redirects of GET requests the parameters from :PARAMETERS are added again to the URI. Thus, the same parameters may be added again and again, giving unpredictable results. 0> Calling (DRAKMA:HTTP-REQUEST "http://finance.yahoo.com/d/quotes.csv" :PARAMETERS (("s" . "XIC.TO") ("f" . "sxl1d1t1"))) 1> Calling (DRAKMA:HTTP-REQUEST #<URI http://download.finance.yahoo.com/d/quotes.csv?s=XIC.TO&f=sxl1d1t1> :REDIRECT 4 :STREAM NIL :ADDITIONAL-HEADERS NIL :PARAMETERS (("s" . "XIC.TO") ("f" . "sxl1d1t1"))) <1 DRAKMA:HTTP-REQUEST returned 7 values : <1 #(34 88 73 67 46 84 79 34 44 34 84 111 114 111 110 116 111 34 44 49 55 46 57 56 44 34 49 48 .......... <1 200 <1 ((:DATE . "Wed, 14 Oct 2009 05:00:51 GMT") (:P3P . "policyref=\"http://p3p.yahoo.com/w3c/p3p.xml\", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV\"") (:CACHE-CONTROL . "private") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "application/octet-stream")) <1 #<URI http://download.finance.yahoo.com/d/quotes.csv?s=XIC.TO&f=sxl1d1t1&s=XIC.TO…> <1 #<FLEXI-STREAMS:FLEXI-IO-STREAM #x15616B76> <1 T <1 "OK" Is it what may be expected from such usage of :PARAMETERS in GET requests? If not, is it worth fixing it? - Eugene

2 2

[drakma-devel] [patch] timeouts, take two
by Stanislaw Halik 19 Oct '09

19 Oct '09

Hey Edi, I made a new patch that uses TRIVIAL-TIMEOUT instead of usocket WAIT-FOR-INPUT. Results are great, it yields no slowdown at all when downloading a 10MB zero-filled file. Patch follows. I can't attach files due to posting to Gmane through tin(1) so it's included in the message body. If it gets mangled in the transmission, it's alternatively available at http://tehran.lain.pl/stuff/20090827-drakma-timeout-gray-stream-take2.diff diff -rN -u old-drakma/drakma.asd new-drakma/drakma.asd --- old-drakma/drakma.asd 2009-08-27 05:59:20.000000000 +0200 +++ new-drakma/drakma.asd 2009-08-27 05:59:20.000000000 +0200 @@ -53,10 +53,13 @@ (:file "util") (:file "read") (:file "cookies") + #-lispworks (:file "timeouts") (:file "request")) :depends-on (:puri :cl-base64 :chunga :flexi-streams + #-lispworks #:trivial-timeout + #-lispworks #:trivial-gray-streams #-:lispworks :usocket #-(or :lispworks :allegro) :cl+ssl)) diff -rN -u old-drakma/request.lisp new-drakma/request.lisp --- old-drakma/request.lisp 2009-08-27 05:59:20.000000000 +0200 +++ new-drakma/request.lisp 2009-08-27 05:59:20.000000000 +0200 @@ -200,9 +200,11 @@ force-binary want-stream stream - #+:lispworks (connection-timeout 20) - #+:lispworks (read-timeout 20) - #+(and :lispworks (not :lw-does-not-have-write-timeout)) + (connection-timeout 20) + (read-timeout 20) + #+(or (not :lispworks) + (and :lispworks + (not :lw-does-not-have-write-timeout))) (write-timeout 20 write-timeout-provided-p) #+openmcl deadline) @@ -385,6 +387,7 @@ time units. If the server fails to respond until that time, a COMMUNICATION-DEADLINE-EXPIRED condition is signalled. DEADLINE is only available on CCL 1.2 and later." + #-lispworks (declare (ignore write-timeout-provided-p)) (unless (member protocol '(:http/1.0 :http/1.1) :test #'eq) (parameter-error "Don't know how to handle protocol ~S." protocol)) (setq uri (cond ((uri-p uri) (copy-uri uri)) @@ -450,12 +453,16 @@ :write-timeout write-timeout :errorp t) #-:lispworks - (usocket:socket-stream - (usocket:socket-connect host port - :element-type 'octet - #+openmcl #+openmcl - :deadline deadline - :nodelay t)))) + (let ((sock (trivial-timeout:with-timeout (connection-timeout) + (usocket:socket-connect host port + :element-type 'octet + #+openmcl #+openmcl + :deadline deadline + :nodelay t)))) + (if (or read-timeout write-timeout) + (usocket-timeout:timeout-stream-for-socket + sock :read-timeout read-timeout :write-timeout write-timeout) + (usocket:socket-stream sock))))) #+openmcl (when deadline ;; it is correct to set the deadline here even though diff -rN -u old-drakma/timeouts.lisp new-drakma/timeouts.lisp --- old-drakma/timeouts.lisp 1970-01-01 01:00:00.000000000 +0100 +++ new-drakma/timeouts.lisp 2009-08-27 05:59:20.000000000 +0200 @@ -0,0 +1,62 @@ +(defpackage #:usocket-timeout + (:use #:cl #:usocket #:trivial-timeout #:trivial-gray-streams) + (:shadowing-import-from #:trivial-timeout #:timeout-error) + (:export #:timeout-stream-for-socket)) + +(in-package #:usocket-timeout) + + +(defclass timeout-mixin () + ((socket-of :initarg :socket + :reader socket-of))) + +(defclass timeout-input-stream (trivial-gray-stream-mixin + fundamental-binary-input-stream + timeout-mixin) + ((read-timeout :initform nil + :initarg :read-timeout + :reader read-timeout-of))) + +(defmethod stream-read-sequence ((stream timeout-input-stream) + sequence start end &key) + (with-timeout ((read-timeout-of stream)) + (read-sequence sequence (socket-of stream) :start start :end end))) + +(defmethod stream-read-byte ((stream timeout-input-stream)) + (with-timeout ((read-timeout-of stream)) + (read-byte (socket-of stream)))) + + +(defclass timeout-output-stream (trivial-gray-stream-mixin + fundamental-binary-output-stream + timeout-mixin) + ((write-timeout :initform nil + :initarg :write-timeout + :reader write-timeout-of))) + +(defmethod stream-finish-output ((stream timeout-output-stream)) + (finish-output (socket-of stream))) + +(defmethod stream-write-sequence ((stream timeout-output-stream) + sequence start end &key) + (with-timeout ((write-timeout-of stream)) + (write-sequence sequence (socket-of stream) + :start (or start 0) + :end end))) + +(defmethod stream-write-byte ((stream timeout-output-stream) integer) + (with-timeout ((write-timeout-of stream)) + (write-byte integer (socket-of stream)))) + + +(defclass usocket-timeout-stream (timeout-input-stream timeout-output-stream) + ()) + +(defun timeout-stream-for-socket (socket &key timeout + (read-timeout timeout) + (write-timeout timeout)) + (make-instance 'usocket-timeout-stream + :socket (socket-stream socket) + :read-timeout read-timeout + :write-timeout write-timeout)) +

2 2

[drakma-devel] NTLM auth headers?
by Carlos Konstanski 14 Oct '09

14 Oct '09

Is there any way to get drakma to write an NTLM auth header? I am unfortunately trying to communicate with a .NET web service in an Active Directory environment. If not, I suppose I could use Ã`additional-headers' to write one myself. Carlos Konstanski

2 1

[drakma-devel] drakma:get-cookies fails to parse cookies with commas in values
by Andrei Stebakov 07 Oct '09

07 Oct '09

One of the web sites started to give me cookies with commas and drakma:get-cookies just crashes on those requests. I distilled my case into a small example like this: (drakma::get-cookies '((:CONTENT-TYPE . "text/html; charset=utf-8") (:LOCATION . "http://www.test") (:SERVER . "Microsoft-IIS/7.0") (:CONTENT-LENGTH . "46") (:DATE . "Sat, 12 Sep 2009 14:58:04 GMT") (:CONNECTION . "close") (:SET-COOKIE . "domain=test.com; expires=Thu, 12-Sep-2109 14:58:04 GMT; session=6,Direct,placeholder,test.com;") (:CACHE-CONTROL . "private")) (puri:parse-uri "http://www.test.com")) It'll throw an exception trying to parse "session=6,Direct,placeholder, test.com" pair and will complain about the commas. I tried to capture the same page with FF Live Http Headers and it has no problems with that. Do you think we could change drakma to be able to digest it as well? Thank you, Andrei

2 5

[drakma-devel] URL-encode and rfc3986
by Red Daly 05 Oct '09

05 Oct '09

Drakma encodes spaces in URLs as pluses instead of "%20"s. It also leaves the characters "$-_.!*'()" unencoded instead of the RFC3986 unencoded set: "-_.~". This is okay for tolerant servers but for other applications it is a bug. For example, in OAuth it is necessary to conform to RFC3986 or the signature will be incorrect for the request. To make Drakma more RFC3986-compliant, replace these four lines in util.lisp: (find char "$-_.!*'()," :test #'char=)) (write-char char out)) ((char= char #\Space) (write-char #\+ out)) with these two: (find char "-_.~" :test #'char=)) (write-char char out)) Best regards, Red

2 1