June 2007 - drakma-devel@common-lisp.net - mailman3.common-lisp.net

[drakma-devel] Two problems with cookies expiry dates format
by Didier Verna 05 Jul '07

05 Jul '07

Hello, I've run into 2 formats that parse-cookie-date can't currently handle: - the first one looks like this: "Sun, 18 Jun 2006 04:01:09 GMT-07:00" I presume it means GMT minus 7 hours. - the second one is almost one of the recognized formats, except that it says PDT (Pacific Daylight Time AFAIK) instead of GMT. Note that I have no idea if these formats are standard or not. I just ran into them... -- Didier Verna, didier(a)lrde.epita.fr, http://www.lrde.epita.fr/~didier EPITA / LRDE, 14-16 rue Voltaire Tel.+33 (1) 44 08 01 85 94276 Le Kremlin-Bicêtre, France Fax.+33 (1) 53 14 59 22 didier(a)xemacs.org

2 3

[drakma-devel] Status text from request (re-post)
by H Duerer 29 Jun '07

29 Jun '07

Hi, sorry about that first email -- I am still learning gmail and hadn't realised I was rudely sending HTML. Trying again in plain text: We have a web server giving Json responses. For proper failures it returns a non-200 status code and a failure description in the status message (e.g. the status line reads "HTTP/1.0 400 Empty method name". AFAICS this status message is not passed through by Drakma's http-request. I locally made the attached changes (doing so, I found that read-status-line added the leading separating whitespace to its third value; I fixed that as well). Any chance of having this or some other means to get to the message in the official code? Holger diff -rN -u old-drakma/read.lisp new-drakma/read.lisp --- old-drakma/read.lisp 2007-06-27 12:23:17.000000000 +0100 +++ new-drakma/read.lisp 2007-06-27 12:23:17.000000000 +0100 @@ -55,7 +55,7 @@ :start (1+ first-space-pos) :end second-space-pos)) (error "Status code in ~S is not an integer." line)) - (and second-space-pos (subseq line second-space-pos))))) + (and second-space-pos (subseq line (1+ second-space-pos)))))) (defun get-content-type (headers) "Reads and parses a `Content-Type' header and returns it as @@ -124,4 +124,4 @@ (trim-whitespace (subseq string (1+ old-position) position))) while old-position when (plusp (length substring)) - collect substring)) \ No newline at end of file + collect substring)) diff -rN -u old-drakma/request.lisp new-drakma/request.lisp --- old-drakma/request.lisp 2007-06-27 12:23:17.000000000 +0100 +++ new-drakma/request.lisp 2007-06-27 12:23:17.000000000 +0100 @@ -172,15 +172,15 @@ URI is where the request is sent to, and it is either a string denoting a uniform resource identifier or a PURI:URI object. The scheme of URI must be `http' or `https'. The function returns -SIX values - the body of the reply \(but see below), the status +SEVEN values - the body of the reply \(but see below), the status code as an integer, an alist of the headers sent by the server where for each element the car \(the name of the header) is a keyword and the cdr \(the value of the header) is a string, the URI the reply comes from \(which might be different from the URI the request was sent to in case of redirects), the stream the -reply was read from, and a generalized boolean which denotes +reply was read from, a generalized boolean which denotes whether the stream should be closed \(and which you can usually -ignore). +ignore), and the message text from the response line. PROTOCOL is the HTTP protocol which is going to be used in the request line, it must be one of the keywords :HTTP/1.0 or @@ -512,15 +512,15 @@ (setf (chunked-stream-output-chunking-p (flexi-stream-stream http-stream)) nil) (finish-output http-stream) - (multiple-value-bind (server-protocol status-code) + (multiple-value-bind (server-protocol status-code status-text) ;; loop until status is NOT 100 - (loop for (server-protocol status-code nil) + (loop for (server-protocol status-code status-text) = (read-status-line http-stream *header-stream*) when (= status-code 100) ;; ignore headers sent until non-100 status is seen do (read-http-headers http-stream *header-stream*) until (/= status-code 100) - finally (return (values server-protocol status-code))) + finally (return (values server-protocol status-code status-text))) (let ((headers (read-http-headers http-stream *header-stream*)) body external-format-body) (let ((connections (header-value :connection headers))) @@ -631,7 +631,8 @@ headers uri http-stream - must-close))))) + must-close + status-text))))) (when (eq content :continuation) (return-from http-request #'finish-request)) (finish-request content))))

2 1

[drakma-devel] Status text from a requst
by H Duerer 27 Jun '07

27 Jun '07

Hi, we have a web server giving Json responses. For proper failures it returns a non-200 status code and a failure description in the status message (e.g. the status line reads "HTTP/1.0 400 Empty method name". AFAICS this status message is not passed through by Drakma's http-request. I locally made the attached changes (doing so, I found that read-status-line added the leading separating whitespace to its third value; I fixed that as well). Any chance of having this or some other means to get to the message in the official code? Holger

1 0

[drakma-devel] File uploading: adding streams and functions to the list of parameters #2
by Andrei Stebakov 17 Jun '07

17 Jun '07

I changed the pathname to file-source (after giving it some thinking and asking for help from the lisp NG, sorry if it's not the best), removed the *header-stream* related modifications, removed the tabs, updated the doc-string for http-request and the html file. (for some reason Edi's response to my previous patch didn't make to my email, which is why my new message may not be part of the original thread) Thank you, Andrew

2 1

[drakma-devel] puri error during HTTP request
by Nicolas Lamirault 15 Jun '07

15 Jun '07

hi, i make some test with Drakma and the Audioscrobbler API. for the identification, the protocol is : http://post.audioscrobbler.com/?hs=true&p=1.2& c=<client-id>&v=<client-ver>&u=<user>&t=<timestamp>&a=<auth> with auth : The algorithm for generating this token is as follows: token := md5(md5(password) + timestamp) Parse error:URI "http://post.audioscrobbler.com/?hs=true&p=1.2&c=tst&v=1.0&u=lam_&t=2007-06-… 21:29:54&a=,÷¯*²h|I3ðÈ" contains illegal character #\ at position 77. [Condition of type PURI:URI-PARSE-ERROR] Restarts: 0: [ABORT] Return to SLIME's top level. 1: [ABORT] Exit debugger, returning to top level. Backtrace: 0: (PURI::.PARSE-ERROR #<unavailable argument> #<unavailable &REST argument>) 1: (PURI::.PARSE-ERROR #<unavailable argument>) Locals: SB-DEBUG::ARG-0 = :<NOT-AVAILABLE> SB-DEBUG::ARG-1 = :<NOT-AVAILABLE> 2: ((FLET PURI::READ-TOKEN) #<unavailable argument> #<unavailable argument>) 3: (PURI::PARSE-URI-STRING #<unavailable argument>) 4: (PURI:PARSE-URI #<unavailable argument> :CLASS #<unavailable argument>) 5: (DRAKMA:HTTP-REQUEST "http://post.audioscrobbler.com/?hs=true&p=1.2&c=tst&v=1.0&u=lam_&t=2007-06-… 21:29:54&a=,÷¯*²h|I3ðÈ") someone have any idea to resolve this problem ? thanks for any help -- Nicolas Lamirault

2 1

[drakma-devel] File uploading: adding streams and functions to the list of parameters
by Andrei Stebakov 08 Jun '07

08 Jun '07

Recently I've been thinking how to make http-request upload a file, but not from the hard drive but from a function which can produce (or write to a) stream. Let's say I want to create an image with cl-gd but instead of writing the image to the file and then creating the http-request with a pathname of the file I can pass a (lambda (stream) (cl-gd-create-image stream your param1 param2)) in place of the pathname parameter. Or a stream which already has the file data. I was trying to do it using :content :continuation but found it too difficult, maybe I missed other ways. Anyway, I created a patch with very minimal changes to the make-form-data-function function which allows to do so. Also I found that I set *header-stream* to not nil, I was missing the debug output produced by the character (character only) stream from make-form-data-function which basically shows all the parameters we are sending. I added the primitive support for this based on make-broadcast-stream. Here I attach the patch for your kind consideration. Thank you, Andrew

2 1

[drakma-devel] Problem with Drakma and character encoding
by Mathias Dahl 04 Jun '07

04 Jun '07

Hi! I have a problem with drakma and character encoding. My goal is to make a small web utility which first GETs some content from a certain page on my wiki, then optionally adds prepends stuff to this content, and then POST the new content back. It works as long as only ASCII characters are involved but fails when I use characters from the higher part of Latin-1, in my case the Swedish character "ä". Below I have simplified the code so that it just GETs the old content and POST it back. What one sees is that the Swedish character is correctly handled in the GET, I see the "ä" in it's full glory, but after having posted it back, the content gets corrupt and the next time I run the function I get an error because of the strange character. Ok, enough blabbing, here is the code: ;;;;; code starts here (defvar *boundary* "-------------------------1852275791466338532535335716") (defconstant +crlf+ #.(format nil "~C~C" #\Return #\Linefeed)) (defun format-field (name value) (format nil "--~a~aContent-Disposition: form-data; name=\"~a\"~a~a~a~a" *boundary* +crlf+ name +crlf+ +crlf+ value +crlf+)) (defun foo () (let* ((old-content (drakma:http-request "http://klibb.com/cgi-bin/wiki.pl?action=browse;id=2007-05-31;raw=1")) (cookie-jar (make-instance 'drakma:cookie-jar)) (new-content (concatenate 'string (format-field "title" "2007-05-31") (format-field "text" old-content) (format-field "recent_edit" "on") (format-field "username" "MathiasDahl") "--" *boundary* "--" +crlf+))) (format t "Old content: ~a" old-content) (setf (drakma:cookie-jar-cookies cookie-jar) (list (make-instance 'drakma:cookie :name "pwd" :value "editeramera" :expires (+ (get-universal-time) 36000) :domain "klibb.com"))) (format t "New content: ~a" new-content) (drakma:http-request "http://klibb.com/cgi-bin/wiki.pl" :method :post :cookie-jar cookie-jar :content-type (format nil "multipart/form-data; boundary=~a" *boundary*) :content new-content))) ;;;;; code ends here Again, the code is simplified, some parts are hardcoded etc, but the above is enough to recreate the problem. Note that after running the code one time, you cannot test it again, because the content on the page is now changed. Here is what I get after running the function the first time: ==== * (foo) Old content: blä New content: ---------------------------1852275791466338532535335716 Content-Disposition: form-data; name="title" 2007-05-31 ---------------------------1852275791466338532535335716 Content-Disposition: form-data; name="text" blä ---------------------------1852275791466338532535335716 Content-Disposition: form-data; name="recent_edit" on ---------------------------1852275791466338532535335716 Content-Disposition: form-data; name="username" MathiasDahl ---------------------------1852275791466338532535335716-- NIL 302 ((:DATE . "Sat, 02 Jun 2007 09:30:53 GMT") (:SERVER . "Apache/2.2.3 (Mandriva Linux/PREFORK-1mdv2007.0)") (:SET-COOKIE . "MuuWiki=username%1EMathiasDahl; path=/; expires=Mon, 01-Jun-2009 09:30:53 GMT") (:LOCATION . "http://klibb.com/cgi-bin/wiki.pl/2007-05-31") (:CONTENT-LENGTH . "0") (:CONNECTION . "close") (:CONTENT-TYPE . "application/x-perl")) #<PURI:URI http://klibb.com/cgi-bin/wiki.pl> #<FLEXI-STREAMS:FLEXI-IO-STREAM {C5728E1}> T ==== As you can see, all looks well; the old content ("blä") looks like it should, and the new content looks the same (it's the data in the form field "text"). However, when I now run the function again, I get this: ==== * (foo) debugger invoked on a FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR in thread #<THREAD "initial thread" {AC14469}>: Unexpected value #xA in UTF-8 sequence. Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [USE-VALUE] Specify a character to be used instead. 1: [ABORT ] Exit debugger, returning to top level. (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR #<FLEXI-STREAMS:FLEXI-IO-STREAM {C5B37D1}> "Unexpected value #x~X in UTF-8 sequence." 10) ==== It fails because that "ä" is now something else. When I do the same thing from a browser, i.e. POST the page again and again, I don't see any problems. I have done some network sniffing with Wireshar and what I can see is that when the browser POSTs the content, the "ä" is correctly encoded in UTF-8 as xC3 xA4. In the POST done by drakma, the character is encoded xE4 (which IS the unicode code point, but not encoded as UTF-8 if I understand things correctly). At first I tried to include the encoding in Content-Type, but when I saw that it did not do any difference and also saw that Firefox does not include this, I removed it. Oh, and I should show this as well: * (sb-impl::default-external-format) :UTF-8 Just so that we are clear that I DO see the content correctly and UTF-8 is used. I also tried with a version where I even hardcoded the content to be sent to be "blä", and that gives the same problem. Maybe I should have shortened the code above to that, but what I wanted to show was that the same content I can GET nicely enough cannot be POSTed without problems. Any ideas on how I can continue debugging this? I feel kinda lost. It feels frustrating to get stuck on a problem like this when I have got the other logic to work, GETing and POSTing and stuff... I am running this in SBCL 1.0 under Mandriva GNU/Linux. Thanks! /Mathias

3 6

[drakma-devel] POST request and content-type
by Nicolas Lamirault 01 Jun '07

01 Jun '07

hello, i make some tests with Drakma. i would like to change the content-type of a POST request, but i've got always "application/x-www-form-urlencoded" : CL-USER> (drakma:http-request "http://lisp.org/" :method :post :content-type "application/atom+xml") POST / HTTP/1.1 Host: lisp.org User-Agent: Drakma/0.7.0 (SBCL 1.0; Linux; 2.6.18-4-486; http://weitz.de/drakma/) Accept: */* Connection: close Content-Type: application/x-www-form-urlencoded Content-Length: 0 HTTP/1.1 307 Temporary Redirect Date: Fri, 01 Jun 2007 10:23:23 GMT Connection: Close Server: AllegroServe/1.2.43 Transfer-Encoding: chunked LOCATION: /index.html #() 307 ((:DATE . "Fri, 01 Jun 2007 10:23:23 GMT") (:CONNECTION . "Close") (:SERVER . "AllegroServe/1.2.43") (:TRANSFER-ENCODING . "chunked") (:LOCATION . "/index.html")) #<PURI:URI http://lisp.org> #<FLEXI-STREAMS:FLEXI-IO-STREAM {AB1AEB9}> T someone knows if i make a mistake ? it is a bug ? thanks for anyhelp ... PS: very good soft ! -- Nicolas Lamirault

2 2