- drakma-devel@lists.common-lisp.net - mailman3.common-lisp.net

[drakma-devel] puri error during HTTP request
by Nicolas Lamirault 15 Jun '07

15 Jun '07

hi, i make some test with Drakma and the Audioscrobbler API. for the identification, the protocol is : http://post.audioscrobbler.com/?hs=true&p=1.2& c=<client-id>&v=<client-ver>&u=<user>&t=<timestamp>&a=<auth> with auth : The algorithm for generating this token is as follows: token := md5(md5(password) + timestamp) Parse error:URI "http://post.audioscrobbler.com/?hs=true&p=1.2&c=tst&v=1.0&u=lam_&t=2007-06-… 21:29:54&a=,÷¯*²h|I3ðÈ" contains illegal character #\ at position 77. [Condition of type PURI:URI-PARSE-ERROR] Restarts: 0: [ABORT] Return to SLIME's top level. 1: [ABORT] Exit debugger, returning to top level. Backtrace: 0: (PURI::.PARSE-ERROR #<unavailable argument> #<unavailable &REST argument>) 1: (PURI::.PARSE-ERROR #<unavailable argument>) Locals: SB-DEBUG::ARG-0 = :<NOT-AVAILABLE> SB-DEBUG::ARG-1 = :<NOT-AVAILABLE> 2: ((FLET PURI::READ-TOKEN) #<unavailable argument> #<unavailable argument>) 3: (PURI::PARSE-URI-STRING #<unavailable argument>) 4: (PURI:PARSE-URI #<unavailable argument> :CLASS #<unavailable argument>) 5: (DRAKMA:HTTP-REQUEST "http://post.audioscrobbler.com/?hs=true&p=1.2&c=tst&v=1.0&u=lam_&t=2007-06-… 21:29:54&a=,÷¯*²h|I3ðÈ") someone have any idea to resolve this problem ? thanks for any help -- Nicolas Lamirault

2 1

[drakma-devel] File uploading: adding streams and functions to the list of parameters
by Andrei Stebakov 08 Jun '07

08 Jun '07

Recently I've been thinking how to make http-request upload a file, but not from the hard drive but from a function which can produce (or write to a) stream. Let's say I want to create an image with cl-gd but instead of writing the image to the file and then creating the http-request with a pathname of the file I can pass a (lambda (stream) (cl-gd-create-image stream your param1 param2)) in place of the pathname parameter. Or a stream which already has the file data. I was trying to do it using :content :continuation but found it too difficult, maybe I missed other ways. Anyway, I created a patch with very minimal changes to the make-form-data-function function which allows to do so. Also I found that I set *header-stream* to not nil, I was missing the debug output produced by the character (character only) stream from make-form-data-function which basically shows all the parameters we are sending. I added the primitive support for this based on make-broadcast-stream. Here I attach the patch for your kind consideration. Thank you, Andrew

2 1

[drakma-devel] Problem with Drakma and character encoding
by Mathias Dahl 04 Jun '07

04 Jun '07

Hi! I have a problem with drakma and character encoding. My goal is to make a small web utility which first GETs some content from a certain page on my wiki, then optionally adds prepends stuff to this content, and then POST the new content back. It works as long as only ASCII characters are involved but fails when I use characters from the higher part of Latin-1, in my case the Swedish character "ä". Below I have simplified the code so that it just GETs the old content and POST it back. What one sees is that the Swedish character is correctly handled in the GET, I see the "ä" in it's full glory, but after having posted it back, the content gets corrupt and the next time I run the function I get an error because of the strange character. Ok, enough blabbing, here is the code: ;;;;; code starts here (defvar *boundary* "-------------------------1852275791466338532535335716") (defconstant +crlf+ #.(format nil "~C~C" #\Return #\Linefeed)) (defun format-field (name value) (format nil "--~a~aContent-Disposition: form-data; name=\"~a\"~a~a~a~a" *boundary* +crlf+ name +crlf+ +crlf+ value +crlf+)) (defun foo () (let* ((old-content (drakma:http-request "http://klibb.com/cgi-bin/wiki.pl?action=browse;id=2007-05-31;raw=1")) (cookie-jar (make-instance 'drakma:cookie-jar)) (new-content (concatenate 'string (format-field "title" "2007-05-31") (format-field "text" old-content) (format-field "recent_edit" "on") (format-field "username" "MathiasDahl") "--" *boundary* "--" +crlf+))) (format t "Old content: ~a" old-content) (setf (drakma:cookie-jar-cookies cookie-jar) (list (make-instance 'drakma:cookie :name "pwd" :value "editeramera" :expires (+ (get-universal-time) 36000) :domain "klibb.com"))) (format t "New content: ~a" new-content) (drakma:http-request "http://klibb.com/cgi-bin/wiki.pl" :method :post :cookie-jar cookie-jar :content-type (format nil "multipart/form-data; boundary=~a" *boundary*) :content new-content))) ;;;;; code ends here Again, the code is simplified, some parts are hardcoded etc, but the above is enough to recreate the problem. Note that after running the code one time, you cannot test it again, because the content on the page is now changed. Here is what I get after running the function the first time: ==== * (foo) Old content: blä New content: ---------------------------1852275791466338532535335716 Content-Disposition: form-data; name="title" 2007-05-31 ---------------------------1852275791466338532535335716 Content-Disposition: form-data; name="text" blä ---------------------------1852275791466338532535335716 Content-Disposition: form-data; name="recent_edit" on ---------------------------1852275791466338532535335716 Content-Disposition: form-data; name="username" MathiasDahl ---------------------------1852275791466338532535335716-- NIL 302 ((:DATE . "Sat, 02 Jun 2007 09:30:53 GMT") (:SERVER . "Apache/2.2.3 (Mandriva Linux/PREFORK-1mdv2007.0)") (:SET-COOKIE . "MuuWiki=username%1EMathiasDahl; path=/; expires=Mon, 01-Jun-2009 09:30:53 GMT") (:LOCATION . "http://klibb.com/cgi-bin/wiki.pl/2007-05-31") (:CONTENT-LENGTH . "0") (:CONNECTION . "close") (:CONTENT-TYPE . "application/x-perl")) #<PURI:URI http://klibb.com/cgi-bin/wiki.pl> #<FLEXI-STREAMS:FLEXI-IO-STREAM {C5728E1}> T ==== As you can see, all looks well; the old content ("blä") looks like it should, and the new content looks the same (it's the data in the form field "text"). However, when I now run the function again, I get this: ==== * (foo) debugger invoked on a FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR in thread #<THREAD "initial thread" {AC14469}>: Unexpected value #xA in UTF-8 sequence. Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [USE-VALUE] Specify a character to be used instead. 1: [ABORT ] Exit debugger, returning to top level. (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR #<FLEXI-STREAMS:FLEXI-IO-STREAM {C5B37D1}> "Unexpected value #x~X in UTF-8 sequence." 10) ==== It fails because that "ä" is now something else. When I do the same thing from a browser, i.e. POST the page again and again, I don't see any problems. I have done some network sniffing with Wireshar and what I can see is that when the browser POSTs the content, the "ä" is correctly encoded in UTF-8 as xC3 xA4. In the POST done by drakma, the character is encoded xE4 (which IS the unicode code point, but not encoded as UTF-8 if I understand things correctly). At first I tried to include the encoding in Content-Type, but when I saw that it did not do any difference and also saw that Firefox does not include this, I removed it. Oh, and I should show this as well: * (sb-impl::default-external-format) :UTF-8 Just so that we are clear that I DO see the content correctly and UTF-8 is used. I also tried with a version where I even hardcoded the content to be sent to be "blä", and that gives the same problem. Maybe I should have shortened the code above to that, but what I wanted to show was that the same content I can GET nicely enough cannot be POSTed without problems. Any ideas on how I can continue debugging this? I feel kinda lost. It feels frustrating to get stuck on a problem like this when I have got the other logic to work, GETing and POSTing and stuff... I am running this in SBCL 1.0 under Mandriva GNU/Linux. Thanks! /Mathias

3 6

[drakma-devel] POST request and content-type
by Nicolas Lamirault 01 Jun '07

01 Jun '07

hello, i make some tests with Drakma. i would like to change the content-type of a POST request, but i've got always "application/x-www-form-urlencoded" : CL-USER> (drakma:http-request "http://lisp.org/" :method :post :content-type "application/atom+xml") POST / HTTP/1.1 Host: lisp.org User-Agent: Drakma/0.7.0 (SBCL 1.0; Linux; 2.6.18-4-486; http://weitz.de/drakma/) Accept: */* Connection: close Content-Type: application/x-www-form-urlencoded Content-Length: 0 HTTP/1.1 307 Temporary Redirect Date: Fri, 01 Jun 2007 10:23:23 GMT Connection: Close Server: AllegroServe/1.2.43 Transfer-Encoding: chunked LOCATION: /index.html #() 307 ((:DATE . "Fri, 01 Jun 2007 10:23:23 GMT") (:CONNECTION . "Close") (:SERVER . "AllegroServe/1.2.43") (:TRANSFER-ENCODING . "chunked") (:LOCATION . "/index.html")) #<PURI:URI http://lisp.org> #<FLEXI-STREAMS:FLEXI-IO-STREAM {AB1AEB9}> T someone knows if i make a mistake ? it is a bug ? thanks for anyhelp ... PS: very good soft ! -- Nicolas Lamirault

2 2

[drakma-devel] Help with utf-8 (SBCL 1.0.5)
by Andrei Stebakov 20 May '07

20 May '07

When I make the request like this: (drakma:http-request "http://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html") It results in error: "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"> <html xmlns=\"http://www.w3.org/1999/xhtml\"> <head><title>UTF-8 test file</title></head> <body> <p>Original by Markus Kuhn, adapted for HTML by Martin Dürst.</p> <pre> UTF-8 encoded sample plain-text file debugger invoked on a SB-INT:STREAM-ENCODING-ERROR in thread #<THREAD "initial t hread" {BBDF5A9}>: encoding error on stream #<SB-SYS:FD-STREAM for "standard output" {BBDF859}> (:EXTERNAL-FORMAT :ASCII): the character with code 8254 cannot be encoded. Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [OUTPUT-NOTHING] Skip output of this character. 1: [ABORT ] Exit debugger, returning to top level. (SB-INT:STREAM-ENCODING-ERROR #<SB-SYS:FD-STREAM for "standard output" {BBDF859}> 8254) 0] Is it some parameter that I forgot to call http-request with to make it aware of utf-8 encoding? Thank you, Andrew

3 2

[drakma-devel] xml-rpc?
by Cyrus Harmon 13 May '07

13 May '07

has anyone layered xml-rpc on top of drakma? seems like a natural thing to do. s-xml-rpc seems nice, but I don't think it supports https. Thanks, Cyrus

1 0

[drakma-devel] Re: drakma help
by Edi Weitz 10 May '07

10 May '07

Hi Cyrus, On Wed, 9 May 2007 16:27:57 -0700, Cyrus Harmon <cyrus(a)cyrusharmon.org> wrote: [Cc to mailing list.] > So I've been trying to do http-requesting with DRAKMA and I see some > weird results: > > (drakma:http-request "http://patft.uspto.gov/netacgi/nph-Parser? > Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch- > adv.htm&r=80&f=G&l=50&d=PTXT&s1=affymetrix&p=2&OS=affymetrix&RS=affymetr > ix") Seems Paul Graham's Arc server is not the only one which sends wrong line endings. Luckily, Chunga was modified to cope with that just two days ago... :) CL-USER 15 > (let ((chunga:*accept-bogus-eols* t) (drakma:*header-stream* *standard-output*)) (nth-value 1 (drakma:http-request "http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fneta…"))) GET /netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.htm&r=80&f=G&l=50&d=PTXT&s1=affymetrix&p=2&OS=affymetrix&RS=affymetrix HTTP/1.1 Host: patft.uspto.gov User-Agent: Drakma/0.7.0 (LispWorks 5.0.2; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.0 200 Script results follow Server: NetAnswer Server 1.0 Content-Type: text/html 200 Cheers, Edi.

1 0

[drakma-devel] Lenient read-line*
by Sean 07 May '07

07 May '07

I've attached a small patch which makes chunga:read-line* a little more lenient with regards to line endings. Currently (drakma:http-request "http://news.ycombinator.com") fails with an EOF since the arc server doesn't respond with CRLF but only uses plain LF. The attached patch works around this by assuming that when a LineFeed is encountered it also designates the end of a line, although it is quite possibly the wrong thing. Cheers, Sean. -- ...Please don't assume Lisp is only useful for Animation and Graphics, AI, Bioinformatics, B2B and E-Commerce, Data Mining, EDA/Semiconductor applications, Expert Systems, Finance, Intelligent Agents, Knowledge Management, Mechanical CAD, Modeling and Simulation, Natural Language, Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web Authoring just because these are the only things they happened to list. Kent Pitman.

5 14

[drakma-devel] drakma causes lisp connection close unexpectedly
by Mostafa Razavi 01 May '07

01 May '07

I'm starting to use Drakma in a program of mine, and it seems to work fine for me except in one strange case. I was trying to retrieve a page, with (drakma:http-request "http://cesa.iust.ac.ir/"), that I got "Lisp connection closed unexpectedly: exited abnormally with code 256". I set *header-stream* to see what's happening. This is the result: GET / HTTP/1.1 Host: cesa.iust.ac.ir User-Agent: Drakma/0.7.0 (SBCL 1.0.3; Linux; 2.6.20-15-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 302 Found Connection: close Date: Mon, 30 Apr 2007 22:52:19 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Location: /Default.aspx?AspxAutoDetectCookieSupport=1 Set-Cookie: AspxAutoDetectCookieSupport=1; path=/ Cache-Control: private Content-Type: text/html GET /Default.aspx?AspxAutoDetectCookieSupport=1 HTTP/1.1 Host: cesa.iust.ac.ir User-Agent: Drakma/0.7.0 (SBCL 1.0.3; Linux; 2.6.20-15-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 302 Found Connection: close Date: Mon, 30 Apr 2007 22:52:28 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Location: /(X(1)S(rmofw345wvkjbzjz54jnyu45))/Default.aspx?AspxAutoDetectCookieSupport=1 Cache-Control: private Content-Type: text/html; charset=utf-8 Content-Length: 194 GET /(X(1)S(rmofw345wvkjbzjz54jnyu45))/Default.aspx?AspxAutoDetectCookieSupport=1 HTTP/1.1 Host: cesa.iust.ac.ir User-Agent: Drakma/0.7.0 (SBCL 1.0.3; Linux; 2.6.20-15-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 200 OK Connection: close Date: Mon, 30 Apr 2007 22:52:34 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Cache-Control: private Content-Type: text/html; charset=utf-8 Content-Length: 72459 Other web pages I have tested work fine, but this is the one page I'm going to work with. Any ideas? Thanks. Homayoon, (new-comer)

4 4

Re: [drakma-devel] Lenient read-line*
by Sean 30 Apr '07

30 Apr '07

On Mon, 2007-04-30 at 11:41 +0000, rm(a)tuxteam.de wrote: > Hmm - IIRC that was causing the problems in Java way back ... imagine > the sequence "blah blah<cr>" (some network delay "<lf>blub more bah". > And then there is "<cr><cr><lf>" (which i've seen at least once!). Is > this one or two or even three newlines. Iff your answer is > 1 than you > might start reading header lines as content .... It would be 2 here, and would quite possibly leave headers unconsumed, which is in error. The current behaviour is definitely preferred for this case (signalling an error) but my feeling is that this behaviour, while quite acceptable in this example, is not the best for all cases (news.ycombinator.com for example). In the same vein the approach I've suggested is definitely not the best for all cases, merely a method to workaround servers that do not honour RFC 2616[1]. Cheers, Sean. [1] Granted this is done by ascribing semantics to single CR's and LF's in an essentially ad hoc manner. -- ...Please don't assume Lisp is only useful for Animation and Graphics, AI, Bioinformatics, B2B and E-Commerce, Data Mining, EDA/Semiconductor applications, Expert Systems, Finance, Intelligent Agents, Knowledge Management, Mechanical CAD, Modeling and Simulation, Natural Language, Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web Authoring just because these are the only things they happened to list. Kent Pitman.

1 0