- drakma-devel@lists.common-lisp.net - mailman3.common-lisp.net

[drakma-devel] :close nil
by Stas Boukarev 27 Aug '08

27 Aug '08

When I call http-request with :close nil argument it takes 6 seconds more: CL-USER> (time (drakma:http-request "http://192.168.1.2/")) Evaluation took: 0.013 seconds of real time 0.008001 seconds of total run time (0.008001 user, 0.000000 system) 61.54% CPU 17,442,126 processor cycles 233,168 bytes consed "..." 200 ((:CONNECTION . "close") (:CONTENT-LENGTH . "917") (:CONTENT-TYPE . "text/html; charset=utf-8") (:DATE . "Mon, 25 Aug 2008 12:46:20 GMT") (:SERVER . "Hunchentoot 0.15.7")) #<PURI:URI http://192.168.1.2> #<FLEXI-STREAMS:FLEXI-IO-STREAM {1005E78681}> T "OK" CL-USER> (time (drakma:http-request "http://192.168.1.2/" :close nil)) Evaluation took: 6.428 seconds of real time 0.488030 seconds of total run time (0.400025 user, 0.088005 system) [ Run times consist of 0.240 seconds GC time, and 0.249 seconds non-GC time. ] 7.59% CPU 212 lambdas converted 994,309,191 processor cycles 18,388,176 bytes consed "..." 200 ((:CONTENT-LENGTH . "917") (:CONTENT-TYPE . "text/html; charset=utf-8") (:DATE . "Mon, 25 Aug 2008 12:46:04 GMT") (:SERVER . "Hunchentoot 0.15.7")) #<PURI:URI http://192.168.1.2> #<FLEXI-STREAMS:FLEXI-IO-STREAM {100799E001}> NIL "OK" Drakma and all dependancies are latest, tested with SBCL and CCL, and different web servers. -- With Best Regards, Stas.

3 10

[drakma-devel] ACL can't find WITH-CHARACTER-STREAM-SEMANTICS
by Chaitanya Gupta 21 Aug '08

21 Aug '08

Hi, When I try to call HTTP-REQUEST in ACL 8/OS X (using the dev version of DRAKMA), I get the following error/backtrace: attempt to call `DRAKMA::WITH-CHARACTER-STREAM-SEMANTICS' which is an undefined function. [Condition of type UNDEFINED-FUNCTION] Restarts: 0: [TRY-AGAIN] Try calling DRAKMA::WITH-CHARACTER-STREAM-SEMANTICS again. 1: [RETURN-VALUE] Return a value instead of calling DRAKMA::WITH-CHARACTER-STREAM-SEMANTICS. 2: [USE-VALUE] Try calling a function other than DRAKMA::WITH-CHARACTER-STREAM-SEMANTICS. 3: [STORE-VALUE] Setf the symbol-function of DRAKMA::WITH-CHARACTER-STREAM-SEMANTICS and call it again. 4: [ABORT-REQUEST] Abort handling SLIME request. 5: [ABORT] Abort entirely from this (lisp) process. Backtrace: 0: (SWANK::DEBUG-IN-EMACS #<UNDEFINED-FUNCTION @ #x108d7f42>) 1: (SWANK:SWANK-DEBUGGER-HOOK #<UNDEFINED-FUNCTION @ #x108d7f42> #<Function SWANK-DEBUGGER-HOOK>) 2: (ERROR #<UNDEFINED-FUNCTION @ #x108d7f42>) 3: (DRAKMA::WITH-CHARACTER-STREAM-SEMANTICS #(60 104 116 109 108 62 60 104 101 97 ...)) 4: ((LABELS DRAKMA:HTTP-REQUEST DRAKMA::FINISH-REQUEST) NIL) 5: (DRAKMA:HTTP-REQUEST #<PURI:URI http://www.google.co.in> :REDIRECT 4 :STREAM NIL :ADDITIONAL-HEADERS NIL) 6: ((LABELS DRAKMA:HTTP-REQUEST DRAKMA::FINISH-REQUEST) NIL) 7: (DRAKMA:HTTP-REQUEST "http://www.google.com") Couldn't locate the definition of WITH-CHARACTER-STREAM-SEMANTICS anywhere in the drakma source code. Am I missing some dependency? Chaitanya

3 5

[drakma-devel] Problem with USER-AGENT string on ACL8/OS X
by Chaitanya Gupta 19 Aug '08

19 Aug '08

Hi, These are the request headers that drakma sends when I run HTTP-REQUEST in ACL8 on OS X. GET / HTTP/1.1 Host: localhost User-Agent: #<Printer Error, obj=#x10000685: NIL is not a package>Accept: */* Connection: close The problem is that DRAKMA::USER-AGENT-STRING in this case returns a string with a tilde inside it i.e. "...~1/RELEASE_I386...". Cheers, Chaitanya

2 3

[drakma-devel] Query string construction, ALIST-TO-URL-ENCODED-STRING
by Maciej Pasternacki 19 Aug '08

19 Aug '08

Hello all, I use Drakma in my cl-openid project, and I need to construct query string on the fly. I need to pass along the URI objects with the query string, and finally I do not only call HTTP-REQUEST, but also call #'HUNCHENTOOT:REDIRECT on them, or just return those URLs as part of a message. Drakma allows to build query string only directly as part of HTTP-REQUEST, and does not make the functionality available to user. At the moment I use unexported ALIST-TO-URL-ENCODED-STRING, but I'd like to ask if there is any possibility of having ALIST-TO-URL-ENCODED-STRING and URL-ENCODE exported as part of official API? It would be certainly useful, and I think I am not the only one who needs to manipulate URIs and query strings. Regards, Maciej. -- -><- Maciej 'japhy' Pasternacki -><- http://www.pasternacki.net/ -><-

2 1

[drakma-devel] dotless cookie domains
by Daniel Janus 16 Jul '08

16 Jul '08

Hello list, The current version of Drakma contains a check of cookie domains' validity. A (textual) domain is considered to be valid iff either it contains a dot or is exactly "localhost". I'm sure there is a reason for this (perhaps some part of a relevant RFC?), but I got bitten by it in a real-world scenario. I was using Drakma for testing of a Java EE-based web application at my company, accessible at the internal address "http://someserver:9090/gui/app". The relevant headers returned are: DRAKMA> (http-request "http://someserver:9090/gui/app") ==> "blah blah blah, some HTML" 200 ((:X-POWERED-BY . "Servlet/2.5") (:SERVER . "Sun Java System Application Server 9.1") (:SET-COOKIE . "JSESSIONID=41165991cdf70add1d9e0a103cfc; Path=/gui") (:CONTENT-TYPE . "text/html;charset=UTF-8") (:DATE . "Tue, 15 Jul 2008 00:14:24 GMT") (:CONNECTION . "close")) #<URI http://someserver:9090/gui/app> #<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {BB78A59}> T "OK" and if I pass a cookie jar to such HTTP-REQUEST, the returned cookie is not included in the jar due to the checking machinery. FWIW, Firefox 3 does include it (and so the app works on Firefox 3, but fails the Drakma tests). I quickly hacked up a patch (attached) to Drakma which adds a special variable *ALLOW-DOTLESS-COOKIE-DOMAINS-P*, which, when set to non-NIL, causes the domains like SOMESERVER to be accepted. Might it be useful to include in the official distribution? Best regards, -- Daniel Janus <daniel(a)sentivision.com> Quality Assurance Specialist, SENTIVISION Mb: +48 501 180 396 | Ph: +48 22 640 0860

2 3

Re: [drakma-devel] Bug report: Overeager encoding of parameters
by Eric Benson 07 Jul '08

07 Jul '08

Ah, I overlooked the :CONTENT keyword. There's a strong argument for not adding another keyword! So, there is a way to avoid re-encoding or de-encoding strings: build a URL for a GET request, or build a CONTENT block for a POST request. Not as convenient, but doable. And of course, it turned out I didn't need it anyway. Thanks again. ----- Original Message ---- From: Edi Weitz <edi(a)agharta.de> To: General interest list for Drakma and Chunga <drakma-devel(a)common-lisp.net> Sent: Monday, July 7, 2008 11:55:14 AM Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters On Mon, 7 Jul 2008 10:07:15 -0700 (PDT), Eric Benson <eric_a_benson(a)yahoo.com> wrote: > Building your own URL only works for GET requests, not for POST > requests. http://weitz.de/drakma/#content _______________________________________________ drakma-devel mailing list drakma-devel(a)common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

1 0

Re: [drakma-devel] Bug report: Overeager encoding of parameters
by Eric Benson 07 Jul '08

07 Jul '08

Building your own URL only works for GET requests, not for POST requests. I don't need to stop drakma from encoding query strings for my application, I can pass it the string with spaces and let it encode them to plus signs. I'm suggesting a useful, minor, upward-compatible addition to drakma's API. Perhaps drakma's interface is too high-level for the kind of application that might need more direct control of encoding and decoding. I can understand not wanting to add another keyword to http-request and another few lines to the documentation. That's your judgement call. Trivial-http and cl-curl are also available. Anyway, thanks for listening, and thanks for the code... ----- Original Message ---- From: Edi Weitz <edi(a)agharta.de> To: General interest list for Drakma and Chunga <drakma-devel(a)common-lisp.net> Sent: Monday, July 7, 2008 9:41:29 AM Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters On Mon, 7 Jul 2008 09:08:18 -0700 (PDT), Eric Benson <eric_a_benson(a)yahoo.com> wrote: > RFC 2396 has been superseded by RFC 3986, although that does not > affect this case. RFC 2396 is what is quoted in RFC 2616, so that's why I was referring to it. > http://www.w3.org/Addressing/URL/4_URI_Recommentations.html > > "Within the query string, the plus sign is reserved as shorthand > notation for a space. Therefore, real plus signs must be encoded. That's what Drakma does. > For the record, neither IE nor Firefox encodes plus signs when they > are entered as part of the query. They encode them if they are GET parameters, like Drakma. Open the attached HTML page in IE or Firefox and enter "a+b", then press the Return key. > It seems to me that http-request should make encoding of the > parameters optional, since a client program may need to deal with > encoded as well as unencoded strings. Otherwise, a caller of > http-request that already has encoded parameters must explicitly > decode them before calling. No, you are free to build the URL for Drakma yourself. Edi. -----Inline Attachment Follows-----

2 1

Re: [drakma-devel] Bug report: Overeager encoding of parameters
by Eric Benson 07 Jul '08

07 Jul '08

RFC 2396 has been superseded by RFC 3986, although that does not affect this case. The freedb server is using plus signs in HTTP requests to substitute for spaces, which are used in CDDBP requests. That appears to be supported by this recommendation from W3: http://www.w3.org/Addressing/URL/4_URI_Recommentations.html "Within the query string, the plus sign is reserved as shorthand notation for a space. Therefore, real plus signs must be encoded. This method was used to make query URIs easier to pass in systems which did not allow spaces." For the record, neither IE nor Firefox encodes plus signs when they are entered as part of the query. However, they probably should! Spaces are encoded as plus signs. If I left the spaces in the CDDB query alone when passing them to http-request, they would be encoded as plus signs. It seems to me that http-request should make encoding of the parameters optional, since a client program may need to deal with encoded as well as unencoded strings. Otherwise, a caller of http-request that already has encoded parameters must explicitly decode them before calling. A keyword argument that specifies whether parameters are already encoded, defaulting to nil, would be a useful addition. Most HTTP client APIs separate the encoding of URLs and query strings from the network request, leaving it up to the caller to handle the encoding explicitly. * (drakma:http-request "http://freedb.org/~cddb/cddb.cgi" :parameters '(("cmd" . "cddb lscat") ("hello" . "user host freedb 1.0") ("proto" . "6"))) GET /~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6 HTTP/1.1 Host: freedb.org User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 200 OK Date: Mon, 07 Jul 2008 16:02:17 GMT Server: Apache/2.0.54 (Debian GNU/Linux) Expires: Mon Jul 7 18:02:17 2008 Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=UTF-8 "210 OK, category list follows (until terminating `.') data folk jazz misc rock country blues newage reggae classical soundtrack . " 200 ((:DATE . "Mon, 07 Jul 2008 16:02:17 GMT") (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)") (:EXPIRES . "Mon Jul 7 18:02:17 2008") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8")) #<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&…> #<FLEXI-STREAMS:FLEXI-IO-STREAM {BAE7859}> T "OK" * ----- Original Message ---- From: Edi Weitz <edi(a)agharta.de> To: General interest list for Drakma and Chunga <drakma-devel(a)common-lisp.net> Sent: Monday, July 7, 2008 7:41:47 AM Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters On Mon, 7 Jul 2008 07:28:13 -0700 (PDT), Eric Benson <eric_a_benson(a)yahoo.com> wrote: > It appears that drakma is encoding plus signs (+) in query > parameters, turning them into %2B when they should be left alone. According to 3.4 of RFC 2396 the character "+" is reserved in query components. Encoding plus signs in GET requests is also something Firefox and Internet Explorer do. > This confuses the server in this case. Maybe the server is confused anyway? Edi. _______________________________________________ drakma-devel mailing list drakma-devel(a)common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

2 1

[drakma-devel] Bug report: Overeager encoding of parameters
by Eric Benson 07 Jul '08

07 Jul '08

It appears that drakma is encoding plus signs (+) in query parameters, turning them into %2B when they should be left alone. This confuses the server in this case. You can see that if I use the :parameters keyword the request fails, but if I include the same parameters as part of the URL the request succeeds. * (setq drakma:*header-stream* *standard-output*) #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {91260B9}> * (drakma:http-request "http://freedb.org/~cddb/cddb.cgi" :parameters '(("cmd" . "cddb+lscat") ("hello" . "user+host+freedb+1.0") ("proto" . "6"))) GET /~cddb/cddb.cgi?cmd=cddb%2Blscat&hello=user%2Bhost%2Bfreedb%2B1.0&proto=6 HTTP/1.1 Host: freedb.org User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 200 OK Date: Mon, 07 Jul 2008 14:17:36 GMT Server: Apache/2.0.54 (Debian GNU/Linux) Expires: Mon Jul 7 16:17:37 2008 Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=UTF-8 "500 Command syntax error: incorrect arg count forhandshake. " 200 ((:DATE . "Mon, 07 Jul 2008 14:17:36 GMT") (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)") (:EXPIRES . "Mon Jul 7 16:17:37 2008") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8")) #<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb%2Blscat&hello=user%2Bhost%2Bfreed…> #<FLEXI-STREAMS:FLEXI-IO-STREAM {BA65021}> T "OK" * (drakma:http-request "http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&…") GET /~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6 HTTP/1.1 Host: freedb.org User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 200 OK Date: Mon, 07 Jul 2008 14:17:42 GMT Server: Apache/2.0.54 (Debian GNU/Linux) Expires: Mon Jul 7 16:17:42 2008 Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=UTF-8 "210 OK, category list follows (until terminating `.') data folk jazz misc rock country blues newage reggae classical soundtrack . " 200 ((:DATE . "Mon, 07 Jul 2008 14:17:42 GMT") (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)") (:EXPIRES . "Mon Jul 7 16:17:42 2008") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8")) #<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&…> #<FLEXI-STREAMS:FLEXI-IO-STREAM {BAAA461}> T "OK" *

2 1

[drakma-devel] Clozure CL Page Specific Error
by Wade Humeniuk 12 Jun '08

12 Jun '08

I am having trouble with Clozure CL. I am not sure this is the right mailing list as it seems to be a CCL problem (unexpected end-of-file). This same request works with LW 5.0.1 and SBCL 1.0.15 on the same machine. CCL also works with other urls like " https://bmo.com". The failing url is being served by an apache web server. I have noticed similiar problems have been posted to this mailing list. (setting chunga:*accept-bogus-eols* does not help). CL-USER> (lisp-implementation-version) "Version 1.2-r9226-RC1 (LinuxX8664)" CL-USER> CL-USER> (drakma:http-request "https://esqa.moneris.com") Unexpected end of file on #<SSL-STREAM for 4> [Condition of type END-OF-FILE] Restarts: 0: [ABORT] Return to SLIME's top level. 1: [ABORT-BREAK] Reset this process 2: [ABORT] Kill this process Backtrace: 0: (READ-BYTE #<SSL-STREAM for 4> 'T 'NIL) 1: (#<CCL::STANDARD-KERNEL-METHOD STREAM-READ-VECTOR (CCL::BINARY-INPUT-STREAM T T T)> #<SSL-STREAM for 4> #(60 72 84 77 76 62 10 60 72 69 ...) 0 2355) 2: (READ-SEQUENCE #(60 72 84 77 76 62 10 60 72 69 ...) #<SSL-STREAM for 4> ':START 0 ':END 2355) 3: (READ-SEQUENCE #(60 72 84 77 76 62 10 60 72 69 ...) #<CHUNGA:CHUNKED-IO-STREAM #x30004141A8FD> ':START 0 ':END 2355) 4: ((:INTERNAL FLEXI-STREAMS::FILL-BUFFER (TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE (FLEXI-STREAMS:FLEXI-INPUT-STREAM T T T))) 2355) 5: (#<STANDARD-METHOD TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE (FLEXI-STREAMS:FLEXI-INPUT-STREAM T T T)> #<FLEXI-STREAMS:FLEXI-IO-STREAM #x30004129C14D> 6: (CCL::%%STANDARD-COMBINED-METHOD-DCODE '((#<STANDARD-METHOD TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE :BEFORE (FLEXI-STREAMS:FLEXI-IO-STREAM T T T)>) NIL #<STANDARD-METHOD TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE (FLEXI-STREAMS:FLEXI-INPUT-STREAM T T T)>) 17575205018776) 7: (READ-SEQUENCE <edited> 8: (DRAKMA::READ-BODY #<FLEXI-STREAMS:FLEXI-IO-STREAM #x30004129C14D> '((:DATE . "Tue, 10 Jun 2008 04:15:25 GMT") (:SERVER . "Apache") (:LAST-MODIFIED . "Thu, 08 Nov 2007 15:41:34 GMT") (:ETAG . "\"42951-931-b3656780\"") (:ACCEPT-RANGES . "bytes") (:CONTENT-LENGTH . "2353") (:CONNECTION . "close") (:CONTENT-TYPE . "text/html; charset=UTF-8")) 'T #<FLEXI-UTF-8-FORMAT (:UTF-8 :EOL-STYLE :LF) #x30004129BCCD>) 9: ((:INTERNAL DRAKMA::FINISH-REQUEST DRAKMA:HTTP-REQUEST) 'NIL 'NIL) 10: (DRAKMA:HTTP-REQUEST #<URI https://esqa.moneris.com>) 11: (CCL::CALL-CHECK-REGS 'DRAKMA:HTTP-REQUEST "https://esqa.moneris.com") 12: (CCL::RUN-PROCESS-INITIAL-FORM #<PROCESS worker(7) [Active] #x30004130E5FD> '(#<COMPILED-LEXICAL-CLOSURE (:INTERNAL CCL::%PROCESS-RUN-FUNCTION) #x30004130E48F>)) 13: ((:INTERNAL CCL::%PROCESS-PRESET-INTERNAL) #<PROCESS worker(7) [Active] #x30004130E5FD> '(#<COMPILED-LEXICAL-CLOSURE (:INTERNAL CCL::%PROCESS-RUN-FUNCTION) #x30004130E48F>)) 14: ((:INTERNAL CCL::THREAD-MAKE-STARTUP-FUNCTION))

2 3