July 2008 - drakma-devel@common-lisp.net - mailman3.common-lisp.net

[drakma-devel] dotless cookie domains
by Daniel Janus 16 Jul '08

16 Jul '08

Hello list, The current version of Drakma contains a check of cookie domains' validity. A (textual) domain is considered to be valid iff either it contains a dot or is exactly "localhost". I'm sure there is a reason for this (perhaps some part of a relevant RFC?), but I got bitten by it in a real-world scenario. I was using Drakma for testing of a Java EE-based web application at my company, accessible at the internal address "http://someserver:9090/gui/app". The relevant headers returned are: DRAKMA> (http-request "http://someserver:9090/gui/app") ==> "blah blah blah, some HTML" 200 ((:X-POWERED-BY . "Servlet/2.5") (:SERVER . "Sun Java System Application Server 9.1") (:SET-COOKIE . "JSESSIONID=41165991cdf70add1d9e0a103cfc; Path=/gui") (:CONTENT-TYPE . "text/html;charset=UTF-8") (:DATE . "Tue, 15 Jul 2008 00:14:24 GMT") (:CONNECTION . "close")) #<URI http://someserver:9090/gui/app> #<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {BB78A59}> T "OK" and if I pass a cookie jar to such HTTP-REQUEST, the returned cookie is not included in the jar due to the checking machinery. FWIW, Firefox 3 does include it (and so the app works on Firefox 3, but fails the Drakma tests). I quickly hacked up a patch (attached) to Drakma which adds a special variable *ALLOW-DOTLESS-COOKIE-DOMAINS-P*, which, when set to non-NIL, causes the domains like SOMESERVER to be accepted. Might it be useful to include in the official distribution? Best regards, -- Daniel Janus <daniel(a)sentivision.com> Quality Assurance Specialist, SENTIVISION Mb: +48 501 180 396 | Ph: +48 22 640 0860

2 3

Re: [drakma-devel] Bug report: Overeager encoding of parameters
by Eric Benson 07 Jul '08

07 Jul '08

Ah, I overlooked the :CONTENT keyword. There's a strong argument for not adding another keyword! So, there is a way to avoid re-encoding or de-encoding strings: build a URL for a GET request, or build a CONTENT block for a POST request. Not as convenient, but doable. And of course, it turned out I didn't need it anyway. Thanks again. ----- Original Message ---- From: Edi Weitz <edi(a)agharta.de> To: General interest list for Drakma and Chunga <drakma-devel(a)common-lisp.net> Sent: Monday, July 7, 2008 11:55:14 AM Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters On Mon, 7 Jul 2008 10:07:15 -0700 (PDT), Eric Benson <eric_a_benson(a)yahoo.com> wrote: > Building your own URL only works for GET requests, not for POST > requests. http://weitz.de/drakma/#content _______________________________________________ drakma-devel mailing list drakma-devel(a)common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

1 0

Re: [drakma-devel] Bug report: Overeager encoding of parameters
by Eric Benson 07 Jul '08

07 Jul '08

Building your own URL only works for GET requests, not for POST requests. I don't need to stop drakma from encoding query strings for my application, I can pass it the string with spaces and let it encode them to plus signs. I'm suggesting a useful, minor, upward-compatible addition to drakma's API. Perhaps drakma's interface is too high-level for the kind of application that might need more direct control of encoding and decoding. I can understand not wanting to add another keyword to http-request and another few lines to the documentation. That's your judgement call. Trivial-http and cl-curl are also available. Anyway, thanks for listening, and thanks for the code... ----- Original Message ---- From: Edi Weitz <edi(a)agharta.de> To: General interest list for Drakma and Chunga <drakma-devel(a)common-lisp.net> Sent: Monday, July 7, 2008 9:41:29 AM Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters On Mon, 7 Jul 2008 09:08:18 -0700 (PDT), Eric Benson <eric_a_benson(a)yahoo.com> wrote: > RFC 2396 has been superseded by RFC 3986, although that does not > affect this case. RFC 2396 is what is quoted in RFC 2616, so that's why I was referring to it. > http://www.w3.org/Addressing/URL/4_URI_Recommentations.html > > "Within the query string, the plus sign is reserved as shorthand > notation for a space. Therefore, real plus signs must be encoded. That's what Drakma does. > For the record, neither IE nor Firefox encodes plus signs when they > are entered as part of the query. They encode them if they are GET parameters, like Drakma. Open the attached HTML page in IE or Firefox and enter "a+b", then press the Return key. > It seems to me that http-request should make encoding of the > parameters optional, since a client program may need to deal with > encoded as well as unencoded strings. Otherwise, a caller of > http-request that already has encoded parameters must explicitly > decode them before calling. No, you are free to build the URL for Drakma yourself. Edi. -----Inline Attachment Follows-----

2 1

Re: [drakma-devel] Bug report: Overeager encoding of parameters
by Eric Benson 07 Jul '08

07 Jul '08

RFC 2396 has been superseded by RFC 3986, although that does not affect this case. The freedb server is using plus signs in HTTP requests to substitute for spaces, which are used in CDDBP requests. That appears to be supported by this recommendation from W3: http://www.w3.org/Addressing/URL/4_URI_Recommentations.html "Within the query string, the plus sign is reserved as shorthand notation for a space. Therefore, real plus signs must be encoded. This method was used to make query URIs easier to pass in systems which did not allow spaces." For the record, neither IE nor Firefox encodes plus signs when they are entered as part of the query. However, they probably should! Spaces are encoded as plus signs. If I left the spaces in the CDDB query alone when passing them to http-request, they would be encoded as plus signs. It seems to me that http-request should make encoding of the parameters optional, since a client program may need to deal with encoded as well as unencoded strings. Otherwise, a caller of http-request that already has encoded parameters must explicitly decode them before calling. A keyword argument that specifies whether parameters are already encoded, defaulting to nil, would be a useful addition. Most HTTP client APIs separate the encoding of URLs and query strings from the network request, leaving it up to the caller to handle the encoding explicitly. * (drakma:http-request "http://freedb.org/~cddb/cddb.cgi" :parameters '(("cmd" . "cddb lscat") ("hello" . "user host freedb 1.0") ("proto" . "6"))) GET /~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6 HTTP/1.1 Host: freedb.org User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 200 OK Date: Mon, 07 Jul 2008 16:02:17 GMT Server: Apache/2.0.54 (Debian GNU/Linux) Expires: Mon Jul 7 18:02:17 2008 Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=UTF-8 "210 OK, category list follows (until terminating `.') data folk jazz misc rock country blues newage reggae classical soundtrack . " 200 ((:DATE . "Mon, 07 Jul 2008 16:02:17 GMT") (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)") (:EXPIRES . "Mon Jul 7 18:02:17 2008") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8")) #<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&…> #<FLEXI-STREAMS:FLEXI-IO-STREAM {BAE7859}> T "OK" * ----- Original Message ---- From: Edi Weitz <edi(a)agharta.de> To: General interest list for Drakma and Chunga <drakma-devel(a)common-lisp.net> Sent: Monday, July 7, 2008 7:41:47 AM Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters On Mon, 7 Jul 2008 07:28:13 -0700 (PDT), Eric Benson <eric_a_benson(a)yahoo.com> wrote: > It appears that drakma is encoding plus signs (+) in query > parameters, turning them into %2B when they should be left alone. According to 3.4 of RFC 2396 the character "+" is reserved in query components. Encoding plus signs in GET requests is also something Firefox and Internet Explorer do. > This confuses the server in this case. Maybe the server is confused anyway? Edi. _______________________________________________ drakma-devel mailing list drakma-devel(a)common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

2 1

[drakma-devel] Bug report: Overeager encoding of parameters
by Eric Benson 07 Jul '08

07 Jul '08

It appears that drakma is encoding plus signs (+) in query parameters, turning them into %2B when they should be left alone. This confuses the server in this case. You can see that if I use the :parameters keyword the request fails, but if I include the same parameters as part of the URL the request succeeds. * (setq drakma:*header-stream* *standard-output*) #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {91260B9}> * (drakma:http-request "http://freedb.org/~cddb/cddb.cgi" :parameters '(("cmd" . "cddb+lscat") ("hello" . "user+host+freedb+1.0") ("proto" . "6"))) GET /~cddb/cddb.cgi?cmd=cddb%2Blscat&hello=user%2Bhost%2Bfreedb%2B1.0&proto=6 HTTP/1.1 Host: freedb.org User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 200 OK Date: Mon, 07 Jul 2008 14:17:36 GMT Server: Apache/2.0.54 (Debian GNU/Linux) Expires: Mon Jul 7 16:17:37 2008 Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=UTF-8 "500 Command syntax error: incorrect arg count forhandshake. " 200 ((:DATE . "Mon, 07 Jul 2008 14:17:36 GMT") (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)") (:EXPIRES . "Mon Jul 7 16:17:37 2008") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8")) #<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb%2Blscat&hello=user%2Bhost%2Bfreed…> #<FLEXI-STREAMS:FLEXI-IO-STREAM {BA65021}> T "OK" * (drakma:http-request "http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&…") GET /~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6 HTTP/1.1 Host: freedb.org User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/) Accept: */* Connection: close HTTP/1.1 200 OK Date: Mon, 07 Jul 2008 14:17:42 GMT Server: Apache/2.0.54 (Debian GNU/Linux) Expires: Mon Jul 7 16:17:42 2008 Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=UTF-8 "210 OK, category list follows (until terminating `.') data folk jazz misc rock country blues newage reggae classical soundtrack . " 200 ((:DATE . "Mon, 07 Jul 2008 14:17:42 GMT") (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)") (:EXPIRES . "Mon Jul 7 16:17:42 2008") (:CONNECTION . "close") (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8")) #<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&…> #<FLEXI-STREAMS:FLEXI-IO-STREAM {BAAA461}> T "OK" *

2 1