- drakma-devel@lists.common-lisp.net - mailman3.common-lisp.net

[drakma-devel] Problem with SBCL, Drakma, and quicklisp
by Patrick May 23 Mar '13

23 Mar '13

I'm trying to get Drakma working with SBCL. I've loaded it with quicklisp but when I try to connect to a website in Slime, I get this error: The function DRAKMA::WITH-CHARACTER-STREAM-SEMANTICS is undefined. [Condition of type UNDEFINED-FUNCTION] Any idea what could be causing this? I've attached the backtrace in case that helps. Thanks, Patrick Backtrace: 0: ("undefined function") 1: (HTTP-REQUEST #<PURI:URI http://ichart.yahoo.com/table.csv?s=IBM&a=0&b=01&c=2012&d=11&e=31&f=2012&g=…>) 2: (GET-YAHOO-SYMBOL-DATA "IBM" "2012-01-01" "2012-12-31") 3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (GET-YAHOO-SYMBOL-DATA "IBM" "2012-01-01" "2012-12-31") #<NULL-LEXENV>) 4: (EVAL (GET-YAHOO-SYMBOL-DATA "IBM" "2012-01-01" "2012-12-31")) 5: ((LAMBDA () :IN SWANK:EVAL-AND-GRAB-OUTPUT)) 6: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME evaluation request." #<CLOSURE (LAMBDA # :IN SWANK:EVAL-AND-GRAB-OUTPUT) {13BBF7E5}>) 7: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA # :IN SWANK:EVAL-AND-GRAB-OUTPUT) {13BBF7CD}>) 8: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:EVAL-AND-GRAB-OUTPUT "(get-yahoo-symbol-data \"IBM\" \"2012-01-01\" \"2012-12-31\")\n") #<NULL-LEXENV>) 9: (EVAL (SWANK:EVAL-AND-GRAB-OUTPUT "(get-yahoo-symbol-data \"IBM\" \"2012-01-01\" \"2012-12-31\")\n")) 10: (SWANK:EVAL-FOR-EMACS (SWANK:EVAL-AND-GRAB-OUTPUT "(get-yahoo-symbol-data \"IBM\" \"2012-01-01\" \"2012-12-31\")\n") ":comp-invest" 17) 11: (SWANK::PROCESS-REQUESTS T) 12: (SWANK::CALL-WITH-BINDINGS ..) 13: (SWANK::REPL-INPUT-STREAM-READ #<SWANK::SINGLETHREADED-CONNECTION {12971A59}> #<SB-SYS:FD-STREAM for "standard input" {11A91B11}>) 14: ((LAMBDA () :IN SB-GRAY:STREAM-READ-CHAR)) 15: ((SB-PCL::FAST-METHOD SB-GRAY:STREAM-READ-CHAR (SWANK-BACKEND::SLIME-INPUT-STREAM)) #<unavailable argument> #<unavailable argument> #<SWANK-BACKEND::SLIME-INPUT-STREAM {12971BC9}>) 16: (READ-CHAR #<SWANK-BACKEND::SLIME-INPUT-STREAM {12971BC9}> T #:EOF-OBJECT #<unused argument>) 17: (SB-IMPL::%READ-PRESERVING-WHITESPACE #<SWANK-BACKEND::SLIME-INPUT-STREAM {12971BC9}> T NIL T) 18: (SB-IMPL::%READ-PRESERVING-WHITESPACE #<SWANK-BACKEND::SLIME-INPUT-STREAM {12971BC9}> T NIL NIL) 19: (READ #<SWANK-BACKEND::SLIME-INPUT-STREAM {12971BC9}> T NIL NIL) 20: (SWANK::SIMPLE-REPL) 21: ((LAMBDA () :IN SWANK::SIMPLE-SERVE-REQUESTS)) 22: ((LAMBDA () :IN SWANK::SIMPLE-SERVE-REQUESTS)) 23: ((LAMBDA (SWANK-BACKEND::HANDLER FUNCTION) :IN "/Users/Patrick/.emlib/slime/swank-backend.lisp") ..) 24: (SWANK-BACKEND::CALL-WITH-BREAK-HOOK #<FUNCTION SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA # :IN SWANK::SIMPLE-SERVE-REQUESTS) {12971ACD}>) 25: ((FLET SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK :IN "/Users/Patrick/.emlib/slime/swank-sbcl.lisp") #<FUNCTION SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA # :IN SWANK::SIMPLE-SERVE-REQUESTS) {12971ACD}>) 26: (SWANK::SIMPLE-SERVE-REQUESTS #<SWANK::SINGLETHREADED-CONNECTION {12971A59}>) 27: (SWANK::ACCEPT-CONNECTIONS #<SB-BSD-SOCKETS:INET-SOCKET fd: -1 {129680D1}> NIL NIL) 28: ((LABELS SWANK::SERVE-LOOP :IN SWANK::SETUP-SERVER)) 29: (SWANK::SETUP-SERVER 0 #<CLOSURE (LAMBDA # :IN SWANK:START-SERVER) {128E5F1D}> NIL NIL NIL) 30: (SB-INT:SIMPLE-EVAL-IN-LEXENV (FUNCALL (READ-FROM-STRING "swank:start-server") "/var/folders/b0/br9v722s5nq0j4m677ncd98c0000gn/T/slime.32805") #<NULL-LEXENV>) 31: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..) 32: (EVAL ..) 33: (SB-EXT:INTERACTIVE-EVAL ..) 34: (SB-IMPL::REPL-FUN NIL) 35: ((LAMBDA () :IN SB-IMPL::TOPLEVEL-REPL)) 36: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #<CLOSURE (LAMBDA # :IN SB-IMPL::TOPLEVEL-REPL) {13363095}>) 37: (SB-IMPL::TOPLEVEL-REPL NIL) 38: (SB-IMPL::TOPLEVEL-INIT) 39: ((FLET #:WITHOUT-INTERRUPTS-BODY-222031 :IN SB-EXT:SAVE-LISP-AND-DIE)) 40: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))

2 4

[drakma-devel] Error while trying to POST some JSON data
by Peter Kuchta 23 Mar '13

23 Mar '13

Hi everybody, I'm trying to POST some JSON data to a web service: (ql:quickload :st-json) (ql:quickload :cl-json) (ql:quickload :drakma) (defvar *rc* (merge-pathnames (user-homedir-pathname) ".apirc")) (defvar *user* (with-open-file (s *rc*) (st-json:read-json s))) (defvar api-url (st-json:getjso "url" *user*)) (defvar api-key (st-json:getjso "key" *user*)) (defvar api-email (st-json:getjso "email" *user*)) (setf drakma:*header-stream* *standard-output*) (defvar *req* '(("dataset" . "tigge") ("step" . "24") ("date" . "20071001") ("time" . "00") ("origin" . "all"))) (format t "json:~S~%" (json:encode-json-to-string *req*)) (defun retrieve (api request) (let* ((cookie-jar (make-instance 'drakma:cookie-jar)) (extra-headers (list (cons "From" api-email) (cons "X-API-KEY" api-key))) (url (concatenate 'string api-url api "/requests")) (stream (drakma:http-request url :additional-headers extra-headers :accept "application/json" :method :post :content-type "application/json" :external-format-out :utf-8 :external-format-in :utf-8 :redirect 100 :cookie-jar cookie-jar :content (json:encode-json-to-string request) :want-stream t))) (st-json:read-json stream))) (retrieve "/datasets/tigge" *req*) Unfortunately, I get an error, although the data seems to be encoded OK to JSON and the headers generated by drakma too, I think. Apparently something is wrong with the :content (the list of integers in the errors message is just the list of ASCII codes of the JSON encoded data). json:"{\"dataset\":\"tigge\",\"step\":\"24\",\"number\":\"all\",\"levtype\":\"sl\",\"date\":\"20071001\",\"time\":\"00\",\"origin\":\"all\",\"type\":\"pf\",\"param\":\"tp\",\"area\":\"70\\/-130\\/30\\/-60\",\"grid\":\"2\\/2\",\"target\":\"data.grib\"}" POST /v1/datasets/tigge/requests HTTP/1.1 Host: api.service.int User-Agent: Drakma/1.3.0 (SBCL 1.1.5; Darwin; 12.2.0; http://weitz.de/drakma/) Accept: application/json Connection: close From: me(a)gmail.com X-API-KEY: 19a0edb6d8d8dda1e6a3b21223e4f86a Content-Type: application/json Content-Length: 193 debugger invoked on a SIMPLE-TYPE-ERROR: The value of CL+SSL::THING is #(123 34 100 97 116 97 115 101 116 34 58 34 ...), which is not of type (SIMPLE-ARRAY (UNSIGNED-BYTE 8) (*)). Any idea what's wrong with this code? Many thanks in advance. Best regards, Piotr

5 9

[drakma-devel] error while parsing set cookie containing httponly & secure
by Simon Koch 07 Jan '13

07 Jan '13

When parsing this Set-Cookie line: Set-Cookie: shssl=4058628; path=/; secure; HttpOnly The resulting Cookie is: #<COOKIE shssl=4058628; path=/; domain=www.base.de> Which misses both features: 'HttpOnly' and 'secure'. I traced the bug down to 'parse-set-cooie' which returns (("Set-Cookie: shssl" "4058628" (("path" . "/") ("secure; HttpOnly")))) instead of (("Set-Cookie: shssl" "4058628" (("path" . "/") ("secure") ("HttpOnly")))) As far as I understood the code the problem is caused by 'read-name-value-pairs' of chunga. I am using chunga 1.1.1 and drakma 1.2.9 Is this a already known bug/problem?

2 5

[drakma-devel] Drakma documentation overhaul
by Hans Hübner 09 Dec '12

09 Dec '12

Hi, while fixing some bugs in Drakma recently, I was notified that the documentation and the docstrings were not matching well anymore. I have decided to overhaul the documentation, and in the process of that, I also made some source changes. There will be more changes soon, so you may experience instabilities when you're using the master branch from github. Comments regarding the documentation would be very welcome, I have put a version up for review at http://netzhansa.com/drakma.html Thanks, Hans

1 0

[drakma-devel] charset errors question
by Jeff Cunningham 24 Sep '12

24 Sep '12

I've been running into some trouble using drakma to retrieve pages from certain commercial websites. It is very likely the HTML they are generating is broken one way or another. But the problem still remains as to how one can retrieve their pages using drakma. For example, if you try this simple case: (http-request "http://www.walmart.com") It will display the following: WARNING: Problems determining charset (falling back to binary): Corrupted Content-Type header: Read character #\;, but expected #\=. And the returned body is binary-encoded ascii. This can be converted to real ascii, of course, but it is inconvenient to say the least. Often the problem is that their metatag for the charset is simply wrong. Sometimes I can figure out what it is and supply this information, like this: (http-request "http://www.walmart.com" :external-format-in :UTF-8) and it will solve he problem. But this particular example does not lend itself to this, at least using the following charsets: :UTF-8 :UTF-7 :iso-8859-1 :iso-8859-2 :iso-8859-3 :iso-8859-4 :iso-8859-5 :iso-8859-6 :iso-8859-7 :iso-8859-8 :iso-8859-9 :BIG5 :US-ASCII :UTF-16 :UTF-32 I have no idea what their server is actually sending - it appears to be invalid for any of these charsets. Is there any way to get around this problem? Best regards, Jeff Cunningham

2 1

[drakma-devel] Drakma release 1.2.7
by Hans Hübner 17 Aug '12

17 Aug '12

Hi, I have released Drakma 1.2.7 yesterday. The following changes have been made since 1.2.6: Support :REPORT method (Cyrus Harmon) Make PRESERVE-URI work better - PURI:URI mangles paths with encoded &'s. The release tarball can be found here: https://github.com/downloads/edicl/drakma/drakma-1.2.7.tar.gz (or on the canonical URLhttp://weitz.de/files/drakma.tar.gz, which always points to the latest release). -Hans

1 0

[drakma-devel] about character set of cookie name
by lookwong 28 Jun '12

28 Jun '12

Here is a complete discussion on the character set of cookie name (for python development): http://bugs.python.org/issue2193 I think *drakma *should follow python's work. In file "util.lisp" of *drakma*, function *split-set-cookie-string* splits cookie string using function* is-valid-cookiename-char* which comes from * chunga*. But function* is-valid-cookiename-char *just follow *RFC2616* which rules that chars of cookie name are legal when they are in* token set *( see bellow)*. * token = 1*<any CHAR except CTLs or separators> separators = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\" | <"> | "/" | "[" | "]" | "?" | "=" | "{" | "}" | SP | HT * *That is not practical!* *In python, letters, digits and "!#$%&'*+-.^_`|~:"* <http://hg.python.org/cpython/rev/8cae3ee7f691#l1.15> *are legal in Cookie name.

1 0

Re: [drakma-devel] NTLM auth headers?
by Brian Connoy 12 Jun '12

12 Jun '12

Sorry to resurrect an old thread, Carlos. On Wed, Oct 14, 2009 at 12:04 AM, Carlos Konstanski <ckonstan...(a)pippiandcarlos.com<mailto:ckonstan...@pippiandcarlos.com>> wrote: > Is there any way to get drakma to write an NTLM auth header? I am > unfortunately trying to communicate with a .NET web service in an > Active Directory environment. > > If not, I suppose I could use Ã`additional-headers' to write one > myself. Were you successful in rolling your own headers? I'm stuck at the moment, figuring what needs to be done. Thanks, Brian C.

1 0

[drakma-devel] Changing content-type seems to have no effect
by Michael Minerva 15 Mar '12

15 Mar '12

I have been trying to get my lisp application to interface and upload a file to my drupal 7 website using Drakma. So far I have been able to to connect to my drupal site and authenticate and store the session cookie (thank you Ryan Davis, I just found your response the other day). Here is the code I use to do that: (with-output-to-string (stream) (setf drakma:*header-stream* stream) (let ((cookie-jar (make-instance 'drakma:cookie-jar))) (drakma:http-request "http://localhost:8888/?q=rest/user/login" :method :post :cookie-jar cookie-jar :parameters '(("username" . "login") ("password" . "password")) ) (drakma:http-request "http://localhost:8888/?q=rest/file" :method :post :cookie-jar cookie-jar :content-length t :parameters '(("file1" . #p"/Users/Mike/hello.png"));("file1" #p"/Users/Mike/Desktop/mario-gif.gif" :content-type "image/gif" :filename "mario-gif.gif")) :content-type "application/x-www-form-urlencoded" ;:parameters ))) This code successfully sends both posts but the second post does not actually upload the file (but it does return 200 OK, no errors). I have successfully uploaded to by drupal site using PHP Curl and in that code, I was only able to get it to work by using a content type of application/x-www-form-urlencoded (if I use a multipart/form-data request on my php page I get similar behavior to what I am now seeing with Drakma). This is why I added the line :content-type "application/x-www-form-urlencoded" to my second request (posted above) but it seems to have no effect. Here is a copy of the posts I am making (as you can see in the second post it still uses multipart/form-data. Is there anyway that I can upload a file using Drakma using a content-type of application/x-www-form-urlencoded (I know this is not the recommended way of making this type of request but I think it may be the only way that my drupal site will accept the file). "POST /?q=rest/user/login HTTP/1.1 Host: localhost:8888 User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/) Accept: */* Connection: close Content-Type: application/x-www-form-urlencoded Content-Length: 34 HTTP/1.1 200 OK Date: Thu, 15 Mar 2012 17:49:51 GMT Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 X-Powered-By: PHP/5.3.6 Expires: Sun, 19 Nov 1978 05:00:00 GMT Last-Modified: Thu, 15 Mar 2012 17:49:51 +0000 Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 ETag: \"1331833791\" Vary: Accept Set-Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ; expires=Sat, 07-Apr-2012 21:23:12 GMT; path=/; HttpOnly Content-Length: 115 Connection: close Content-Type: text/yaml POST /?q=rest/file HTTP/1.1 Host: localhost:8888 User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/) Accept: */* Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ Connection: close Content-Type: multipart/form-data; boundary=----------ndnZ9xjDrDpzStPEQo97xwqPHEKXAhAOd8Ho6C8P3jtKbvNINm Content-Length: 244958 HTTP/1.1 200 OK Date: Thu, 15 Mar 2012 17:49:52 GMT Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 X-Powered-By: PHP/5.3.6 Expires: Sun, 19 Nov 1978 05:00:00 GMT Last-Modified: Thu, 15 Mar 2012 17:49:52 +0000 Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 ETag: \"1331833792\" Vary: Accept Content-Length: 4 Connection: close Content-Type: text/yaml " Thanks, --Michael

2 2

[drakma-devel] Unwanted url-encoding of GET parameters.
by William Halliburton 11 Mar '12

11 Mar '12

Hello folks, I'm trying to use drakma to fetch urls that contain utf8 characters but HTTP-REQUEST automatically url encodes any non latin-1 ascii characters. On my cursory reading of the RFCs, this seems conforming behavor, but in this case it is definitely unwanted. For example, the following url http://translate.google.com/translate_tts?tl=ru&q=вы if entered directly into the browser correctly returns the text-to-speech audo file but when attempting to use HTTP-REQUEST, the url is being url encoded into http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B of which google does not url-decode and fails to return the correct data. So, for this case, the url-encoding is unwanted. I am willing to submit patch an additional argument into HTTP-REQUEST to disallow the encoding. Thoughts? Thank you, William

3 3