November 2007 - drakma-devel@common-lisp.net

[drakma-devel] Problems with a specific URI
by Vilson Vieira 27 Nov '07

27 Nov '07

Hi, first congratulations about Drakma, Weitz! I'm using it to web crawler at my work and it's running like a charm. Well, I'm getting a strange error when trying to request patents from USPTO website: (http-request "http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2F…") And the error is something like that: end of file on #<FLEXI-STREAMS::FLEXI-LATIN-1-IO-STREAM {BE2FDF1}> [Condition of type END-OF-FILE] Thanks for any suggestions. -- Vilson Vieira vilsonv(a)gmail.com

2 2

[drakma-devel] http-request hangs on utf-8 sites
by Andrei Stebakov 24 Nov '07

24 Nov '07

Calling (drakma:http-request "http://www.cl.cam.ac.uk/~mgk25/ucs/examples/digraphs.txt") just hangs. Requests to other sites works. I am using drakma-0.10.2, chunga-0.4.0, usocket-0.3.5 and SBCL 1.0.11 on Linux. Sometimes I get "broken pipe" exception, but it may be not relevant to this case. Andrew

3 3

[drakma-devel] http-request error...
by Andrei Stebakov 20 Nov '07

20 Nov '07

I run across this error time after time but it's really hard to reproduce but I can replicate exactly the same backtrace of the problem if instead of some valid http domain I choose any fake one. If I run: (drakma:http-request "http://some-fake-domain.com" :method :get :parameters `(("param1" . "text"))) The backtrace will be: The value 0 is not of type (OR (SINGLE-FLOAT (0.0)) (DOUBLE-FLOAT (0.0d0)) (INTEGER 1)). [Condition of type TYPE-ERROR] Restarts: 0: [ABORT] Return to SLIME's top level. 1: [ABORT] Exit debugger, returning to top level. Backtrace: 0: (RANDOM 0) 1: (USOCKET::GET-RANDOM-HOST-BY-NAME "some-fake-domain.com") 2: (USOCKET:SOCKET-CONNECT "some-fake-domain.com" 80) 3: (DRAKMA:HTTP-REQUEST "http://some-fake-domain.com") 4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DRAKMA:HTTP-REQUEST "http://some-fake-domain.com" :METHOD :GET :PARAMETERS '(("param1" . "text"))) #<NULL-LEXENV>) 5: ((LAMBDA ())) In my case it looks like sometimes even for the real domain it calls (RANDOM 0) which gives this log. Even if the domain is wrong do you think it should throw an exception? I use SBCL 1.0.5. with the latest drakma and usocket (usocket-0.3.2). Thank you, Andrew

3 3

[drakma-devel] Strange response message calling http-request
by Andrei Stebakov 19 Nov '07

19 Nov '07

Sometimes I get the response like "Your browser sent a request that this server could not understand". I am scratching my head where this response might be coming from. Probably not from the server I am sending a http-request to. >From what I could find on the net, this message might be coming from an Apache server. I am using hunchentoot behind apache mod_proxy, but drakma has nothing to do with it. I'd appreciate any ideas where this message might be coming from or what would be the best way to debug the problem. Thank you, Andrew

2 3

[drakma-devel] HTTPS via Proxy Server
by Kevin Crosbie 08 Nov '07

08 Nov '07

Hi, I was wondering how Drakma supports talking to a proxy server when using HTTPS connections. AllegroServe seems to make a HTTP connection to the proxy server then if the proxy server supports SSL Forwarding (For Apache, say, the conf file will read: SSLProxyEngine On) then the proxy server will make your HTTPS connection for you. This however seems to fail when SSLProxyEngine is not set to On. I've seen using WireShark that modern browsers implement a different protocol for making SSL connections via Proxy Servers... ftp://ftp.isi.edu/in-notes/rfc2817.txt Possibly: http://www.ietf.org/rfc/rfc2818.txt It seems that they first send a CONNECT request, asking for the connection to be upgraded to TLS. From then on they negotiate the connection via a tunnel directly to the destination server. So my questions are: 1. Does Drakma support this 2. Does anybody know of any Lisp HTTP Client that does? 3. Does anybody have any suggestions about how to go about doing this? For instance, I could use a Java or C library that supports this and make native calls or I could extend whatever HTTP Client I choose to support this. Best Regards, Kevin Crosbie Ravenpack International S.L.

2 1