[drakma-devel] Re: [patch] drakma, support for connection-timeout on SBCL
I've updated the patch to support the IO-TIMEOUT keyword argument. It's not possible to non-intrusively support read- and write-timeouts separately to bring the API in congruency with LispWorks. The resolver error gets changed to SIMPLE-ERROR because it's not an ERROR, but a CONDITION (signalled with ERROR, though). I've stress-tested the patch, running 30 threads doing HTTP-REQUEST. My rationale for including the patch - it doesn't change the program logic much, and except for the function definition and socket connection there's no special-casing for SBCL. The timeout option is crucial for me: my typical usage is running concurrent threads accessing URLs posted in spam to make the senders think I'm a used accessing the page, in turn sending more spam, addresses of which land in my spamtrap. Some of the URLs point to broken httpds, leaving an open connection without sending any replies. This leads to unfinished requests using threads and hanging for days. If you feel like maintaing a piece of code for SBCL, by all means please do so. I think other SBCL users might benefit from it. -- /\ / Jabber ID :: sthalik@jabber.autocom.pl \ \/ Unix stuff :: http://tehran.lain.pl \/\ Yet Another RBL :: http://rbl.lain.pl
On Sun, Dec 23, 2007, Stanislaw Halik wrote:
[...]
As for the reason I haven't submitted the patch to usocket itself: usocket is a compatibility layer, comprising of features readily available in most of CL implementations. With socket timeouts supported so far by only for LispWorks and SBCL, I believe the patch would have no chances of getting accepted. -- /\ / Jabber ID :: sthalik@jabber.autocom.pl \ \/ Unix stuff :: http://tehran.lain.pl \/\ Yet Another RBL :: http://rbl.lain.pl
On Sun, 23 Dec 2007 06:13:43 +0100, Stanislaw Halik <sthalik@tehran.lain.pl> wrote:
As for the reason I haven't submitted the patch to usocket itself: usocket is a compatibility layer, comprising of features readily available in most of CL implementations. With socket timeouts supported so far by only for LispWorks and SBCL, I believe the patch would have no chances of getting accepted.
Let's do it like this: 1. Ask them. That doesn't cost anything. They can still offer to support timeouts and make that a no-op for implementations which don't have them (like Drakam does). (Also, are you sure it's only LispWorks and SBCL? Have you checked AllegroCL for example?) 2. If they say yes, that's fine and I'll update Drakma to work with the latest usocket release. 3. If they say no, let's talk about integrating your patch into Drakma. Actually, I hope they say yes. Several people tried to convince me to switch Drakma from trivial-sockets to usocket because there's active development over there...
On Dec 23, 2007 11:09 PM, Edi Weitz <edi@agharta.de> wrote:
On Sun, 23 Dec 2007 06:13:43 +0100, Stanislaw Halik <sthalik@tehran.lain.pl> wrote:
As for the reason I haven't submitted the patch to usocket itself: usocket is a compatibility layer, comprising of features readily available in most of CL implementations. With socket timeouts supported so far by only for LispWorks and SBCL, I believe the patch would have no chances of getting accepted.
Let's do it like this:
1. Ask them. That doesn't cost anything. They can still offer to support timeouts and make that a no-op for implementations which don't have them (like Drakam does). (Also, are you sure it's only LispWorks and SBCL? Have you checked AllegroCL for example?)
2. If they say yes, that's fine and I'll update Drakma to work with the latest usocket release.
Well, I think they'll accept (a patch which fits into the usocket framework), especially since timeouts come for free in Allegro and CLISP (through a with-timeout macro and a :timeout parameter respectively). Research should be able to turn up ways to achieve the same thing in other lisps. In a portability library you generally can't be expected to implement a feature for all supported platforms. However: the more you submit with the original patch, the bigger the chances for integration. bye, Erik.
On Wed, 26 Dec 2007 15:25:54 +0100, "Erik Huelsmann" <ehuels@gmail.com> wrote:
Well, I think they'll accept (a patch which fits into the usocket framework),
To the OP: In case that wasn't clear - Erik is the maintainer of usocket. So, if /he/ says so...
especially since timeouts come for free in Allegro and CLISP
And LispWorks. See :TIMEOUT, :READ-TIMEOUT, and :WRITE-TIMEOUT here: http://www.lispworks.com/documentation/lw50/LWRM/html/lwref-35.htm Edi.
On Dec 26, 2007 6:00 PM, Edi Weitz <edi@agharta.de> wrote:
On Wed, 26 Dec 2007 15:25:54 +0100, "Erik Huelsmann" <ehuels@gmail.com> wrote:
Well, I think they'll accept (a patch which fits into the usocket framework),
To the OP: In case that wasn't clear - Erik is the maintainer of usocket. So, if /he/ says so...
especially since timeouts come for free in Allegro and CLISP
And LispWorks.
Right, I meant "in addition to the LispWorks and SBCL timeouts which are implemented in the patch". Actually, if I re-read the original posting and the reason for the time-outs, I think I have it already implemented in the usocket trunk (except for SBCL/Win32): usocket trunk supports a select() like interface to determine which sockets are available for reading. With that code, you could implement the 10 socket readers in 1 thread instead of in 10. The interface returns the sockets which are ready for reading. If the timeout exceeds, an empty list is returned. The interface is called WAIT-FOR-INPUT. The idea is that you only start reading the response until you have input on the socket available. I understand this is not exactly the same solution as the route you have taken, but it's available today on many more lisps than the 2 you implemented so far. Do you think this approach might work for you? The only problem is that I fixed the LispWorks/Win32 code last weekend and haven't had the chance to commit it yet. (Meaning that I did implement: ECL/Unix, ECL/Windows, Allegro, SBCL/Unix, CMUCL, ArmedBear, CLISP, Clozure CL (aka OpenMCL). Douglas Crosher implemented Scieneer CL.) I could use someone with a real life application to test the library. bye, Erik.
On Wed, Dec 26, 2007, Erik Huelsmann wrote:
usocket trunk supports a select() like interface to determine which sockets are available for reading. With that code, you could implement the 10 socket readers in 1 thread instead of in 10. The interface returns the sockets which are ready for reading. If the timeout exceeds, an empty list is returned. The interface is called WAIT-FOR-INPUT.
How about wrapping the socket in a gray stream when timeouts are used, and doing WAIT-FOR-INPUT before read operations? That sounds better than changing every place Drakma and Chunga read from sockets. -- /\ / Jabber ID :: sthalik@jabber.autocom.pl \ \/ Unix stuff :: http://tehran.lain.pl \/\ Yet Another RBL :: http://rbl.lain.pl
On Fri, Dec 28, 2007, Stanislaw Halik wrote:
On Wed, Dec 26, 2007, Erik Huelsmann wrote:
usocket trunk supports a select() like interface to determine which sockets are available for reading. With that code, you could implement the 10 socket readers in 1 thread instead of in 10. The interface returns the sockets which are ready for reading. If the timeout exceeds, an empty list is returned. The interface is called WAIT-FOR-INPUT.
How about wrapping the socket in a gray stream when timeouts are used, and doing WAIT-FOR-INPUT before read operations? That sounds better than changing every place Drakma and Chunga read from sockets.
This can't be done this easily. WAIT-FOR-INPUT interacts badly with buffering. Consider this: CL-USER> (usocket:socket-connect "ananke" 25) #<USOCKET:STREAM-USOCKET {B550781}> CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3) (#<USOCKET:STREAM-USOCKET {B550781}>) 3 CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3) (#<USOCKET:STREAM-USOCKET {B550781}>) 3 CL-USER> (read-char (usocket:socket-stream (car '(#<USOCKET:STREAM-USOCKET {B550781}>)))) #\2 CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3) NIL NIL The SMTP banner has already been slurped into the buffer and select(2) returns 0. This could work with :buffering :none. usocket doesn't give an option as whether to enable input buffering. As for read timeout support for usocket, only SBCL and LispWorks support it. CLISP has WITH-TIMEOUT, but that would require wrapping the socket in a gray stream with WITH-TIMEOUT. Note that WITH-TIMEOUT in CLISP depends on threads, not enabled by default nor in Debian package. -- /\ / Jabber ID :: sthalik@jabber.autocom.pl \ \/ Unix stuff :: http://tehran.lain.pl \/\ Yet Another RBL :: http://rbl.lain.pl
On Dec 28, 2007 10:53 PM, Stanislaw Halik <sthalik@tehran.lain.pl> wrote:
On Fri, Dec 28, 2007, Stanislaw Halik wrote:
On Wed, Dec 26, 2007, Erik Huelsmann wrote:
usocket trunk supports a select() like interface to determine which sockets are available for reading. With that code, you could implement the 10 socket readers in 1 thread instead of in 10. The interface returns the sockets which are ready for reading. If the timeout exceeds, an empty list is returned. The interface is called WAIT-FOR-INPUT.
How about wrapping the socket in a gray stream when timeouts are used, and doing WAIT-FOR-INPUT before read operations? That sounds better than changing every place Drakma and Chunga read from sockets.
This can't be done this easily. WAIT-FOR-INPUT interacts badly with buffering. Consider this:
In the LispWorks variant (on Windows) I realised this problem and I've implemented something that takes buffering into account. I'll need to backport that. I think however that this subject is starting to be more about usocket than about Drakma, so, possibly we need to switch this discussion to usocket-devel@ ? I'd gladly work with you (and others) to make WAIT-FOR-INPUT usable.
CL-USER> (usocket:socket-connect "ananke" 25) #<USOCKET:STREAM-USOCKET {B550781}>
CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3) (#<USOCKET:STREAM-USOCKET {B550781}>) 3 CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3) (#<USOCKET:STREAM-USOCKET {B550781}>) 3
CL-USER> (read-char (usocket:socket-stream (car '(#<USOCKET:STREAM-USOCKET {B550781}>)))) #\2
CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3) NIL NIL
The SMTP banner has already been slurped into the buffer and select(2) returns 0. This could work with :buffering :none. usocket doesn't give an option as whether to enable input buffering.
Right. I haven't decided what to do about access to buffering streams (or at least setting the parameter), because in general this can only be an advisory parameter (not all implementations support setting 'bufferedness' of a stream).
As for read timeout support for usocket, only SBCL and LispWorks support it. CLISP has WITH-TIMEOUT, but that would require wrapping the socket in a gray stream with WITH-TIMEOUT. Note that WITH-TIMEOUT in CLISP depends on threads, not enabled by default nor in Debian package.
The UNIX socket FAQ (http://www.developerweb.net/forum/archive/index.php/t-3439.html) suggests that it's nearly impossible to correctly implement timeouts using SO_RCVTIMEO and SO_SENDTIMEO in a portable manner. Too many differences occur on different platforms... That's why I rather use select()/LISTEN/READ-CHAR/READ-CHAR-NO-HANG/etc. to implement time-out behaviour which can be used on all lisps on all OSes. I hope you don't mind (and that you - as I do - think that the result is what counts). Thanks for taking the time to test this new usocket behaviour! bye, Erik.
On Sat, Dec 29, 2007, Erik Huelsmann wrote:
That's why I rather use select()/LISTEN/READ-CHAR/READ-CHAR-NO-HANG/etc. to implement time-out behaviour which can be used on all lisps on all OSes. I hope you don't mind (and that you - as I do - think that the result is what counts).
I have implemented timeouts for non-LispWorks lisps. Latest usocket trunk and working usocket:wait-for-input required. Patch attached. -- /\ / Jabber ID :: sthalik@jabber.autocom.pl \ \/ Unix stuff :: http://tehran.lain.pl \/\ Yet Another RBL :: http://rbl.lain.pl
On Sat, 29 Dec 2007 16:32:48 +0100, Stanislaw Halik <sthalik@tehran.lain.pl> wrote:
I have implemented timeouts for non-LispWorks lisps. Latest usocket trunk and working usocket:wait-for-input required.
Patch attached.
Ugh, does that mean usocket fully supports timeouts now? I must have missed parts of the discussion. Still, I'd rather wait until there's an official usocket release, so people who install Drakma can use ASDF-INSTALL if they want.
On Dec 29, 2007 6:49 PM, Edi Weitz <edi@agharta.de> wrote:
On Sat, 29 Dec 2007 16:32:48 +0100, Stanislaw Halik <sthalik@tehran.lain.pl> wrote:
I have implemented timeouts for non-LispWorks lisps. Latest usocket trunk and working usocket:wait-for-input required.
Patch attached.
Ugh, does that mean usocket fully supports timeouts now? I must have missed parts of the discussion.
I don't think you missed parts of the discussion: usocket supports time-outs only in a way supported by select(). This means we can only wait for the signal "there's input". This is different than the first patch submitted in this thread which waits a certain amount of time for a specified minimum amount of data. When reading about the reason *why* he wants to implement timeouts is not because he's getting incomplete data, but some servers are broken and send no response at all (but still hang on to the connection!). This use-case can be solved without any send/receive time-outs. In order to be able to use the select() approach with Drakma, Stanislaw implemented a stream which waits for *some* data to become available within the specified timeout period. If it does not happen, it raises an error, if it does, it just reads the data from the network and returns it to Drakma.
Still, I'd rather wait until there's an official usocket release, so people who install Drakma can use ASDF-INSTALL if they want.
That'd be my preferred approach too. (Although testing trunk code is highly appreciated, ofcourse!) I have found in the past it's hard to manage a project which has dependencies on non-released code, if you don't own the dependency project too. bye, Erik. PS: I think Stanislaw can use his patch and usocket trunk without problems, it's just that I'd prefer not to have projects depend on unreleased usocket code - yet.
participants (3)
-
Edi Weitz -
Erik Huelsmann -
Stanislaw Halik