Re: [usocket-devel] [drakma-devel] [patch] Resubmit drakma timeout for sbcl.
Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings: 1. When doing TCP work, it means a "connection timeout"; 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP); The TIMEOUT keyword arguments was added from 0.4.0 (released on Oct 2008, more than two years ago), so I think Drakma should try to use it. However, it may not work on all supported platforms -- I remembered. But this should be considered as bug, I'll try to fix it after a small rewrite of our unit test framework. On the other hand, for "read timeout" and "write timeout", they're changeable properties (or options) for any socket objects. A new API called SOCKET-OPTION (learnt from CLISP) should be added in next USOCKET major version. Dynamically setting options like "broadcast" and "reuse address" should also be added under SOCKET-OPTION. I see hunchentoot has a function called SET-TIMEOUTS, which can be used to set "read timeout" and "write timeout" for any exist usocket, I want to merge it into usocket under the new SOCKET-OPTION API. --binghe 在 2011-3-28，01:50， Edi Weitz 写道：
2011/3/27 Chun Tian (binghe) <binghe.lisp@gmail.com>:
Could you tell me what exact timeout feature they want? Setting Read/Write timeout when doing SOCKET-CONNECT? or Apply new timeout values to exist usocket object?
The LW version of Drakma supports connection timeouts, read timeouts, and write timeouts. Existing usocket objects wouldn't be modified.
Cheers, Edi.
2011/3/27 Chun Tian (binghe) <binghe.lisp@gmail.com>:
Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings:
1. When doing TCP work, it means a "connection timeout"; 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP);
For SBCL backend at least this is not correct. There :TIMEOUT is a read timeout for any single blocking read from the socket. Connections and writes are unaffected -- though SBCL changes may apply the timeouts to writes as well. Waiting for a connection is not affected -- but adding support for that is pretty easy. Sidenote: I haven't really looked at usocket's SBCL backend beyond SOCKET-CONNECT prior to this, but a quick peek shows that it's pretty cavalier about using SBCL internals. This is not good. Internals _will_ change, and then usocket will break. Specifically, SB-UNIX package is an internal implementation package -- using it is bad. SB-POSIX is the supported POSIX api. Some SB-FOO::BAR stuff in there as well. :/ Cheers, -- Nikodemus
Hi, Nikodemus 在 2011-3-28，04:26， Nikodemus Siivola 写道：
2011/3/27 Chun Tian (binghe) <binghe.lisp@gmail.com>:
Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings:
1. When doing TCP work, it means a "connection timeout"; 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP);
For SBCL backend at least this is not correct.
I see. But for other backends (at least LW and CCL), seems correct. Among all CL platforms, Clozure CL have the most feature-rich networking API. Its MAKE-SOCKET [1] provided three keyword arguments: INPUT-TIMEOUT, OUTPUT-TIMEOUT and CONNECT-TIMEOUT, and an additional DEADLINE. LispWorks only has TCP support, but its OPEN-TCP-STREAM [2] also provided three keyword arguments: READ-TIMEOUT, WRITE-TIMEOUT, and TIMEOUT. [1] http://ccl.clozure.com/manual/chapter7.2.html [2] http://www.lispworks.com/documentation/lw60/LW/html/lw-553.htm#pgfId-888355
There :TIMEOUT is a read timeout for any single blocking read from the socket. Connections and writes are unaffected -- though SBCL changes may apply the timeouts to writes as well.
Waiting for a connection is not affected -- but adding support for that is pretty easy.
Then I suggest adding these missing part, at lest adjust the meaning of "TIMEOUT" argument of SOCKET-MAKE-STREAM into "connection timeout", and add READ-TIME / WRITE-TIMEOUT support.
Sidenote: I haven't really looked at usocket's SBCL backend beyond SOCKET-CONNECT prior to this, but a quick peek shows that it's pretty cavalier about using SBCL internals. This is not good. Internals _will_ change, and then usocket will break. Specifically, SB-UNIX package is an internal implementation package -- using it is bad. SB-POSIX is the supported POSIX api. Some SB-FOO::BAR stuff in there as well. :/
Well, I think maybe the two projects (USOCKET and SBCL) didn't collaborate quite well in past years. USOCKET authors (at least me) want to support as most versions of SBCL as possible. This means, if we ask SBCL to add all necessary support to make things on usocket side become trivial, then we will loose support for older SBCL versions. Sometimes we have to do some dirty work (all around SBCL backend code) to detect if a "feature" exist in a SBCL version... I admit this is not good, but consider SBCL's current networking API lack of many features, and SBCL's release cycle is too fast to make all in-use SBCL versions quickly become old, I don't think a full rewrite of usocket's SBCL backend is possible at current stage. But if SBCL official can start to improve its networking part from now on, after maybe one year, or after at least 10 new SBCL releases, maybe we can force switch to a new backend with "good code" without using any internals. Glad to know your thoughts on this. Regards, --binghe
Cheers,
-- Nikodemus
2011/3/28 Chun Tian (binghe) <binghe.lisp@gmail.com>:
Well, I think maybe the two projects (USOCKET and SBCL) didn't collaborate quite well in past years. USOCKET authors (at least me) want to support as most versions of SBCL as possible. This means, if we ask SBCL to add all necessary support to make things on usocket side become trivial, then we will loose support for older SBCL versions. Sometimes we have to do some dirty work (all around SBCL backend code) to detect if a "feature" exist in a SBCL version...
While you can try to support as many versions as possible, by using eg. SB-UNIX:FAST-SELECT you're virtually guaranteeing that at some unspecified date an SBCL update will break usocket. (Sooner or later SB-UNIX:SELECT will probably go away and be SB-UNIX:FAST-SELECT will be renamed SB-UNIX:SELECT. For that matter, because SB-UNIX is one of the more common causes of "things break due to an SBCL update because someone used undocumented internals", we might rename the whole package something less tempting. Of course it's unrealistic for usocket to stop using stuff it has grown up using tomorrow... but I think trying to support old SBCL versions indefinitely in future usocket releases is a failing strategy. The manpower to take write and maintain the different versions just isn't there. I would suggested that usocket should happily use as new SBCL versions as it wants, and tells users of older SBCL versions to either update SBCL not update usocket... (Or donate their time or money to maintain the temporal portability if they really must be able to use new usocket releases with old SBCL versions.) As for what SBCL can do to support usocket better: Generally speaking, if you tell us that you need X, and X is either impossible or irritating to write using supported interfaces, we can either provide you with a supported X or support necessary lower-level interfaces. X can be anything, but generally it is easier to expose low-level functionality that gives you more rope than trying to get a high-level interface right the first time. If X is "a better networking substrate" ... it's going to take a while. :) Easier things can happen very quickly, more specific ones are typically easier. Cheers, -- Nikodemus
Hi, Nikodemus Today I think out another way to solve the SBCL connection timeout issue, I wrap a SB-EXT:WITH-TIMEOUT on SB-BSD-SOCKET:SOCKET-CONNNECT [1], and the result work seems working well: * (time (ignore-errors (usocket:socket-connect "8.8.8.8" 80 :timeout 3))) Evaluation took: 3.010 seconds of real time 0.010234 seconds of total run time (0.009223 user, 0.001011 system) 0.33% CPU 32 lambdas converted 6,608,582,602 processor cycles 916,672 bytes consed NIL #<USOCKET:TIMEOUT-ERROR {1002F34851}> Do you think this is an acceptable solution? --binghe [1] http://trac.common-lisp.net/usocket/changeset/589 在 2011-3-28，04:26， Nikodemus Siivola 写道：
2011/3/27 Chun Tian (binghe) <binghe.lisp@gmail.com>:
Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings:
1. When doing TCP work, it means a "connection timeout"; 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP);
For SBCL backend at least this is not correct.
There :TIMEOUT is a read timeout for any single blocking read from the socket. Connections and writes are unaffected -- though SBCL changes may apply the timeouts to writes as well.
Waiting for a connection is not affected -- but adding support for that is pretty easy.
Sidenote: I haven't really looked at usocket's SBCL backend beyond SOCKET-CONNECT prior to this, but a quick peek shows that it's pretty cavalier about using SBCL internals. This is not good. Internals _will_ change, and then usocket will break. Specifically, SB-UNIX package is an internal implementation package -- using it is bad. SB-POSIX is the supported POSIX api. Some SB-FOO::BAR stuff in there as well. :/
Cheers,
-- Nikodemus
2011/3/28 Chun Tian (binghe) <binghe.lisp@gmail.com>:
Today I think out another way to solve the SBCL connection timeout issue, I wrap a SB-EXT:WITH-TIMEOUT on SB-BSD-SOCKET:SOCKET-CONNNECT [1], and the result work seems working well:
That's along the lines I was thinking off, except that SB-EXT:WITH-TIMEOUT is a broken construct. (Soon to be deprecated, in all likelihood.) Consider this: (with-timeout 1.0 (handler-case (with-timeout 4.0 (sleep 2)) (sb-ext:timeout ()))) which is to say that you cannot distinguish an outer timeout from an inner one, which is bad. You need something like this, instead: (defmacro with-timeout-handler ((seconds timeout-form) &body body) "Runs BODY as an implicit PROGN with timeout of SECONDS. If timeout occurs before BODY has finished, BODY is unwound and TIMEOUT-FORM is executed with its values returned instead. Note that BODY is unwound asynchronously when a timeout occurs, so unless all code executed during it -- including anything down the call chain -- is asynch unwind safe, bad things will happen. Use with care." (alexandria:with-gensyms (exec unwind timer timeout block) `(block ,block (tagbody (flet ((,unwind () (go ,timeout)) (,exec () ,@body)) (declare (dynamic-extent #',exec #',unwind)) (let ((,timer (sb-ext:make-timer #',unwind))) (declare (dynamic-extent ,timer)) (sb-sys:without-interrupts (unwind-protect (progn (sb-ext:schedule-timer ,timer ,seconds) (return-from ,block (sb-sys:with-local-interrupts (,exec)))) (sb-ext:unschedule-timer ,timer))))) ,timeout (return-from ,block ,timeout-form))))) with which (with-timeout-handler (1.0 :outer) (with-timeout-handler (4.0 :inner) (sleep 10.0) :ok)) does the right thing. Gods, I hate asynch timeouts. Is there a sane way to tell connect() to time out without needing SIGALRM? Cheers, -- Nikodemus
Gods, I hate asynch timeouts. Is there a sane way to tell connect() to time out without needing SIGALRM?
Nope. Have you considered using a non-blocking connect() call, combining that with a select() call with a timeout? Bye, Erik.
On 28 March 2011 23:42, Erik Huelsmann <ehuels@gmail.com> wrote:
Gods, I hate asynch timeouts. Is there a sane way to tell connect() to time out without needing SIGALRM?
Nope. Have you considered using a non-blocking connect() call, combining that with a select() call with a timeout?
If that's the way it needs to be done, then that's the way it needs to be done... But realistically, I'm not going to have time to work on that in near future unless things suddenly change. Cheers, -- Nikodemus
I know SBCL's WITH-TIMEOUT cannot nest, I learn this from GBBopen's portable-threads.lisp [1], and it also give a nested version SBCL's WITH-TIMEOUT, much shorter than yours: #+sbcl (defmacro with-timeout ((seconds &body timeout-body) &body timed-body) (let ((tag-sym (gensym)) (timer-sym (gensym))) `(block ,tag-sym (let ((,timer-sym (sb-ext:make-timer #'(lambda () (return-from ,tag-sym (progn ,@timeout-body)))))) (sb-ext:schedule-timer ,timer-sym ,seconds) (unwind-protect (progn ,@timed-body) (sb-ext:unschedule-timer ,timer-sym)))))) I didn't use this version simply because I think the WITH-TIMEOUT form in usocket's SOCKET-CONNECT has no chance to be nested. --binghe 在 2011-3-29，04:35， Nikodemus Siivola 写道：
2011/3/28 Chun Tian (binghe) <binghe.lisp@gmail.com>:
Today I think out another way to solve the SBCL connection timeout issue, I wrap a SB-EXT:WITH-TIMEOUT on SB-BSD-SOCKET:SOCKET-CONNNECT [1], and the result work seems working well:
That's along the lines I was thinking off, except that SB-EXT:WITH-TIMEOUT is a broken construct. (Soon to be deprecated, in all likelihood.)
Consider this:
(with-timeout 1.0 (handler-case (with-timeout 4.0 (sleep 2)) (sb-ext:timeout ())))
which is to say that you cannot distinguish an outer timeout from an inner one, which is bad.
You need something like this, instead:
(defmacro with-timeout-handler ((seconds timeout-form) &body body) "Runs BODY as an implicit PROGN with timeout of SECONDS. If timeout occurs before BODY has finished, BODY is unwound and TIMEOUT-FORM is executed with its values returned instead.
Note that BODY is unwound asynchronously when a timeout occurs, so unless all code executed during it -- including anything down the call chain -- is asynch unwind safe, bad things will happen. Use with care." (alexandria:with-gensyms (exec unwind timer timeout block) `(block ,block (tagbody (flet ((,unwind () (go ,timeout)) (,exec () ,@body)) (declare (dynamic-extent #',exec #',unwind)) (let ((,timer (sb-ext:make-timer #',unwind))) (declare (dynamic-extent ,timer)) (sb-sys:without-interrupts (unwind-protect (progn (sb-ext:schedule-timer ,timer ,seconds) (return-from ,block (sb-sys:with-local-interrupts (,exec)))) (sb-ext:unschedule-timer ,timer))))) ,timeout (return-from ,block ,timeout-form)))))
with which
(with-timeout-handler (1.0 :outer) (with-timeout-handler (4.0 :inner) (sleep 10.0) :ok))
does the right thing.
Gods, I hate asynch timeouts. Is there a sane way to tell connect() to time out without needing SIGALRM?
Cheers,
-- Nikodemus
2011/3/29 Chun Tian (binghe) <binghe.lisp@gmail.com>:
I know SBCL's WITH-TIMEOUT cannot nest, I learn this from GBBopen's portable-threads.lisp [1], and it also give a nested version SBCL's WITH-TIMEOUT, much shorter than yours:
Amusingly, neither SBCL's own, nor GBBopen's WITH-TIMEOUT is asynch unwind safe. The one I posted is -- that's what the WITHOUT-INTERRUPTS and WITH-LOCAL-INTERRUPTS were for. :) But yeah, it's miles saner than the SB-EXT:WITH-TIMEOUT. The GBBopen's WITH-TIMEOUT also suffers from the minor defect that the dynamic context (special variables, handlers, restarts, etc) the timeout-body runs in is unpredictable -- for most cases this does not matter, but it can lead to hard to reproduce bugs.
I didn't use this version simply because I think the WITH-TIMEOUT form in usocket's SOCKET-CONNECT has no chance to be nested.
It has. A function in a library cannot know if it will be nested in some dynamic context or not. The corrollary is that library code should never use SB-EXT;WITH-TIMEOUT. (defun foo () ...stuff that uses socket-connect with a local timeout > 1.0 somewhere in its guts...) (with-timeout 1.0 (handler-case (foo) (error () :bad-foo))) If the fetch takes longer than intended, the outer timeout can get converted into an USOCKET-TIMEOUT-ERROR (whatever the exact name), which in turn gets swallowed by the HANDLER-CASE. Cheers, -- Nikodemus
participants (3)
-
Chun Tian (binghe) -
Erik Huelsmann -
Nikodemus Siivola