March 2011 - usocket-devel - mailman3.common-lisp.net

Re: [usocket-devel] [usocket-cvs] r604 - usocket/branches/0.5.x/backend
by Erik Huelsmann 30 Mar '11

30 Mar '11

Hi Chun, Nice work (and great activity!). Just a remark from personal experience: Don't forget to backport to trunk when you're working directly on the branch. Bye, Erik. On Tue, Mar 29, 2011 at 7:04 PM, Chun Tian <ctian(a)common-lisp.net> wrote: > Author: ctian > Date: Tue Mar 29 13:04:30 2011 > New Revision: 604 > > Log: > [CLISP] Fixed SOCKET-CONNECT / UDP for RAWSOCK; Basic FFI framework. > > Modified: > usocket/branches/0.5.x/backend/clisp.lisp > > Modified: usocket/branches/0.5.x/backend/clisp.lisp > ============================================================================== > --- usocket/branches/0.5.x/backend/clisp.lisp (original) > +++ usocket/branches/0.5.x/backend/clisp.lisp Tue Mar 29 13:04:30 2011 > @@ -5,9 +5,15 @@ > > (in-package :usocket) > > +(eval-when (:compile-toplevel :load-toplevel :execute) > + #-ffi > + (warn "This image doesn't contain FFI package, GET-HOST-NAME won't work.") > + #-(or ffi rawsock) > + (warn "This image doesn't contain either FFI or RAWSOCK package, no UDP support.")) > + > ;; utility routine for looking up the current host name > #+ffi > -(FFI:DEF-CALL-OUT get-host-name-internal > +(ffi:def-call-out get-host-name-internal > (:name "gethostname") > (:arguments (name (FFI:C-PTR (FFI:C-ARRAY-MAX ffi:character 256)) > :OUT :ALLOCA) > @@ -61,26 +67,36 @@ > timeout deadline (nodelay t nodelay-specified) > local-host local-port) > (declare (ignore nodelay)) > - (when timeout (unsupported 'timeout 'socket-connect)) > (when deadline (unsupported 'deadline 'socket-connect)) > (when nodelay-specified (unsupported 'nodelay 'socket-connect)) > (when local-host (unsupported 'local-host 'socket-connect)) > (when local-port (unsupported 'local-port 'socket-connect)) > - > - (let ((socket) > - (hostname (host-to-hostname host))) > - (with-mapped-conditions (socket) > - (setf socket > - (if timeout > - (socket:socket-connect port hostname > - :element-type element-type > - :buffered t > - :timeout timeout) > - (socket:socket-connect port hostname > - :element-type element-type > - :buffered t)))) > - (make-stream-socket :socket socket > - :stream socket))) ;; the socket is a stream too > + (case protocol > + (:stream > + (let ((socket) > + (hostname (host-to-hostname host))) > + (with-mapped-conditions (socket) > + (setf socket > + (if timeout > + (socket:socket-connect port hostname > + :element-type element-type > + :buffered t > + :timeout timeout) > + (socket:socket-connect port hostname > + :element-type element-type > + :buffered t)))) > + (make-stream-socket :socket socket > + :stream socket))) ;; the socket is a stream too > + (:datagram > + #+rawsock > + (socket-create-datagram (or local-port *auto-port*) > + :local-host (or local-host *wildcard-host*) > + :remote-host host > + :remote-port port) > + #+(and ffi (not rawsock)) > + () > + #-(or rawsock ffi) > + (unsupported '(protocol :datagram) 'socket-connect)))) > > (defun socket-listen (host port > &key reuseaddress > @@ -146,7 +162,6 @@ > (defmethod get-peer-port ((usocket stream-usocket)) > (nth-value 1 (get-peer-name usocket))) > > - > (defun %setup-wait-list (wait-list) > (declare (ignore wait-list))) > > @@ -176,14 +191,12 @@ > (setf (state x) :READ))) > wait-list)))) > > - > -;; > -;; UDP/Datagram sockets! > -;; > +;;; > +;;; UDP/Datagram sockets (RAWSOCK version) > +;;; > > #+rawsock > (progn > - > (defun make-sockaddr_in () > (make-array 16 :element-type '(unsigned-byte 8) :initial-element 0)) > > @@ -209,7 +222,7 @@ > (connect sock rsock_addr)) > (make-datagram-socket sock :connected-p (if rsock_addr t nil)))) > > - (defun socket-receive (socket buffer &key (size (length buffer))) > + (defun socket-receive (socket buffer length &key) > "Returns the buffer, the number of octets copied into the buffer (received) > and the address of the sender as values." > (let* ((sock (socket socket)) > @@ -218,44 +231,74 @@ > (rv (if sockaddr > (rawsock:recvfrom sock buffer sockaddr > :start 0 > - :end size) > + :end length) > (rawsock:recv sock buffer > :start 0 > - :end size)))) > + :end length)))) > (values buffer > rv > - (list (ip-from-octet-buffer (sockaddr-data sockaddr) 4) > - (port-from-octet-buffer (sockaddr-data sockaddr) 2))))) > + (ip-from-octet-buffer (sockaddr-data sockaddr) 4) > + (port-from-octet-buffer (sockaddr-data sockaddr) 2)))) > > - (defun socket-send (socket buffer &key address (size (length buffer))) > + (defun socket-send (socket buffer length &key host port) > "Returns the number of octets sent." > (let* ((sock (socket socket)) > - (sockaddr (when address > + (sockaddr (when (and host port) > (rawsock:make-sockaddr :INET > (fill-sockaddr_in > (make-sockaddr_in) > - (host-byte-order > - (second address)) > - (first address))))) > - (rv (if address > + (host-byte-order host) > + port)))) > + (rv (if (and host port) > (rawsock:sendto sock buffer sockaddr > :start 0 > - :end size) > + :end length) > (rawsock:send sock buffer > :start 0 > - :end size)))) > + :end length)))) > rv)) > > (defmethod socket-close ((usocket datagram-usocket)) > (when (wait-list usocket) > (remove-waiter (wait-list usocket) usocket)) > (rawsock:sock-close (socket usocket))) > - > - ) > +) ; progn > + > +;;; > +;;; UDP/Datagram sockets (FFI version) > +;;; > > -#-rawsock > +#+(and ffi (not rawsock)) > (progn > - (warn "This image doesn't contain the RAWSOCK package. > -To enable UDP socket support, please be sure to use the -Kfull parameter > -at startup, or to enable RAWSOCK support during compilation.") > - ) > + (ffi:def-c-struct sockaddr > + ) > + > + (ffi:def-c-struct sockaddr_in > + ) > + > + (ffi:def-call-out %sendto (:name "sendto") > + (:arguments (socket ffi:int) > + (buffer (ffi:c-ptr ffi:uint8)) > + (length ffi:int) > + (flags ffi:int) > + (address (ffi:c-ptr sockaddr)) > + (address-len ffi:int)) > + #+win32 (:library "WS2_32") > + #-win32 (:library :default) > + (:language #-win32 :stdc > + #+win32 :stdc-stdcall) > + (:return-type ffi:int)) > + > + (ffi:def-call-out %recvfrom (:name "recvfrom") > + (:arguments (socket ffi:int) > + (buffer (ffi:c-ptr ffi:uint8) :out) > + (length ffi:int) > + (flags ffi:int) > + (address (ffi:c-ptr sockaddr) :out) > + (address-len (ffi:c-ptr ffi:int) :out)) > + #+win32 (:library "WS2_32") > + #-win32 (:library :default) > + (:language #-win32 :stdc > + #+win32 :stdc-stdcall) > + (:return-type ffi:int)) > +) ; progn > > _______________________________________________ > usocket-cvs mailing list > usocket-cvs(a)common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/usocket-cvs >

2 1

Re: [usocket-devel] [drakma-devel] [patch] Resubmit drakma timeout for sbcl.
by Chun Tian (binghe) 29 Mar '11

29 Mar '11

Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings: 1. When doing TCP work, it means a "connection timeout"; 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP); The TIMEOUT keyword arguments was added from 0.4.0 (released on Oct 2008, more than two years ago), so I think Drakma should try to use it. However, it may not work on all supported platforms -- I remembered. But this should be considered as bug, I'll try to fix it after a small rewrite of our unit test framework. On the other hand, for "read timeout" and "write timeout", they're changeable properties (or options) for any socket objects. A new API called SOCKET-OPTION (learnt from CLISP) should be added in next USOCKET major version. Dynamically setting options like "broadcast" and "reuse address" should also be added under SOCKET-OPTION. I see hunchentoot has a function called SET-TIMEOUTS, which can be used to set "read timeout" and "write timeout" for any exist usocket, I want to merge it into usocket under the new SOCKET-OPTION API. --binghe 在 2011-3-28，01:50， Edi Weitz 写道： > 2011/3/27 Chun Tian (binghe) <binghe.lisp(a)gmail.com>: > >> Could you tell me what exact timeout feature they want? Setting Read/Write timeout when doing SOCKET-CONNECT? or Apply new timeout values to exist usocket object? > > The LW version of Drakma supports connection timeouts, read timeouts, > and write timeouts. Existing usocket objects wouldn't be modified. > > Cheers, > Edi.

3 9

Re: [usocket-devel] [drakma-devel] [patch] Resubmit drakma timeout for sbcl.
by Daniel Weinreb 28 Mar '11

28 Mar '11

Edi Weitz wrote: I think this is something that belongs into usocket and not into Drakma. I'm not monitoring usocket, but if it had the ability to optionally specify timeouts and deadlines for those implementations supporting these features (and making them no-ops on others), I'd happily integrate that into Drakma. I don't want to litter Drakma with conditional expression for various Lisps, though. and in later mail: > > I didn't know that usocket supports timeouts now. But in the latest Drakma, in http-request, there is a :deadline argument, which is passed through to usocket:socket-connect. Unfortunately, it is specific to CCL (OpenMCL). Drakma also knows, for CCL, to set the ccl:stream-deadline of the stream, too, so that the deadline applies not only to the connect work itself, but to the entire connection. This is quite useful for anyone writing a program that needs to have end-to-end timeouts. Servers usually need this more than clients, in my experience. We use this in our server for that reason: every request is given a deadline, and we have to make sure that anything that might block for input or output will honor the deadline. For Lispworks, I see that there is a different set of keyword arguments, that have "timeout" in their names. I completely sympathize with your not wanting to litter Drakma with implemenation-specific code. It would be good to get as much of this out of Drakma as possible. For Lispworks this may be difficult; I notice that Hunchentoot also needs to have a lot of Lispworks-specfic code. However, it looks to me as if usocket:socket-connect does support a :timeout argument for SBCL, as well as for CCL (OpenMCL). Also abcl and mcl, but not Allegro, cmucl, scl, nor clisp. Presumably this is because the underling implementation doens't support it; like many things, it's not part of the CL standard. Erik says, about usocket: Well, there has been some development in usocket in the direction of supporting timeouts. On platforms where timeouts are not supported, an UNSUPPORTED-FEATURE condition is signalled. If that's ignored, everything should just continue without the timeout settings. Indeed, if you look at socket-connect for SBCL,if the deadline argument is not null, it signals a condition to say that the feature is unsupported. socket-connect does, however, support the :timeout argument. I don't know SBCL can't accept :deadline, subtract from it the current time, and then use that as a :timeout, but maybe there's a good reason for that. The documentation of socket-connect, which is in the file usocket.lisp, doesn't document all of the arguments to socket-connect, so I might be missing something about the semantics. -- Dan

2 1

Re: [usocket-devel] [drakma-devel] [patch] Resubmit drakma timeout for sbcl.
by Erik Huelsmann 27 Mar '11

27 Mar '11

Hi Edi, 2011/3/26 Edi Weitz <edi(a)weitz.de>: > On Sat, Mar 26, 2011 at 6:17 PM, Александр Литвинов <lan(a)academsoft.ru> wrote: >> There was the same decision at Dec 2007. Now, three years later, you >> have not changed it. > > Well, I don't think it's too bad not to change your mind every year... :) > > Actually, I had hoped that there had been some development in usocket since. Well, there has been some development in usocket in the direction of supporting timeouts. On platforms where timeouts are not supported, an UNSUPPORTED-FEATURE condition is signalled. If that's ignored, everything should just continue without the timeout settings. If the above is not good enough, could you indicate what more needs to happen to make it so? Bye, Erik.

2 2

[usocket-devel] SBCL/Win32: finalizer problem, etc.
by Anton Kovalenko 22 Mar '11

22 Mar '11

Greetings to all USOCKET developers and contributors. First, thank you for the wonderful package. But of course, I'm posting here to report some problems (on a particular platform: #+(and sbcl win32)) 1. Event handles are leaking in current SBCL backend implementation, because of SBCL-unfriendly usage of finalizers. SBCL never calls a finalizer that closes over a finalized object: a reference from that closure prevents its collection forever. That's the case with USOCKET in %SETUP-WAIT-LIST. I use the following redefinition of %SETUP-WAIT-LIST: (defun %setup-wait-list (wait-list) (setf (wait-list-%wait wait-list) (sb-alien:make-alien ws-event)) (setf (os-wait-list-%wait wait-list) (wsa-event-create)) (sb-ext:finalize wait-list (let ((event-handle (os-wait-list-%wait wait-list)) (alien (wait-list-%wait wait-list))) #'(lambda () (wsa-event-close event-handle) (unless (null alien) (sb-alien:free-alien alien)))))) Of course it may be rewritten with more clarity, but you can see the core idea: I'm closing over those components of WAIT-LIST that I need for finalization, not the wait-list itself. With the original %SETUP-WAIT-LIST, hunchentoot stops working after ~100k accepted connections; it doesn't happen with redefined %SETUP-WAIT-LIST. 2. SB-BSD-SOCKETS:SOCKET-ACCEPT method returns NIL for EAGAIN/EINTR, instead of raising a condition. It's always possible for SOCKET-ACCEPT on non-blocking socket to fail, even after the socket was detected to be ready: connection might be reset, for example. I had to redefine SOCKET-ACCEPT method of STREAM-SERVER-USOCKET to handle this situation. Here is the redefinition: (defmethod socket-accept ((socket stream-server-usocket) &key element-type) (with-mapped-conditions (socket) (let ((sock (sb-bsd-sockets:socket-accept (socket socket)))) (if sock (make-stream-socket :socket sock :stream (sb-bsd-sockets:socket-make-stream sock :input t :output t :buffering :full :element-type (or element-type (element-type socket)))) ;; next time wait for event again if we had EAGAIN/EINTR ;; or else we'd enter a tight loop of failed accepts (setf (%ready-p socket) nil))))) P.S. For some months I'm working on threading support for SBCL/Windows (X86 and AMD64) [ http://github.com/akovalenko/sbcl-win32-threads/wiki ], continuing the work of Dmitry Kalyanov, who implemented threading support initially. Traditionally, I use HUNCHENTOOT as a kind of smoke test for my builds, and it's a wonderful choice. Two issues described above, however, are universal -- it's not something that we meet (only) on my experimental threads builds. #1 applies to _any_ SBCL/Windows, #2 might not apply to some earlier versions that are still in use; both deserves fixing. Here is another issue with USOCKET that _can't_ affect upstream SBCL until Windows/AMD64 support is accepted: 3. SOCKET is defined as intptr_t in Windows headers; however, WS-SOCKET is defined as unsigned-int, i.e. 32-bit even on 64-bit platform. It seems to be a good thing to redefine WS-SOCKET as SB-ALIEN:SIGNED, which is always machine word-sized (exactly as intptr_t; N.B. as of Windows/x64, long and signed-long are 32-bit, and thus not enough -- potentially). Fortunately, this last issue doesn't cause any actual damage now: HUNCHENTOOT works on my x64 build (when redefinitions mentioned above are loaded). However, it happens just because SBCL does more than MS ABI specification requires, i.e. doesn't leave garbage in the upper 32 bits of the argument. I'm afraid that some future optimization (and/or reimplementation) of SBCL FFI callout may eventually break this behavior -- so it still makes sense to define WS-SOCKET according to Windows headers. -- Regards, Anton Kovalenko +7(916)345-34-02 | Elektrostal' MO, Russia

2 4

[usocket-devel] socket-connect and hostnames resolved to multiple addresses
by Alexey Martynov 17 Mar '11

17 Mar '11

I have following problem: remote hostname resolves to multiple addresses and I want to try all one by one until connection is succeeded. But: 1. current policy when multiple addresses given for hostname - try random one. 2. no API for resolving addresses. I can try to propose patch for this problem but in which direction I should go: provide resolving API or involve sequential trying of addresses? -- Alexey Martynov

4 4

[usocket-devel] [ANNOUNCE] usocket 0.5.0
by Erik Huelsmann 12 Mar '11

12 Mar '11

I'm happy to be able to release usocket 0.5.0. The major point in this release is the availability of UDP protocol support, next to lots of fixes since the 0.4.1 release. The API documentation page on the website still needs to be updated for the additional external symbols. (Any takers?) If you want to download this release, please check out http://common-lisp.net/project/usocket/releases/ Special thanks go to: Chun Tian: * UDP socket support Kamil Shakirov: * Enable concurrent recv/send on mutiple UDP sockets in LispWorks 6.0 Nikodemus Siivola: * Better SOCKET-CONNECT for SBCL Terje Norderhaug and James Anderson: * Macintosh Common Lisp support As always, any feedback - bugs or hugs - is greatly appreciated. If you use usocket, please let us know what for! Bye, Erik.

1 0

[usocket-devel] better SOCKET-CONNECT for SBCL
by Nikodemus Siivola 10 Mar '11

10 Mar '11

Current implementation has the following defects: * The socket stream ends up with a bogus name as it is created before the socket is connected, making things harder to debug than they need to be. * In case of error, SB-BSD-SOCKETS:SOCKET-CLOSE was not being called with :ABORT T. * A unwind from SOCKET-CONNECT that didn't go through the handler leaked an FD. * An innocent non-error SIGNAL for any reason during SOCKET-CONNECT was taken to be a fatal error due to the handler on type T. * On datagram sockets CONNECTED-P wasn't set. Attached patch addresses these issues. Not deeply tested, though. Cheers, -- Nikodemus

2 1