Hi Douglas,
I recently added some functionality in usocket and even though I think I could have copy-pasted from the CMU backend, I want to verify with you if this is true.
The functions I added: - get-host-name (as you can see in the commit mail below based on copy-paste from CMU) - wait-for-input-internal which I didn't copy-paste yet.
I would be especially gratefull if you could copy-paste and verify the latter for me and optionally even supply a patch.
For testing of the multiplexing code, I use this 'script':
(load "asdf.lisp") (asdf:oos 'asdf:load-op :usocket)
(defvar sockets nil) (defparameter crlf (format nil "~C~C" #\Return #\Linefeed)) (defparameter host "mirror.w3media.nl") (defparameter offset-path "/apache")
(dotimes (i 10) (let ((s (usocket:socket-connect host 80))) (format (usocket:socket-stream s) "GET http://~A~A/httpd/httpd-2.0.~A.tar.gz HTTP/1.0~%~%" host offset-path (+ 50 i)) (force-output (usocket:socket-stream s)) (push s sockets)))
(let ((buffer (make-array 8192 :element-type 'character))) (loop (let ((ready (usocket:wait-for-input sockets :timeout 15))) (unless ready (sleep 1) (format t "No ready sockets!~%")) (dolist (r ready) (format t "Reading from ~A~%" r) (read-sequence buffer (usocket:socket-stream r))))))
HTH and thanks for your reaction in advance!
bye,
Erik.
---------- Forwarded message ---------- From: ehuelsmann@common-lisp.net ehuelsmann@common-lisp.net Date: May 18, 2007 12:03 AM Subject: [usocket-cvs] r245 - usocket/trunk/backend To: usocket-cvs@common-lisp.net
Author: ehuelsmann Date: Thu May 17 18:03:55 2007 New Revision: 245
Modified: usocket/trunk/backend/scl.lisp Log: Add cl-smtp 'requirement': get-host-name (SCL backend); needs verification.
Modified: usocket/trunk/backend/scl.lisp ============================================================================== --- usocket/trunk/backend/scl.lisp (original) +++ usocket/trunk/backend/scl.lisp Thu May 17 18:03:55 2007 @@ -129,3 +129,6 @@ (t (error 'ns-unknown-error :host-or-ip name :real-error errno)))))))) + +(defun get-host-name () + (unix:unix-gethostname)) _______________________________________________ usocket-cvs mailing list usocket-cvs@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/usocket-cvs
Hello Erik,
Thank you for keeping the Scieneer CL in mind. The get-host-name implementation does work in the SCL. An implementation of wait-for-input-internal is attached.
Changing the wait-for-input-internal function to set a flag in the socket objects to indicate they are ready would avoid the need to cons up a list of ready sockets which can become very inefficient when managing a large number of connections.
Please consider also adding support for write waiting which is necessary when writing event loops for servers. This could be done by adding another flag to sockets to indication their waiting direction: :input, :output, :io, or nil.
Regards Douglas Crosher
Erik Huelsmann wrote:
Hi Douglas,
I recently added some functionality in usocket and even though I think I could have copy-pasted from the CMU backend, I want to verify with you if this is true.
The functions I added:
- get-host-name (as you can see in the commit mail below based on
copy-paste from CMU)
- wait-for-input-internal which I didn't copy-paste yet.
I would be especially gratefull if you could copy-paste and verify the latter for me and optionally even supply a patch.
For testing of the multiplexing code, I use this 'script':
(load "asdf.lisp") (asdf:oos 'asdf:load-op :usocket)
(defvar sockets nil) (defparameter crlf (format nil "~C~C" #\Return #\Linefeed)) (defparameter host "mirror.w3media.nl") (defparameter offset-path "/apache")
(dotimes (i 10) (let ((s (usocket:socket-connect host 80))) (format (usocket:socket-stream s) "GET http://~A~A/httpd/httpd-2.0.~A.tar.gz HTTP/1.0~%~%" host offset-path (+ 50 i)) (force-output (usocket:socket-stream s)) (push s sockets)))
(let ((buffer (make-array 8192 :element-type 'character))) (loop (let ((ready (usocket:wait-for-input sockets :timeout 15))) (unless ready (sleep 1) (format t "No ready sockets!~%")) (dolist (r ready) (format t "Reading from ~A~%" r) (read-sequence buffer (usocket:socket-stream r))))))
HTH and thanks for your reaction in advance!
bye,
Erik.
---------- Forwarded message ---------- From: ehuelsmann@common-lisp.net ehuelsmann@common-lisp.net Date: May 18, 2007 12:03 AM Subject: [usocket-cvs] r245 - usocket/trunk/backend To: usocket-cvs@common-lisp.net
Author: ehuelsmann Date: Thu May 17 18:03:55 2007 New Revision: 245
Modified: usocket/trunk/backend/scl.lisp Log: Add cl-smtp 'requirement': get-host-name (SCL backend); needs verification.
Modified: usocket/trunk/backend/scl.lisp
--- usocket/trunk/backend/scl.lisp (original) +++ usocket/trunk/backend/scl.lisp Thu May 17 18:03:55 2007 @@ -129,3 +129,6 @@ (t (error 'ns-unknown-error :host-or-ip name :real-error errno))))))))
+(defun get-host-name ()
- (unix:unix-gethostname))
usocket-cvs mailing list usocket-cvs@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/usocket-cvs
Index: backend/scl.lisp =================================================================== --- backend/scl.lisp (revision 268) +++ backend/scl.lisp (working copy) @@ -132,3 +132,37 @@
(defun get-host-name () (unix:unix-gethostname)) + +(defun wait-for-input-internal (sockets &key timeout) + (let* ((pollfd-size (alien:alien-size (alien:struct unix::pollfd) :bytes)) + (nfds (length sockets)) + (bytes (* nfds pollfd-size))) + (alien:with-bytes (fds-sap bytes) + (do ((sockets sockets (rest sockets)) + (base 0 (+ base 8))) + ((endp sockets)) + (let ((fd (socket (first sockets)))) + (setf (sys:sap-ref-32 fds-sap base) fd) + (setf (sys:sap-ref-16 fds-sap (+ base 4)) unix::pollin))) + (multiple-value-bind (result errno) + (let ((thread:*thread-whostate* "Poll wait") + (timeout (if timeout + (truncate (* timeout 1000)) + -1))) + (declare (inline unix:unix-poll)) + (unix:unix-poll (alien:sap-alien fds-sap + (* (alien:struct unix::pollfd))) + nfds timeout)) + (cond ((not result) + (error "~@<Polling error: ~A~:@>" + (unix:get-unix-error-msg errno))) + (t + (do ((sockets sockets (rest sockets)) + (base 0 (+ base 8)) + (ready nil)) + ((endp sockets) + (nreverse ready)) + (let ((flags (sys:sap-ref-16 fds-sap (+ base 6)))) + (unless (zerop (logand flags unix::pollin)) + (push (first sockets) ready)))))))))) +
On 6/8/07, Douglas Crosher dtc@scieneer.com wrote:
Hello Erik,
Thank you for keeping the Scieneer CL in mind. The get-host-name implementation does work in the SCL. An implementation of wait-for-input-internal is attached.
And thank you for your continued contributions! I've committed the patch last night and updated the website accordingly.
Changing the wait-for-input-internal function to set a flag in the socket objects to indicate they are ready would avoid the need to cons up a list of ready sockets which can become very inefficient when managing a large number of connections.
That's true. On the other hand, the efficiency of needing to walk all provided socket objects to find out which ones changed seems to be the main objection against the select() system call (which is why poll(), epoll() and others have been invented, right?).
Please consider also adding support for write waiting which is necessary when writing event loops for servers. This could be done by adding another flag to sockets to indication their waiting direction: :input, :output, :io, or nil.
I'm not only considering that, I really want to provide it, but, there's a bigger problem (than providing input waiting): while it was possible to provide input waiting on most implementations, output waiting is less broadly supported and will require more FFI code. It's not that I don't want to do that, it just takes more time.
bye,
Erik.
Hello Erik,
Changing the wait-for-input-internal function to set a flag in the socket objects to indicate they are ready would avoid the need to cons up a list of ready sockets which can become very inefficient when managing a large number of connections.
That's true. On the other hand, the efficiency of needing to walk all provided socket objects to find out which ones changed seems to be the main objection against the select() system call (which is why poll(), epoll() and others have been invented, right?).
The lack of performance probably shows that the 'wait-for-input-internal interface is too high-level - it implicitly defines a socket set as a list and this list needs to be consed when calling and on return and scanning the list slow. In contrast a CL implementation with an interface to poll() and select() can implement non-consing socket sets that have higher performance. The poll() function has an advantage when waiting on only a small subset of the open sockets and select() may have an advantage when waiting on a large set of sockets because the bit sets can be scanned in word sized chunks.
Perhaps a portable interface to the select() and poll() functions would be better.
Regards Douglas Crosher
On Sun, Jun 17, 2007 at 9:48 AM, Douglas Crosher dtc@scieneer.com wrote:
Hello Erik,
Changing the wait-for-input-internal function to set a flag in the socket objects to indicate they are ready would avoid the need to cons up a list of ready sockets which can become very inefficient when managing a large number of connections.
That's true. On the other hand, the efficiency of needing to walk all provided socket objects to find out which ones changed seems to be the main objection against the select() system call (which is why poll(), epoll() and others have been invented, right?).
The lack of performance probably shows that the 'wait-for-input-internal interface is too high-level - it implicitly defines a socket set as a list and this list needs to be consed when calling and on return and scanning the list slow. In contrast a CL implementation with an interface to poll() and select() can implement non-consing socket sets that have higher performance. The poll() function has an advantage when waiting on only a small subset of the open sockets and select() may have an advantage when waiting on a large set of sockets because the bit sets can be scanned in word sized chunks.
Well, indeed did the interface meet some problems. That's why I've now introduced the concept of a wait-list, something like a poll-set or Java channel-selector. The wait-list (which is a structure btw) gets set up once and reused many times. Also, the sockets now have a STATE slot where the fact of read-readyness is stored.
I think I succesfully updated all backends, except for the Scieneer backend. Could you have a look at how to efficiently use this new wait-list structure? The %WAIT slot is free for use by the backend, the WAITERS slot contains a list of all waiting sockets.
Perhaps a portable interface to the select() and poll() functions would be better.
Unfortunately, that seems impossible with the bounds I've imposed on the project (for now): I'd hate to have to depend on CFFI because that's quite a strong requirement (when looking at Windows). [Not all LispWorks developers have a C compiler installed.]
Anyway, if you could have a look, I would be very greatful. The code is at svn://common-lisp.net/project/usocket/svn/usocket/branches/new-wfi.
Thanks in advance!
Bye,
Erik.
Hi there,
I think one of the design idea of usocket is just not use CFFI, and any non-lisp code. IOlib is another approach of Lisp networking, it has some C wrapper code to be used with CFFI.
Use select() or pool() will cause non_OS-portable code and other difficulties. I suggest not use them directly... usocket should keep it simple, and shouldn't depends on CFFI, or any other ASDF package.
--binghe
在 2008-7-23，下午3:48， Attila Lendvai 写道：
Perhaps a portable interface to the select() and poll() functions would be better.
fyi, iolib exactly that: using cffi, it exposes the socket api.
-- attila _______________________________________________ usocket-devel mailing list usocket-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/usocket-devel
I think one of the design idea of usocket is just not use CFFI, and any non-lisp code. IOlib is another approach of Lisp networking, it has some C wrapper code to be used with CFFI.
Yes, basically I agree.
Use select() or pool() will cause non_OS-portable code and other difficulties. I suggest not use them directly... usocket should keep it simple, and shouldn't depends on CFFI, or any other ASDF package.
And here, I agree too. However, some people have suggested that usocket could incorporate an alternative backend which is built 'closer to the OS', for example using CFFI.
Even though I will not be building such a backend myself, I can definitely see why anyone would want to create it. And as such, when a maintainer pops up for the task, I'll happily integrate the contribution (and sustained maintenance!) into the project.
I hope that we may see a contribution like this some time in the future (ie a contribution specifically targetted at performance).
Bye,
Erik.
Hi, Erik
Very good, and it's my pleasure to have same thoughts with you:)
By the way, I should publish some news:
For the local binding patch, I've get contacted with the CMUCL maintainer (Raymond Toy), and we were still talking about the local- bind-patch on CMUCL mailing list. I think it's possible to merge my CMUCL patch.
For the UDP patch [1,2] (which the first reason I go into your life), Now I have a portable UDP server code. It's quite easy to wrote using UDP local-binding, wait-for-input, and socket-receive/socket-send. I don't know when you'll have time to look it and merge it, but I can wait:)
--binghe
[1] https://cl-net-snmp.svn.sourceforge.net/svnroot/cl-net-snmp/usocket-udp/trun... [2] http://www.cliki.net/usocket-udp
在 2008-7-23，下午9:06， Erik Huelsmann 写道：
I think one of the design idea of usocket is just not use CFFI, and any non-lisp code. IOlib is another approach of Lisp networking, it has some C wrapper code to be used with CFFI.
Yes, basically I agree.
Use select() or pool() will cause non_OS-portable code and other difficulties. I suggest not use them directly... usocket should keep it simple, and shouldn't depends on CFFI, or any other ASDF package.
And here, I agree too. However, some people have suggested that usocket could incorporate an alternative backend which is built 'closer to the OS', for example using CFFI.
Even though I will not be building such a backend myself, I can definitely see why anyone would want to create it. And as such, when a maintainer pops up for the task, I'll happily integrate the contribution (and sustained maintenance!) into the project.
I hope that we may see a contribution like this some time in the future (ie a contribution specifically targetted at performance).
Bye,
Erik.
-
Attila Lendvai -
Chun Tian (binghe) -
Douglas Crosher -
Erik Huelsmann