February 2009 - iolib-devel - mailman3.common-lisp.net

[iolib-devel] stuck sockets
by Attila Lendvai 28 Feb '09

28 Feb '09

Stelian, i need your help. our server went non-responsive and my investigation results in this: all worker threads were waiting here: 15: (IOLIB.STREAMS::%FILL-IBUF IOLIB.SOCKETS::SOCKET-READ-FN 15 #S(IOLIB.STREAMS::IOBUF :DATA #.(SB-SYS:INT-SAP #X00658620) :SIZE 4096 :START 0 :END 0)) 16: ((SB-GRAY:STREAM-READ-BYTE (IOLIB.STREAMS:DUAL-CHANNEL-GRAY-STREAM)) #<active IPv6 stream socket connected to ::ffff:81.183.137.134/1237 {100F6EEF71}>) looking at netstat shows that there are exactly as many of these as stuck threads: tcp6 0 0 192.168.101.105:8080 81.183.137.134:1249 ESTABLISHED 1004 736341 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1265 ESTABLISHED 1004 736356 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1261 ESTABLISHED 1004 736352 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1237 ESTABLISHED 1004 736329 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1281 ESTABLISHED 1004 736999 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1257 ESTABLISHED 1004 736348 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1273 ESTABLISHED 1004 736363 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1242 ESTABLISHED 1004 736336 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1253 ESTABLISHED 1004 736344 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1269 ESTABLISHED 1004 736360 - do you think that this remote host is really keeping the connection open? if so, do you think that it will help if i do this on the socket returned by accept: (setf (iolib:socket-option stream-socket :receive-timeout) 5) thanks in advance, -- attila

1 0

[iolib-devel] New patches: 27-Feb-2009
by Stelian Ionescu 28 Feb '09

28 Feb '09

commit 5d19d0d03863df65460bae54f1c13ed061f7238a Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:14:45 2009 +0100 Fix printing of paths. src/pathnames/file-path-unix.lisp | 38 +++++++++++++++++------------------- src/pathnames/file-path.lisp | 11 +++++---- 2 files changed, 24 insertions(+), 25 deletions(-) commit 810caf4f1d56761575774fbb2f4ae7e9015da078 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:14:18 2009 +0100 Remove PARSE-FILE-PATH-TYPE. src/pathnames/file-path-unix.lisp | 8 +++----- src/pathnames/file-path.lisp | 20 ++++---------------- 2 files changed, 7 insertions(+), 21 deletions(-) commit cb722bb7a493c14de932b021b6013cc60a269d11 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:10:27 2009 +0100 Export FILE-PATH methods and classes. src/pathnames/pkgdcl.lisp | 29 ++++++++++++++++++++++++++++- 1 files changed, 28 insertions(+), 1 deletions(-) commit e4d402e4b8cdeb5381059710c38102bf0a4c253a Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:04:03 2009 +0100 IOLIB.SOCKETS:CONNECT : keyword arg TIMEOUT has been coalesced into WAIT CHANGES | 1 + src/sockets/base-sockets.lisp | 4 +- src/sockets/socket-methods.lisp | 46 ++++++++++++++++++++------------------ 3 files changed, 27 insertions(+), 24 deletions(-) commit 2f061d87c34ed6a664195d4505c729530eabc53c Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:02:50 2009 +0100 IOLIB.SOCKETS:ACCEPT-CONNECTION : keyword arg TIMEOUT has been coalesced into WAIT CHANGES | 2 ++ src/sockets/base-sockets.lisp | 9 +++++---- src/sockets/socket-methods.lisp | 12 ++++++------ 3 files changed, 13 insertions(+), 10 deletions(-) commit f6aa4c38911bb78ed9ca1157562bbb40415fc6cc Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 10:42:06 2009 +0100 Add WAIT->TIMEOUT helper. src/sockets/common.lisp | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) commit 321f8ec33735e06db2e839c230bede079b516f22 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 10:34:33 2009 +0100 Add types TIMEOUT-DESIGNATOR and POSITIVE-TIMEOUT-DESIGNATOR. src/base/pkgdcl.lisp | 1 + src/base/time.lisp | 6 ++++++ 2 files changed, 7 insertions(+), 0 deletions(-) commit 8adc21d876584aae74b32e2236a2da790c0973dd Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 10:33:37 2009 +0100 Add file-path tests. tests/file-paths-unix.lisp | 129 ++++++++++++++++++++++++++++++++++++++++++++ tests/iolib-tests.asd | 4 +- tests/pkgdcl.lisp | 2 +- 3 files changed, 133 insertions(+), 2 deletions(-) commit 1d7ddd6f1a85ca29b7806dcc4c3d313fd34a7f31 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 10:31:50 2009 +0100 Fix indentation. tests/sockets.lisp | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090227.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New patches: 19-Feb-2009
by Stelian Ionescu 20 Feb '09

20 Feb '09

commit 4b74ac7cbc8f7ab0b22514eb74da5ed2aa395f39 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Thu Feb 19 16:56:06 2009 +0100 Fix use of MAKE-SOCKADDR-UN in SOCKADDR->SOCKADDR-STORAGE. src/sockets/address.lisp | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090219.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] OHLOH
by Stelian Ionescu 19 Feb '09

19 Feb '09

I've added IOLib to ohloh.net: https://www.ohloh.net/p/CL-IOLib/ -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.

1 0

[iolib-devel] the remove-timer issue & redundant multiplexer code
by Matthew Mondor 18 Feb '09

18 Feb '09

The REMOVE-TIMER issue: People here might remember a previous post of mine suspecting a problem with timers of the multiplexer. Well lately with more testing, I've noticed the issue again, and that this only occurs on restartable timers which I want to conditionally disable. After looking at the source, I discovered the obvious reason: my test case used REMOVE-TIMER as part of its timeout handler code, while the time scheduler reschedules/restarts non-:one-shot timers after the handler callback returns. It would be possible to modify the scheduler so that REMOVE-TIMER be callback-safe, but it probably isn't a requirement, considering that for such cases :one-shot can easily be used and as wanted the user code could restart the timer anyway if wanted later on. Thus it is probably best to simply document this in the API, I simply made wrong assumptions about the timer scheduler mechanism. Redundant multiplexer code: This currently only is at the stage of a pipe-dream (not yet a project) but I've sometimes been thinking about rewriting the multiplexer so the main code be more tailored to kqueue, so that redundant sanity checking code and timer scheduler wouldn't have to be used with that backend, and other backends could share a library based on the current code for the scheduler and all the sanity checking. To explain better about the redundancy overhead: - Kqueue drops any registered events for an fd at fd close time. Thus only timers would have to be unregistered (which can be made via a single syscall, or can pend in an array to pass to the next natural single syscall of the main scheduler loop). - Kqueue also allows to use a per-event user-data pointer which can be used to efficiently map to internal user objects, eliminating the need for custom fd->object (or timerid->object) indexing tables. This udata is returned by the framework at both event reception and errors (which are simply provided as events). - Event registration, cancellation and polling are all done in a single call to kevent, and kevent can report if an filter was already registered or not, precisely, removing the need for userland code to maintain redundant tables and related sanity checks. - kqueue also supports timers (one-shot and restarting ones). Restartable timers are atomically restarted so userland doesn't have to care. A timeout callback could use REMOVE-TIMER which would simply add an element to the kevent array which'd be passed to the next syscall in the main loop. Of course if REMOVE-TIMER was declared callback-safe, the userland scheduler for less powerful backends would probably also be fixed to flag timers in a way for them not to be rescheduled. Moving timers to backends which support it would be more efficient than the user rolled timer scheduler code. But then again, I'll use the current iolib multiplexer for now, as long as it works :) -- Matt

1 0

[iolib-devel] asd files move to src/
by Matthew Mondor 18 Feb '09

18 Feb '09

Hi again and thanks for commiting the diffs, Will official releases still have the asd files at the top level like other asdf packages? If not, would it make sense to move those at the first level but to keep the code under src? A script adding all the first level package directories to asdf:*central-registry* can no longer work with iolib >0.6.0 unless it also locates asd files or unless I "mv src/* ." for instance. Although this isn't a major problem, I've been wondering as all other packages I've used have asd files at the first level. In case it was your intention to move the asd files only back to the package's root, here is attached a diff (tested against 20090217's tarball snapshot, and applied after "mv src/*.asd ." to avoid the diff -N mess for such minor changes). Thanks, -- Matt

3 2

[iolib-devel] New patches: 17-Feb-2009
by Stelian Ionescu 18 Feb '09

18 Feb '09

commit 61d0e7cfefa2a688e6c28039cc10303835fdd130 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Tue Feb 17 08:42:36 2009 +0100 Export TIMER from the MULTIPLEXER package. src/multiplex/pkgdcl.lisp | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090217.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] iolib 0.6.0/netbsd/sbcl build failure
by Matthew Mondor 17 Feb '09

17 Feb '09

I decided to upgrade from 20090124 snapshot to 0.6.0 release today, and noticed the netbsd fixes I proposed were checked in, so tried using it, yet there would be a "NO-OP" build error of some kind, which wouldn't occur if I re-evaluated the require form at repl, but then an assertion failure would occur in SB-C about a SIGNED-32 type. I looked again at the source and noticed the fix for netbsd wasn't applied the same as I had applied it on my tree. I had kept :pointer for the udata kevent structure rather than :intptr_t. I thus tried the same on 0.6.0 and it works fine. I propose the attached diff. Thanks, -- Matt

2 4

[iolib-devel] New patches: 16-Feb-2009
by Stelian Ionescu 17 Feb '09

17 Feb '09

commit 3766d9186c7a444f5fcdb1e4c2039303c0953608 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Tue Feb 17 02:22:36 2009 +0100 Fix struct kevent groveling on NetBSD. src/syscalls/ffi-types-unix.lisp | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090216.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New patches: 8-Feb-2009
by Stelian Ionescu 09 Feb '09

09 Feb '09

commit 750592b6449b6b9f887d0cd215300f4317c46cbd Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sun Feb 8 13:47:44 2009 +0100 Better subtype IOLIB-ERROR. CHANGES | 1 + src/sockets/dns/conditions.lisp | 2 +- 2 files changed, 2 insertions(+), 1 deletions(-) commit 00b9598b8826d17515d65bd2793c75b9423f5e20 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sun Feb 8 13:40:29 2009 +0100 Make RESOLVER-ERROR a subtype of ERROR. src/sockets/dns/conditions.lisp | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090208.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] Bug when using READ-LINE on non-blocking fds
by Chaitanya Gupta 08 Feb '09

08 Feb '09

Hi, Iolib behaves in the following way when using READ-LINE with a non-blocking fd: ;; *client-stream is the stream associated with a client connected to our server IRCD> *client-stream #<active IPv6 stream socket connected to ::ffff:127.0.0.1/41460 {1004D5D451}> IRCD> (fd-non-blocking *) T ;; client sends the string "foo" down the socket, no newline is sent IRCD> (ignore-errors (read-line *client-stream)) NIL #<System-Error EWOULDBLOCK(11) "Resource temporarily unavailable"> ;; client sends the string "bar", followed by a newline IRCD> (ignore-errors (read-line *client-stream)) "bar" NIL So, essentially, the string "foo" is lost. Shouldn't the correct behaviour be to wait till a newline arrives on the socket? Chaitanya

3 6

[iolib-devel] New patches: 7-Feb-2009
by Stelian Ionescu 08 Feb '09

08 Feb '09

commit 6eb6bf1b65a376a9242e7380a18f76c5c542973c Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sat Feb 7 21:49:39 2009 +0100 Handle EINTR in %WRITE-OCTETS-FROM-FOREIGN-MEMORY. src/streams/gray/gray-stream-methods.lisp | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) commit 5e4d96947ac12c5ea3c6515e591db2e4972ff78e Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sat Feb 7 21:22:45 2009 +0100 Style change in %FILL-IBUF. src/streams/gray/gray-stream-methods.lisp | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) commit 9a23208f09d15b8fa9f3288048404700bee89fba Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sat Feb 7 20:47:26 2009 +0100 Reading from socket streams with non-blocking FDs now works (bug reported by Chaitanya Gupta) CHANGES | 2 ++ src/streams/gray/gray-stream-methods.lisp | 19 +++++++++++++------ 2 files changed, 15 insertions(+), 6 deletions(-) commit 3af2a2a83195509c3e8d8b5504a436270220a061 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sat Feb 7 16:24:23 2009 +0100 Update the README with info about generating docs. README | 13 ++++++++++++- 1 files changed, 12 insertions(+), 1 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090207.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] [RELEASE] Version 0.6.0
by Stelian Ionescu 06 Feb '09

06 Feb '09

18 months after 0.5.4, I've released 0.6.0: a tarball and its GPG signature are available from http://common-lisp.net/project/iolib/download.shtml . Preliminary documentation can be found here: http://common-lisp.net/project/iolib/manual , but only for the socket library. My next goals are to finish the new stream library(Zeta-streams) and the pathname library. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.

2 4

[iolib-devel] io.multiplex::timer
by Matthew Mondor 06 Feb '09

06 Feb '09

Probably that the timer type should be exported as well in pkgdcl.lisp considering that to use exported remove-timer objects of type timer have to be provided... Thus, it's useful to define a struct or class holding an object of type (or timer null) to remove-timer if t later on (if set to a timer returned by add-timer)? Also, is there, or was there recently a known bug that prevents remove-timer from actually disabling a timer? I have a timer in repeat mode which appears to be triggered even after it was removed... Thanks, -- Matt

1 1

[iolib-devel] New patches: 3-Feb-2009
by Stelian Ionescu 04 Feb '09

04 Feb '09

commit 247b5cada28cd6ba3981e686228823c0d25b2983 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Tue Feb 3 15:50:53 2009 +0100 Add credits. CHANGES | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) commit af10342dbf42f9bda025252c94a062e79e97b620 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Tue Feb 3 15:48:27 2009 +0100 UNIX addresses are now printed as strings. Abstract addresses have a leading #\@ CHANGES | 1 + src/sockets/address.lisp | 7 ++++--- 2 files changed, 5 insertions(+), 3 deletions(-) commit fa2da45427d045f59697cee021b8ae588369a476 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Tue Feb 3 15:38:47 2009 +0100 Add support for abstract UNIX socket addresses. CHANGES | 2 ++ src/sockets/common.lisp | 32 +++++++++++++++++++++++++------- src/sockets/socket-methods.lisp | 8 ++++---- 3 files changed, 31 insertions(+), 11 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090203.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] Abstract sockets
by Julian Stecklina 03 Feb '09

03 Feb '09

Hello, the attached patch fixes abstract sockets for Linux. It least it tries as it does not work: strace output from dbus-monitor: socket(PF_FILE, SOCK_STREAM, 0) = 3 connect(3, {sa_family=AF_FILE, path=@"/tmp/dbus-RhgMDI9FXg"...}, 23) = 0 strace output from my app using iolib: socket(PF_FILE, SOCK_STREAM, 0) = 5 connect(5, {sa_family=AF_FILE, path=@"/tmp/dbus-RhgMDI9FXg"...}, 110) = -1 ECONNREFUSED (Connection refused) It seems as if connect (and bind) have to be passed "exact" sizes as addrlen parameter. But what is a good (i.e. portable) way to compute these? Using (+ 3 (length address)) (counting the address family word, the zero byte, and the string) would be quite a hack... Regards, -- Julian Stecklina Well, take it from an old hand: the only reason it would be easier to program in C is that you can't easily express complex problems in C, so you don't. - Erik Naggum (in comp.lang.lisp)

2 2

[iolib-devel] New patches: 1-Feb-2009
by Stelian Ionescu 02 Feb '09

02 Feb '09

commit 9bbf61ca9871228512ed488179fae1ec99eabbb4 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sun Feb 1 16:49:09 2009 +0100 %SYS-GETENV: return only the variable's value, not its length. src/syscalls/ffi-functions-unix.lisp | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) commit 16d6ee1495d8a02e64ea5b74cf3f352070c96305 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sun Feb 1 16:08:35 2009 +0100 Fix %SYS-SETENV. src/syscalls/ffi-functions-unix.lisp | 13 +++++++++++-- 1 files changed, 11 insertions(+), 2 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090201.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New patches: 31-Jan-2009
by Stelian Ionescu 01 Feb '09

01 Feb '09

commit ce53965b9a3864c772d4423c013ed07b36997ea4 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sun Feb 1 02:15:22 2009 +0100 Add entries about the last bug fixes in the multiplexer. CHANGES | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) commit ae6922c24f0dc958087eca74020bceebeb208a05 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sun Feb 1 02:07:36 2009 +0100 Fix select() multiplexer: possible fd lossage in RECALC-FD-MASKS. src/multiplex/backend-select.lisp | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) commit e4c7ae934c06776386e23bd66a7cf29f26f9eb21 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sat Jan 31 22:03:29 2009 +0100 Fill poll timeout calculation in EVENT-DISPATCH, minor other fixes. src/iolib.multiplex.asd | 9 ++++----- src/multiplex/event-loop.lisp | 26 ++++++++++++++++---------- src/multiplex/scheduler.lisp | 4 +++- src/multiplex/time.lisp | 19 ------------------- src/multiplex/timers.lisp | 15 +++++++++++---- src/multiplex/utils.lisp | 6 ------ 6 files changed, 34 insertions(+), 45 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090131.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0