February 2009 - iolib-devel - mailman3.common-lisp.net

[iolib-devel] stuck sockets
by Attila Lendvai 28 Feb '09

28 Feb '09

Stelian, i need your help. our server went non-responsive and my investigation results in this: all worker threads were waiting here: 15: (IOLIB.STREAMS::%FILL-IBUF IOLIB.SOCKETS::SOCKET-READ-FN 15 #S(IOLIB.STREAMS::IOBUF :DATA #.(SB-SYS:INT-SAP #X00658620) :SIZE 4096 :START 0 :END 0)) 16: ((SB-GRAY:STREAM-READ-BYTE (IOLIB.STREAMS:DUAL-CHANNEL-GRAY-STREAM)) #<active IPv6 stream socket connected to ::ffff:81.183.137.134/1237 {100F6EEF71}>) looking at netstat shows that there are exactly as many of these as stuck threads: tcp6 0 0 192.168.101.105:8080 81.183.137.134:1249 ESTABLISHED 1004 736341 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1265 ESTABLISHED 1004 736356 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1261 ESTABLISHED 1004 736352 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1237 ESTABLISHED 1004 736329 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1281 ESTABLISHED 1004 736999 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1257 ESTABLISHED 1004 736348 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1273 ESTABLISHED 1004 736363 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1242 ESTABLISHED 1004 736336 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1253 ESTABLISHED 1004 736344 - tcp6 0 0 192.168.101.105:8080 81.183.137.134:1269 ESTABLISHED 1004 736360 - do you think that this remote host is really keeping the connection open? if so, do you think that it will help if i do this on the socket returned by accept: (setf (iolib:socket-option stream-socket :receive-timeout) 5) thanks in advance, -- attila

1 0

[iolib-devel] New patches: 27-Feb-2009
by Stelian Ionescu 28 Feb '09

28 Feb '09

commit 5d19d0d03863df65460bae54f1c13ed061f7238a Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:14:45 2009 +0100 Fix printing of paths. src/pathnames/file-path-unix.lisp | 38 +++++++++++++++++------------------- src/pathnames/file-path.lisp | 11 +++++---- 2 files changed, 24 insertions(+), 25 deletions(-) commit 810caf4f1d56761575774fbb2f4ae7e9015da078 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:14:18 2009 +0100 Remove PARSE-FILE-PATH-TYPE. src/pathnames/file-path-unix.lisp | 8 +++----- src/pathnames/file-path.lisp | 20 ++++---------------- 2 files changed, 7 insertions(+), 21 deletions(-) commit cb722bb7a493c14de932b021b6013cc60a269d11 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:10:27 2009 +0100 Export FILE-PATH methods and classes. src/pathnames/pkgdcl.lisp | 29 ++++++++++++++++++++++++++++- 1 files changed, 28 insertions(+), 1 deletions(-) commit e4d402e4b8cdeb5381059710c38102bf0a4c253a Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:04:03 2009 +0100 IOLIB.SOCKETS:CONNECT : keyword arg TIMEOUT has been coalesced into WAIT CHANGES | 1 + src/sockets/base-sockets.lisp | 4 +- src/sockets/socket-methods.lisp | 46 ++++++++++++++++++++------------------ 3 files changed, 27 insertions(+), 24 deletions(-) commit 2f061d87c34ed6a664195d4505c729530eabc53c Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 11:02:50 2009 +0100 IOLIB.SOCKETS:ACCEPT-CONNECTION : keyword arg TIMEOUT has been coalesced into WAIT CHANGES | 2 ++ src/sockets/base-sockets.lisp | 9 +++++---- src/sockets/socket-methods.lisp | 12 ++++++------ 3 files changed, 13 insertions(+), 10 deletions(-) commit f6aa4c38911bb78ed9ca1157562bbb40415fc6cc Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 10:42:06 2009 +0100 Add WAIT->TIMEOUT helper. src/sockets/common.lisp | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) commit 321f8ec33735e06db2e839c230bede079b516f22 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 10:34:33 2009 +0100 Add types TIMEOUT-DESIGNATOR and POSITIVE-TIMEOUT-DESIGNATOR. src/base/pkgdcl.lisp | 1 + src/base/time.lisp | 6 ++++++ 2 files changed, 7 insertions(+), 0 deletions(-) commit 8adc21d876584aae74b32e2236a2da790c0973dd Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 10:33:37 2009 +0100 Add file-path tests. tests/file-paths-unix.lisp | 129 ++++++++++++++++++++++++++++++++++++++++++++ tests/iolib-tests.asd | 4 +- tests/pkgdcl.lisp | 2 +- 3 files changed, 133 insertions(+), 2 deletions(-) commit 1d7ddd6f1a85ca29b7806dcc4c3d313fd34a7f31 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Fri Feb 27 10:31:50 2009 +0100 Fix indentation. tests/sockets.lisp | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090227.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New patches: 19-Feb-2009
by Stelian Ionescu 20 Feb '09

20 Feb '09

commit 4b74ac7cbc8f7ab0b22514eb74da5ed2aa395f39 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Thu Feb 19 16:56:06 2009 +0100 Fix use of MAKE-SOCKADDR-UN in SOCKADDR->SOCKADDR-STORAGE. src/sockets/address.lisp | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090219.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] OHLOH
by Stelian Ionescu 19 Feb '09

19 Feb '09

I've added IOLib to ohloh.net: https://www.ohloh.net/p/CL-IOLib/ -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.

1 0

[iolib-devel] the remove-timer issue & redundant multiplexer code
by Matthew Mondor 18 Feb '09

18 Feb '09

The REMOVE-TIMER issue: People here might remember a previous post of mine suspecting a problem with timers of the multiplexer. Well lately with more testing, I've noticed the issue again, and that this only occurs on restartable timers which I want to conditionally disable. After looking at the source, I discovered the obvious reason: my test case used REMOVE-TIMER as part of its timeout handler code, while the time scheduler reschedules/restarts non-:one-shot timers after the handler callback returns. It would be possible to modify the scheduler so that REMOVE-TIMER be callback-safe, but it probably isn't a requirement, considering that for such cases :one-shot can easily be used and as wanted the user code could restart the timer anyway if wanted later on. Thus it is probably best to simply document this in the API, I simply made wrong assumptions about the timer scheduler mechanism. Redundant multiplexer code: This currently only is at the stage of a pipe-dream (not yet a project) but I've sometimes been thinking about rewriting the multiplexer so the main code be more tailored to kqueue, so that redundant sanity checking code and timer scheduler wouldn't have to be used with that backend, and other backends could share a library based on the current code for the scheduler and all the sanity checking. To explain better about the redundancy overhead: - Kqueue drops any registered events for an fd at fd close time. Thus only timers would have to be unregistered (which can be made via a single syscall, or can pend in an array to pass to the next natural single syscall of the main scheduler loop). - Kqueue also allows to use a per-event user-data pointer which can be used to efficiently map to internal user objects, eliminating the need for custom fd->object (or timerid->object) indexing tables. This udata is returned by the framework at both event reception and errors (which are simply provided as events). - Event registration, cancellation and polling are all done in a single call to kevent, and kevent can report if an filter was already registered or not, precisely, removing the need for userland code to maintain redundant tables and related sanity checks. - kqueue also supports timers (one-shot and restarting ones). Restartable timers are atomically restarted so userland doesn't have to care. A timeout callback could use REMOVE-TIMER which would simply add an element to the kevent array which'd be passed to the next syscall in the main loop. Of course if REMOVE-TIMER was declared callback-safe, the userland scheduler for less powerful backends would probably also be fixed to flag timers in a way for them not to be rescheduled. Moving timers to backends which support it would be more efficient than the user rolled timer scheduler code. But then again, I'll use the current iolib multiplexer for now, as long as it works :) -- Matt

1 0

[iolib-devel] asd files move to src/
by Matthew Mondor 18 Feb '09

18 Feb '09

Hi again and thanks for commiting the diffs, Will official releases still have the asd files at the top level like other asdf packages? If not, would it make sense to move those at the first level but to keep the code under src? A script adding all the first level package directories to asdf:*central-registry* can no longer work with iolib >0.6.0 unless it also locates asd files or unless I "mv src/* ." for instance. Although this isn't a major problem, I've been wondering as all other packages I've used have asd files at the first level. In case it was your intention to move the asd files only back to the package's root, here is attached a diff (tested against 20090217's tarball snapshot, and applied after "mv src/*.asd ." to avoid the diff -N mess for such minor changes). Thanks, -- Matt

3 2

[iolib-devel] New patches: 17-Feb-2009
by Stelian Ionescu 18 Feb '09

18 Feb '09

commit 61d0e7cfefa2a688e6c28039cc10303835fdd130 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Tue Feb 17 08:42:36 2009 +0100 Export TIMER from the MULTIPLEXER package. src/multiplex/pkgdcl.lisp | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090217.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] iolib 0.6.0/netbsd/sbcl build failure
by Matthew Mondor 17 Feb '09

17 Feb '09

I decided to upgrade from 20090124 snapshot to 0.6.0 release today, and noticed the netbsd fixes I proposed were checked in, so tried using it, yet there would be a "NO-OP" build error of some kind, which wouldn't occur if I re-evaluated the require form at repl, but then an assertion failure would occur in SB-C about a SIGNED-32 type. I looked again at the source and noticed the fix for netbsd wasn't applied the same as I had applied it on my tree. I had kept :pointer for the udata kevent structure rather than :intptr_t. I thus tried the same on 0.6.0 and it works fine. I propose the attached diff. Thanks, -- Matt

2 4

[iolib-devel] New patches: 16-Feb-2009
by Stelian Ionescu 17 Feb '09

17 Feb '09

commit 3766d9186c7a444f5fcdb1e4c2039303c0953608 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Tue Feb 17 02:22:36 2009 +0100 Fix struct kevent groveling on NetBSD. src/syscalls/ffi-types-unix.lisp | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090216.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New patches: 8-Feb-2009
by Stelian Ionescu 09 Feb '09

09 Feb '09

commit 750592b6449b6b9f887d0cd215300f4317c46cbd Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sun Feb 8 13:47:44 2009 +0100 Better subtype IOLIB-ERROR. CHANGES | 1 + src/sockets/dns/conditions.lisp | 2 +- 2 files changed, 2 insertions(+), 1 deletions(-) commit 00b9598b8826d17515d65bd2793c75b9423f5e20 Author: Stelian Ionescu <sionescu(a)common-lisp.net> Date: Sun Feb 8 13:40:29 2009 +0100 Make RESOLVER-ERROR a subtype of ERROR. src/sockets/dns/conditions.lisp | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090208.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0