- iolib-devel - mailman3.common-lisp.net

[iolib-devel] poll error leaking from write-sequence
by Attila Lendvai 19 Nov '09

19 Nov '09

Stelian, i think it's not against the rules to signal other exceptions than stream-error from sequence calls, but it's causing me more and more headaches. the situation: i want to log errors happening while our web server is serving requests, but i want to keep the error log noise-free. network related errors are considered noise, so i have a function that decides whether the error is coming from the network stream connected to the client: (def function is-error-from-client-stream? (error client-stream) (bind ((client-stream-fd (iolib:fd-of client-stream))) (or (and (typep error 'stream-error) (eq (stream-error-stream error) client-stream)) ;; TODO the rest is fragile and easily breaks when iolib changes behavior (and client-stream-fd (and (typep error 'iolib:socket-error) (eql client-stream-fd (isys:handle-of error))) ;; TODO signalling non stream-error conditions might be an iolib bug (and (typep error 'iolib.multiplex:poll-error) (eql client-stream-fd (iolib.multiplex:poll-error-fd error))))))) this function keeps getting longer, but the bigger issue is that it highly depends on iolib's internal implementation. is there any chance to add it to iolib's contracts that it only signals stream-error from stream calls like write-sequence? it could chain the original condition not to lose information... or do you have some other suggestions? -- attila ps: the new error that started to show up after i temporarily made the client sockets blocking until we fully implement connection multiplexing (the error is most probably that the connection was reset, client cancelled the download): Error caught while polling file descriptor 9: Unknown error *** Backtrace: 6: (HANDLE-TOPLEVEL-ERROR #<HOME-APPLICATION \"/\" 0 {1000FD4429}> #<POLL-ERROR {100B7CB041}>) 7: ((FLET HU.DWIM.WUI::HANDLE-REQUEST-ERROR) #<POLL-ERROR {100B7CB041}>) 8: ((LABELS HU.DWIM.UTIL::HANDLE-LEVEL-1-ERROR) #<POLL-ERROR {100B7CB041}>) 9: (SIGNAL #<POLL-ERROR {100B7CB041}>) [:EXTERNAL] 10: (ERROR POLL-ERROR) [:EXTERNAL] 11: (IOLIB.MULTIPLEX::PROCESS-POLL-REVENTS 24 9) 12: (WAIT-UNTIL-FD-READY 9 :OUTPUT NIL T) 13: (IOLIB.STREAMS::%WRITE-OCTETS-FROM-FOREIGN-MEMORY IOLIB.SOCKETS::SOCKET-WRITE-FN 9 #.(SB-SYS:INT-SAP #X100B7C7010) 8192) 14: (IOLIB.STREAMS::%WRITE-SIMPLE-ARRAY-UB8 #<active IPv6 stream socket, unconnected {100B617891}> #(173 24 188 120 227 48 246 217 166 31 219 246 223 156 15: (WRITE-SEQUENCE #(173 24 188 120 227 48 246 217 166 31 219 246 223 156 67 143 56 133 143 102 199 237 58 116 214 169 16 138 87 110 26 198 192 45 54 96 16: ((LABELS HU.DWIM.WUI::WITH-CONTENT-SERVING-LOGIC-BODY) ((\"Content-Length\" . \"1957376\") (\"Content-Type\" . \"application/msword\") (\"Date\" . \" 17: (HU.DWIM.WUI::SERVE-FILE #P\"/var/opt/darcs/ebr42-hydrogen/documents/ebr42 - Felhasználói kézikönyv önkormányzatoknak.doc\") [:EXTERNAL]

2 1

[iolib-devel] c10k HTTP server with iolib
by Red Daly 06 Nov '09

06 Nov '09

Hi, I was wondering how best to use IOLib to write an efficient HTTP server that can handle perhaps 10,000+ simultaneous connections. It seems like iolib has all the right ingredients: a sytem-level sockets interface and a io multiplexer that uses epoll/kqueue for efficiently querying sockets. There is quite a bit of code already written so I was hoping for some advice about how this would be best implemented. Here is a possible architecture for a server that can handle tons of connections at once: Some lisp thread originally sets up a passive-socket (server socket) to listen for connections on some port. There are a few worker threads (on the order of the number of processors in the machine). When a connection is received, one of these worker threads will dequeue an active socket with ACCEPT. However, after the initial connection all the HTTP headers and content must be read from the socket. Presumably not all the data will be ready as soon as a connection is received, and read operations will block if allowed to block. While it waits for the full HTTP request to come across the wire, the worker thread could be accepting new connections or processing older ones where the fully request is available. To quickly send HTTP responses off, writes to sockets should also never block--so if we try to send more bytes than a socket can handle, we should handle that asynchronously so the worker can get on to the next thing to do. So, a worker thread will either 1) be processing a request (arbitrary lisp code to respond appropriately to an HTTP request). When a response is is ready, it should be written to some non-blocking gray stream. 2) be waiting for the next of the following events: a) socket writable. Some gray stream that was written to in (1) but blocked now has enough room in its buffers to allow more data to be sent immediately. (ie a "would block" message was received in a "send to" call). b) socket readable. Some socket that we are listening to (gray stream) has more data available. When this data is sufficient to respond to the request, this connection is now elgible for the processing described in (1). c) socket accepted. Some socket is available in the queue of the passive socket. We can now begin listening to the socket's read events for processing as in (2.b) or process the socket as in (1). My question about IOLib is how this sort of processing model could be implemented on top of iolib. Do passive sockets generate epoll/kqueue events when a new connection is available to accept? If so it seems like the multiplexer could be used to listen for events 2.a,2.b, and 2.c all simultaneously. I see there are some gray stream implementations in the code right now, though I have not figured out how to use them. How do I, for example, create a stream with an underlying socket? Could these sockets work with the multiplexer implementation to accomplish the processing model described? I think that sums it up. Thanks for the great library! -Red

3 3

[iolib-devel] New patches: 22-Oct-2009
by Stelian Ionescu 23 Oct '09

23 Oct '09

commit e8b89bfbc6bf79704cc4af90facb2b4dc43033bb Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Thu Oct 22 14:49:11 2009 +0200 Various improvements to WALK-DIRECTORY. src/os/os-unix.lisp | 61 +++++++++++++++++++++----------------------------- 1 files changed, 26 insertions(+), 35 deletions(-) commit 845705a181458892c8ee1b9c0b5269ef79a61da3 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Thu Oct 22 14:48:26 2009 +0200 Some optimizations in CSTRING code. src/syscalls/unix-syscall-path-strings.lisp | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) commit 403eae5d5937c259f9c1dba3389a95a5ce560564 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Thu Oct 22 14:46:53 2009 +0200 Fix typos. src/syscalls/unix-syscall-path-strings.lisp | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) commit 85d0c5ea877db9bbcbc679c912ea551ad1031398 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Mon Oct 19 16:44:23 2009 +0200 Whitespace. src/iolib.os.asd | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) commit e0f7110bfc785654882e609b32e8366c6a2f5039 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sun Oct 18 04:36:53 2009 +0200 Pass CONSTANTP the current environment when using it in a macro. src/sockets/common.lisp | 4 ++-- src/sockets/make-socket.lisp | 10 ++++++---- src/sockets/socket-methods.lisp | 8 ++++---- 3 files changed, 12 insertions(+), 10 deletions(-) commit 08b41f407b167824b7453b52cabdf684072b4703 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sun Oct 18 03:10:50 2009 +0200 Update my email address in the LICENCE file. LICENCE | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20091022.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New patches: 3-Oct-2009
by Stelian Ionescu 04 Oct '09

04 Oct '09

commit 188eb91ba7fd3ee8a82fc3f757b23e75f216e6c7 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sat Oct 3 22:50:02 2009 +0200 Remove ABSOLUTE-PATHS arg of LIST-DIRECTORY. src/os/os-unix.lisp | 9 +++------ 1 files changed, 3 insertions(+), 6 deletions(-) commit 4aa373942ad3f5937925de8559bb7a10b5493557 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sat Oct 3 22:30:58 2009 +0200 Various improvements to WALK-DIRECTORY. src/os/os-unix.lisp | 29 ++++++++++++++++++----------- 1 files changed, 18 insertions(+), 11 deletions(-) commit 31be397354c6ab532d430a4dfaeb52d78b235956 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sat Oct 3 22:30:35 2009 +0200 Fix DELETE-FILES. src/os/os-unix.lisp | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) commit 55c56a7b7c718fcfc9fbfba19adc44faf31c51a6 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sat Oct 3 22:29:20 2009 +0200 WALK-DIRECTORY: sanity check for MINDEPTH and MAXDEPTH. src/os/os-unix.lisp | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) commit c8ac1165aa738812f648dc685bdf9221ea827cae Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sat Oct 3 22:28:37 2009 +0200 Don't use WITH-CURRENT-DIRECTORY in CALL-WITH-DIRECTORY-ITERATOR. src/os/os-unix.lisp | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-) commit ce60dd622b1fab6b2c1deb7a011e78cdb069e566 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sat Oct 3 22:27:25 2009 +0200 Export ABSOLUTE-FILE-PATH, make the second arg optional. src/os/os-unix.lisp | 3 ++- src/os/pkgdcl.lisp | 1 + 2 files changed, 3 insertions(+), 1 deletions(-) commit 9b21be3f21070ca1d26473f24c5a9c678be6c613 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sat Oct 3 22:25:59 2009 +0200 Remove type STRING-DESIGNATOR which is now in Alexandria. src/base/pkgdcl.lisp | 1 - src/base/types.lisp | 3 --- 2 files changed, 0 insertions(+), 4 deletions(-) commit 3c715da87796e326b89e3b06356d17da96b6aec2 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Sat Oct 3 22:24:36 2009 +0200 Fix stat(2) wrappers on Linux. Thanks to David Lichtblau who reported this bug. src/syscalls/ffi-functions-unix.lisp | 3 --- src/syscalls/ffi-types-unix.lisp | 3 +++ 2 files changed, 3 insertions(+), 3 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20091003.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New patches: 2-Oct-2009
by Stelian Ionescu 03 Oct '09

03 Oct '09

commit 18499d242d2dffe80fbeae14642cfc712afa3d51 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Fri Oct 2 13:52:02 2009 +0200 WALK-DIRECTORY: fix :IF-DOES-NOT-EXIST src/os/os-unix.lisp | 32 +++++++++++++++----------------- 1 files changed, 15 insertions(+), 17 deletions(-) commit 964aed931fab2f82b4ca5ca2c354b8e02e070a8a Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Fri Oct 2 13:51:36 2009 +0200 WALK-DIRECTORY: fix recursion. src/os/os-unix.lisp | 29 +++++++++++++++-------------- 1 files changed, 15 insertions(+), 14 deletions(-) commit d47be62eafdda89325110da3c39c51ca8b60be4c Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Fri Oct 2 12:57:28 2009 +0200 Add DELETE-FILES. src/os/os-unix.lisp | 19 +++++++++++++++++++ src/os/pkgdcl.lisp | 3 +-- 2 files changed, 20 insertions(+), 2 deletions(-) commit 50a8a8b27ba02d78ac8b353b6964c8cf4edf83b6 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 19:28:30 2009 +0200 Fix FILE-KIND. src/os/os-unix.lisp | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20091002.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New patches: 30-Sep-2009
by Stelian Ionescu 01 Oct '09

01 Oct '09

commit 75d8e243888f1b95e77f4ae43891f86a81a3edc8 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 19:08:15 2009 +0200 WALK-DIRECTORY: provide restart ignore-file-system-error in case FN signals an error src/os/os-unix.lisp | 15 ++++++++++----- 1 files changed, 10 insertions(+), 5 deletions(-) commit 6c3be885b07a97e8938e33d3a654c59ae2a3a5c0 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 16:54:44 2009 +0200 WALK-DIRECTORY: rename :ORDER to :DIRECTORIES src/os/os-unix.lisp | 14 ++++++++------ 1 files changed, 8 insertions(+), 6 deletions(-) commit 08c0102e96331d778c3fe22f58eeaae515eee07c Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 16:01:08 2009 +0200 More fixes to WALK-DIRECTORY. src/os/os-unix.lisp | 98 +++++++++++++++++++++++++-------------------------- 1 files changed, 48 insertions(+), 50 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090930.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] Termios support - new variant
by marsijanin 30 Sep '09

30 Sep '09

Good ${current_time_of_day} again! I've made some changes: 0 - cffi-greovel file with termios constants (icanon etc.) is generating automatically from constants list; 1 - non posix features, such iutf8 are `#+linux' or `#+bsd' marked. I've run it only under linux, and all `#+bsd' marked features still need to be checking under BDS box (currently i don't have any). Still hooping, that it can be usefully, Nikolay V. Razbegaiev.

1 0

[iolib-devel] New patches: 29-Sep-2009
by Stelian Ionescu 30 Sep '09

30 Sep '09

commit 19b2ee4de250348d4ae8b325f9476168bda96f74 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 05:41:17 2009 +0200 Fix WALK-DIRECTORY. src/os/os-unix.lisp | 18 +++++++++--------- 1 files changed, 9 insertions(+), 9 deletions(-) commit 93b1d83747453112fdf174b12711a16b0432514b Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 05:28:43 2009 +0200 Compile tests at definition-time. tests/base.lisp | 14 ++-- tests/events.lisp | 10 +- tests/file-paths-unix.lisp | 178 ++++++++++++++++++++++---------------------- tests/sockets.lisp | 124 +++++++++++++++--------------- tests/streams.lisp | 2 +- 5 files changed, 164 insertions(+), 164 deletions(-) commit 2b4e7f376824d0f906883bb354574caeb87c80ce Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 04:34:26 2009 +0200 Add CALL-WITH-DIRECTORY-ITERATOR, WITH-DIRECTORY-ITERATOR, MAPDIR, LIST-DIRECTORY and WALK-DIRECTORY. src/os/os-unix.lisp | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++ src/os/pkgdcl.lisp | 10 ++-- 2 files changed, 132 insertions(+), 5 deletions(-) commit 428ce5ee6f8348b79c30c447c347ceb9c4ef02eb Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 02:22:38 2009 +0200 Don't use IOLIB.SYSCALLS in IOLIB.OS src/os/os-unix.lisp | 91 ++++++++++++++++++++++++++------------------------- src/os/pkgdcl.lisp | 2 +- 2 files changed, 47 insertions(+), 46 deletions(-) commit a2b41c0102cba546dd5ac4c322a81027d0c42503 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 01:04:47 2009 +0200 Handle ELOOP in GET-FILE-KIND. src/os/os-unix.lisp | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) commit 1811a0137ed802a605496d86d30ceecaeb8031ca Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 00:42:59 2009 +0200 Rename READ-LINK to READ-SYMLINK, split MAKE-LINK into MAKE-SYMLINK and MAKE-HARDLINK. src/os/os-unix.lisp | 44 +++++++++++++++++++++++++++----------------- src/os/pkgdcl.lisp | 5 +++-- 2 files changed, 30 insertions(+), 19 deletions(-) commit 585cca2b1b410ece0cefe644b2adf11028321118 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 30 00:31:58 2009 +0200 Get rid of the trailing slash in FILE-PATH namestrings. src/os/os-unix.lisp | 3 +-- src/pathnames/file-path-unix.lisp | 34 ++++++++++++---------------------- src/pathnames/file-path.lisp | 10 ++-------- tests/file-paths-unix.lisp | 24 ++++++++++++------------ 4 files changed, 27 insertions(+), 44 deletions(-) commit 4a368795851e5709a5dfa3d5ee9d95c1d2e871e4 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Tue Sep 29 21:57:58 2009 +0200 Fix docstrings: FILE-PATH objects cannot be wild. src/os/os-unix.lisp | 17 ++++++----------- 1 files changed, 6 insertions(+), 11 deletions(-) commit fce9618dad0a443df9f39b0084cdcc35599be7b8 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Mon Sep 28 23:51:00 2009 +0200 Better error message. src/pathnames/file-path-unix.lisp | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) commit 9d2a9259e574d50186f0ef47000feda686aaff47 Author: Stelian Ionescu <sionescu(a)cddr.org> Date: Wed Sep 23 22:18:19 2009 +0200 Eliminate AS-DIRECTORY argument of PARSE-FILE-PATH. src/os/os-unix.lisp | 8 +-- src/pathnames/file-path-unix.lisp | 21 +++---- tests/file-paths-unix.lisp | 112 ++++--------------------------------- 3 files changed, 23 insertions(+), 118 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20090929.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git

1 0

[iolib-devel] New bugtracker
by Stelian Ionescu 29 Sep '09

29 Sep '09

I've created a project on Launchpad, so if you fancy there's https://launchpad.net/iolib now -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib

1 0

[iolib-devel] Termios (3p) support for iolib (proof of concept)
by marsijanin＠gmail.com 19 Sep '09

19 Sep '09

Good ${time_of_the_day}! I wrote some code that implement API for working with serial interface devices (RS-232/422/485), i.e. termios API. It's covering now only posix specific features of the termios specification (i.e. no iutf8 etc.). I hope, the following code can be useful.

1 0