Hi binghe,
I found my old note in :iolib.usocket:
;;; At first, I tried to ;;; implement an iolib backend for usocket but found that was not the ;;; way to go due to heavy posix bias of the usocket backend ;;; interface. Implementing the usocket API directly is better.
The usocket internal backend api was not a good a match for iolib, that's why the iolib.usocket and not usocket.iolib.
On 06/24/2013 01:11 PM, Chun Tian (binghe) wrote:
On 24/giu/2013, at 18:49, Tomas Hlavaty tomas.hlavaty@knowledgetools.de wrote:
Hi binghe,
the portability layer lives in the package :iolib.usocket and implements the usocket api. How would you implement and api without emulating another package's interface? Isn't the api the package's interface? If the user loads iolib.usocket.asd, what else should he expect instead of usocket with all functionalities of usocket in it?
It works, but if you try to depend on another package which depend on the real usocket, you'll got conflicts. Therefore such approaches cannot goes into Quicklisp as a common solution, I think.
There are no conflicts. The user either loads iolib.usocket.asd or not. There is no such concept as "trying to depend on another package which depend on the real usocket" because the dependency is on the API, not on the implementation.
iolib.usocket.asd creates an alias system and package so that users don't have to make any explicit code changes to their own code base.
yes, usocket is brilliant for covering the platforms where cffi and iolib are not available/desirable.
I should have also mentioned, that the most brilliant part of usocket is that it provides an API on which most common lisp libraries agree:-)
The problem with the wait-* functions is that they are inherently non-portable (posix specific) and also assume certain programming model which is so far incompatible with wide-spread common lisp libraries. For example, I added a timeout argument to socket-accept so that hunchentoot can work without assuming posix (on winapi). Going forward, it might be worth acknowledging different programming models and embrace them in the usocket api.
I don't quite under this part. usocket's WAIT-FOR-INPUT, depends on select() on UNIX-like systems, and WSAEventSelect() on Windows. When setting the timeout to zero, it should equals to poll()/epoll(). It's design and firstly implemented by original usocket authors, I think it's quite useful. The blocking approach is also supported when user can live with timeouts.
I guess you're using Hunchentoot with the usocket emulations based on IOlib, but you didn't implement WAIT-FOR-INPUT, right? I think that's because IOlib never port their powerful I/O multiplexing interfaces to Windows, so far it only support three OS-provided functions: select (on general Unix), epoll (on Linux) and kqueue (on BSD systems). Maybe you should encourage IOlib author to support "WSAEventSelect", then you don't need to use a timeout arguemnt to socket-accept any more...
There is only one place in hunchentoot, which depends on a wait-* function and it is to implement timeout on socket-accept because this posix idea of how timeouts are implemented leaks through usocket api iirc. If you look at many common lisp libraries, they dont have a concept of waiting. This can be fixed by adding a timeout to the socket-accept so that library code can clearly state its intent without imposing a specific (non-portable) implementation strategy. (If you really think wait-* is the right thing, why some usocket functions have the timeout argument and some are supposed to use the wait-* function?)
When I was porting iolib to windows, I found it useful to identify several programming models (summarized in the iolib README found in the iolib porting repo).
1 traditional blocking 2 traditional blocking with timeouts 3 select based 4 io completion ports based
Common Lisp libraries have basically code for 1 and 2 which are highly portable and lead to natural programming model. 3 and 4 require radically different programming models and are pretty much not used (except when 3 leaks through a broken api as in case of usocket and thus hunchentoot). Some people try to optimize by programing in 3 but there is a heavy price for that in terms of how one writes code and related consequences. I think 3 also in principle imposes at least one copy of io buffers unlike 4 where data don't have to be copied at all (from optimisation point of view 4 would be superior to posix way of doing it; from programming point of view, code is even worse than 3). Until it is clearer what an api for 3 and 4 should be, I think wait-* should not be in usocket. I actually ported iolib to 4 (io completion port backend) but haven't found a good way of integrating it with lisp so far, and using it just to emulate 1 and 2 seems stupid. 3 could be emulated on winapi I think, but then wouldn't 4 better way to go? Or maybe wouldn't a good api for 3 be a subset of api for 4? So far, there is no lisp code that could use 3 and 4, and it's not clear to me how such code would be made usable/interoperable in the context of common lisp.
For example, how does your code deal with partial input/output (a concept introduced by 3 and 4 programming models)?
There are some options:
a) buffer enough so that the partial io concept disappears above the io loop b) CPS - automatic - manual c) suspendable streams
a) is a good cheat. b) are very bad idea imho. c) is the thing I would like to explore and is the way I wish it works out in the future.
I would be really interested in how do you deal with that to write non-trivial software while exploiting the optimisation of posix select/(e)poll/kqueue. A web server that uses 3 or 4 to send bytes without any need for deep integration with the lisp environment is trivial from conceptual point of view. Every single attempt I've seen to use 3 so far has been of this kind without really addressing and solving the partial io issue. iolib doesn't really address this issue either, it works only for 1 and 2 programming models. if you want to go 3 or 4 with iolib, you get the ffi wrappers for the posix calls but as far as streams are concerned, iolib users are on their own.
Cheers,
Tomas