[iolib-devel] On iolib-posix, OSICAT and FFI interfaces
hi, I'd like to ask your advice concerning how to implement a good OS interface. I and luis are working on the FFI interface to the OS which is part of IOlib(iolib-posix) and we are a bit undecided over the approach to take especially regarding how to handle IN/OUT or OUT struct arguments: 1) return multiple arguments, the first one being the foreign function's return value(if it's not "void"), the rest being the struct members unpacked. this has the advantage of being mainly non-consing - except maybe for large numbers of values - IIRC with stat() I get 14 multiple values. If I'm not mistaken most implementation stack-allocate multiple values(perhaps only up to a certain number ?) 2) copy each struct into a structure object or CLOS object(like sb-posix). this has the disadvantage of consing one or more objects with each call 3) make two packages: one with an interface very close to C, without unpacking or consing objects, ie. STAT would need to be passed a foreign pointer to a "struct stat" and the programmer would use directly CFFI to retrieve desired data. the second interface would be a high-level one, lispy in the sense of having ENVIRONMENT-VARIABLE and (SETF ENVIRONMENT-VARIABLE) instead of GETENV and SETENV. the function names would be quite different from the OS names also because we could have something like (SETF FILE-STAT) as "frontend" for several syscalls such as chown, chmod and utime (example taken from http://clisp.cons.org/impnotes/syscalls.html). in all 3 options I imply reporting errors via conditions the choice I prefer is n° 3, because it allows me to have both a (mostly) non-consing interface for when I really need speed and a lispy interface that *most* people would want to use since dealing with a FFI is most often obnoxious, and yes, I'm a bit obsessed by speed/efficiency :) among existing libraries, OSICAT seems to be the one closer to what I prefer and we(luis and I) have been thinking of continuing development of OSICAT if you agree, abandoning iolib-posix on favour of something like osicat-posix-ffi and osicat-windows-ffi - the low-level side of it suggestions ? -- (sign :name "Stelian Ionescu" :aka "fe[nl]ix" :quote "Quidquid latine dictum sit, altum videtur.")
On 7/28/07, Stelian Ionescu <sionescu@common-lisp.net> wrote:
1) return multiple arguments, the first one being the foreign function's return value(if it's not "void"), the rest being the struct members unpacked. this has the advantage of being mainly non-consing - except maybe for large numbers of values - IIRC with stat() I get 14 multiple values. If I'm not mistaken most implementation stack-allocate multiple values(perhaps only up to a certain number ?)
This is good for some things with a sane number of values -- like pipe(). For others like stat() less so. Returning a "handfull" of values tends to be very fast, yes.
2) copy each struct into a structure object or CLOS object(like sb-posix). this has the disadvantage of consing one or more objects with each call.
Yeah. You forget the possibility of using an optional argument to pass in a pre-allocated object, though: (let ((stat (make-instance 'stat))) (loop for f in paths do (frob (stat path stat))))
3) make two packages: one with an interface very close to C, without unpacking or consing objects, ie. STAT would need to be passed a foreign pointer to a "struct stat" and the programmer would use directly CFFI to retrieve desired data. the second interface would be a high-level one, lispy in the sense of having ENVIRONMENT-VARIABLE and (SETF ENVIRONMENT-VARIABLE) instead of GETENV and SETENV. the function names would be quite different from the OS names also because we could have something like (SETF FILE-STAT) as "frontend" for several syscalls such as chown, chmod and utime (example taken from http://clisp.cons.org/impnotes/syscalls.html).
in all 3 options I imply reporting errors via conditions
the choice I prefer is n° 3, because it allows me to have both a (mostly) non-consing interface for when I really need speed and a lispy interface that *most* people would want to use since dealing with a FFI is most often obnoxious, and yes, I'm a bit obsessed by speed/efficiency :)
among existing libraries, OSICAT seems to be the one closer to what I prefer and we(luis and I) have been thinking of continuing development of OSICAT if you agree, abandoning iolib-posix on favour of something like osicat-posix-ffi and osicat-windows-ffi - the low-level side of it
I havent' worked actively on Osicat for a while (which is not to say I've abandoned it, but rather that I'm very happy to pass out commit bits...), so my gut feelings are not too well trained here, but... YES! OSICAT: something very much like now, including pathname functions like NATIVE-NAMESTRING and PARSE-NATIVE-NAMESTRING. Nice and lispy, portable across multiple platforms -- either using interfaces that are flexible enough, or staying clear of platform specific hair even when it means missing functionality. WIN32-SYS & POSIX-SYS: low-level APIs. Should be possible to use without thinking about "CFFI", or "Alien", or whatever the FFI layer is. As high-level as possible while remaining an accurate mapping and as low-level as it needs to be for the needs of accuracy and efficiency. My 0.02EUR. Cheers, -- Nikodemus
On 28/07/07, Nikodemus Siivola <nikodemus@random-state.net> wrote:
OSICAT: something very much like now, including pathname functions like NATIVE-NAMESTRING and PARSE-NATIVE-NAMESTRING. Nice and lispy, portable across multiple platforms -- either using interfaces that are flexible enough, or staying clear of platform specific hair even when it means missing functionality.
WIN32-SYS & POSIX-SYS: low-level APIs. Should be possible to use without thinking about "CFFI", or "Alien", or whatever the FFI layer is. As high-level as possible while remaining an accurate mapping and as low-level as it needs to be for the needs of accuracy and efficiency.
Here's a first shot at that. Includes texinfo documentation, tests and a few new functions mostly stolen from cl-fad: git clone http://common-lisp.net/~loliveira/soc07/osicat.git (Converted the CVS history to Git.) The posix module works, compiles and passes all tests on windows but I haven't tested the high-level bits yet or thought about how to handle the missing functionality there at all. I don't have any immediate plans to work much more on it at least until August 20th except for perhaps updating the Osicat webpage if/when you (Nikodemus and the others) feel this branch is OK. -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/
On 06/08/07, Luís Oliveira <luismbo@gmail.com> wrote:
The posix module works, compiles and passes all tests on windows but I haven't tested the high-level bits yet or thought about how to handle the missing functionality there at all.
That sentence is probably a bit misleading. All tests -- both the ones for the posix module and Osicat's high-level interface -- pass on Linux and Darwin. -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/
Erik, please add Luis Oliviera to the Osicat group. On 8/6/07, Luís Oliveira <luismbo@gmail.com> wrote:
Here's a first shot at that. Includes texinfo documentation, tests and a few new functions mostly stolen from cl-fad:
git clone http://common-lisp.net/~loliveira/soc07/osicat.git
(Converted the CVS history to Git.)
The posix module works, compiles and passes all tests on windows but I haven't tested the high-level bits yet or thought about how to handle the missing functionality there at all.
I don't have any immediate plans to work much more on it at least until August 20th except for perhaps updating the Osicat webpage if/when you (Nikodemus and the others) feel this branch is OK.
Haven't looked at it yet, and won't have time for a few days at least -- but great! Once the commit bit arrives you can move the repo under Osicat directory. :) Cheers, -- Nikodemus
Done. On 8/6/07, Nikodemus Siivola <nikodemus@random-state.net> wrote:
Erik, please add Luis Oliviera to the Osicat group.
On 8/6/07, Luís Oliveira <luismbo@gmail.com> wrote:
Here's a first shot at that. Includes texinfo documentation, tests and a few new functions mostly stolen from cl-fad:
git clone http://common-lisp.net/~loliveira/soc07/osicat.git
(Converted the CVS history to Git.)
The posix module works, compiles and passes all tests on windows but I haven't tested the high-level bits yet or thought about how to handle the missing functionality there at all.
I don't have any immediate plans to work much more on it at least until August 20th except for perhaps updating the Osicat webpage if/when you (Nikodemus and the others) feel this branch is OK.
Haven't looked at it yet, and won't have time for a few days at least -- but great! Once the commit bit arrives you can move the repo under Osicat directory. :)
Cheers,
-- Nikodemus
_______________________________________________ admin mailing list admin@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/admin
On 8/6/07, Luís Oliveira <luismbo@gmail.com> wrote:
On 28/07/07, Nikodemus Siivola <nikodemus@random-state.net> wrote:
OSICAT: something very much like now, including pathname functions like NATIVE-NAMESTRING and PARSE-NATIVE-NAMESTRING. Nice and lispy, portable across multiple platforms -- either using interfaces that are flexible enough, or staying clear of platform specific hair even when it means missing functionality.
WIN32-SYS & POSIX-SYS: low-level APIs. Should be possible to use without thinking about "CFFI", or "Alien", or whatever the FFI layer is. As high-level as possible while remaining an accurate mapping and as low-level as it needs to be for the needs of accuracy and efficiency.
Here's a first shot at that. Includes texinfo documentation, tests and a few new functions mostly stolen from cl-fad:
git clone http://common-lisp.net/~loliveira/soc07/osicat.git
(Converted the CVS history to Git.)
The posix module works, compiles and passes all tests on windows but I haven't tested the high-level bits yet or thought about how to handle the missing functionality there at all.
I don't have any immediate plans to work much more on it at least until August 20th except for perhaps updating the Osicat webpage if/when you (Nikodemus and the others) feel this branch is OK.
I managed to spend a _very_ short while looking at it. Looking good. The only things I'm unhappy about are grabbing package names like MACH. On one hand, I fully see the motivation for terse (and descriptive!) names, since those packages are liable to contain symbols named directly after the native interfaces available -- so they are going to be mostly used with explicit package prefixes, but... Few options come to mind: (a) Use names like OSIX, OSMACH, OSWIN -- compromise between putting them in the OSICAT "namespace" and using something terse. (b) Prefix the symbol names. I'm not suggesting prefixing them with "OSICAT-", but rather possibly "-" or "_". Ugly, but practically speaking allows the packages to be USE-PACKAGE'd, so having a long package name like OSICAT-POSIX is ok. (c) Reader macro magic. I'm inclined towards (a), but can live with others as well. Other stuff (aka directions to move in): while I'm OK with having OSICAT-POSIX / OSIX on Windows, in the long term the Windows side should probably not build on it. (Ditto if there are things that are more naturally done using the mach interfaces on Darwin.) So things like USER-INFO should be calling GetUserName (or whatever) on Windows, not getpwnam &co. So that everything doesn't become a huge #+foo mess, there should be src/windows.lisp, which implements the nonportable contents of src/osicat.lisp for Windows, etc. Cheers, -- Nikodemus
On 07/08/07, Nikodemus Siivola <nikodemus@random-state.net> wrote:
The only things I'm unhappy about are grabbing package names like MACH. [...] (a) Use names like OSIX, OSMACH, OSWIN -- compromise between putting them in the OSICAT "namespace" and using something terse.
Those names do sound like a good compromise. (And I think Corman already uses WIN.) What do you think Stelian? Also, OSIMACH and OSIWIN? Or is that one character too many?
Other stuff (aka directions to move in): while I'm OK with having OSICAT-POSIX / OSIX on Windows, in the long term the Windows side should probably not build on it.
Sure.
So that everything doesn't become a huge #+foo mess, there should be src/windows.lisp, which implements the nonportable contents of src/osicat.lisp for Windows, etc.
Agreed. That approach worked really well for CFFI. -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/
On Tue, Aug 07, 2007 at 04:05:39PM +0100, Luis Oliveira wrote:
On 07/08/07, Nikodemus Siivola <nikodemus@random-state.net> wrote:
The only things I'm unhappy about are grabbing package names like MACH. [...] (a) Use names like OSIX, OSMACH, OSWIN -- compromise between putting them in the OSICAT "namespace" and using something terse.
Those names do sound like a good compromise. (And I think Corman already uses WIN.) What do you think Stelian? Also, OSIMACH and OSIWIN? Or is that one character too many?
I think OSWIN and OSIX(OSNIX ?) are ok(maybe nicknamed OW and OX :) ) about Mach, I don't think it should have a package on its own, especially since the only Mach system used around is Darwin/OSX which is almost identical to a FreeBSD. perhaps one day a brave coder will want to port everything on GNU/Hurd and we'll have OSHURD :D
Other stuff (aka directions to move in): while I'm OK with having OSICAT-POSIX / OSIX on Windows, in the long term the Windows side should probably not build on it.
Sure.
I agree too -- (sign :name "Stelian Ionescu" :aka "fe[nl]ix" :quote "Quidquid latine dictum sit, altum videtur.")
participants (4)
-
Erik Enge -
Luís Oliveira -
Nikodemus Siivola -
Stelian Ionescu