You might also want to take a look at foreign-array in Grid Structured Data (GSD), which uses iolib and is built around Tamas's "AFFI" (affine). GSLL uses it extensively, and we are working on porting other libraries. Formerly, I had built foreign array access on ffa, but iolib is more portable. Also, I've come to think that seeing the same array from both the Lisp and foreign side is not all that important; with grid:gref you reference the foreign element just like aref. However I can imagine if you use CL array functions and foreign code together, this would be handy.
http://repo.or.cz/w/gsd.git http://common-lisp.net/project/gsll/
The gsll-devel mailing list is a good place to bring up issues as this is under ongoing development.
Liam
On Sat, Sep 11, 2010 at 11:49 PM, JTK jetmonk@gmail.com wrote:
Thanks for all your links. I'm learning a lot.
You may find Tamas' experiences with various Lisp implementations useful:
http://tkpapp.blogspot.com/2010/05/upgraded-array-element-types-and-pinned.h...
This is indeed interesting, but the idea of keeping track of all these implementations is very daunting. That's CFFI's job ;)
Some Lisp implementations have an alternative to pinning arrays, which is instead to allocate the array in memory that the GC promises never to move ("static arrays"). The following library wraps that functionality:
That is especially interesting; I wish that I had know about it. After looking at SBCL's source, It appears that SBCL does real pinning, not just suspending GC (at least for x86, the only platform where you worry about threads).
My sense is that this is the right way to go: it's safer (no calling malloc() and free()), and it's more scalable in parallel (no need to interrupt or pause the GC).
Here's my argument why copying to malloc'ed memory is the most sensible compromise, after messing with this issue for a few years (and passing Lisp arrays directly):
- it uses the hard work that the authors of CFFI have done to understand various Lisp
implementations. CFFI has a broad user base and is here to stay. Things like gitorious.org/iolib/static-vectors/ and FFA are impressive, but they depend on one or two dedicated people digging through many compilers' source code and often inferring unpublished internals.
It's what the Lisp compiler writers expect you to do, by providing an FFI and malloc.
In an age of 64 bit machines, insufficient address space for malloc should be much less
of a problem. I just allocated and freed 1 billion bytes on my laptop in both CCL and SBCL.
- The penalty of copying isn't too bad. Copying is O(N), and the things you do in foreign memory are usually
O(N^a) or AxO(N) for A>>1. For instance, matrix inversion is N^(3/2) for an N element array. The exception might be some sort of very simple IO with no processing, but why is that being done in FFI-land?
- If a implementation is slow to copy into foreign memory, then it will be slow
for everything else, and the copying will be the least of one's worries.
-John
cffi-devel mailing list cffi-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel