Thanks for all your links. I'm learning a lot.
You may find Tamas' experiences with various Lisp implementations useful:
http://tkpapp.blogspot.com/2010/05/upgraded-array-element-types-and-pinned.h...
This is indeed interesting, but the idea of keeping track of all these implementations is very daunting. That's CFFI's job ;)
Some Lisp implementations have an alternative to pinning arrays, which is instead to allocate the array in memory that the GC promises never to move ("static arrays"). The following library wraps that functionality:
That is especially interesting; I wish that I had know about it. After looking at SBCL's source, It appears that SBCL does real pinning, not just suspending GC (at least for x86, the only platform where you worry about threads).
My sense is that this is the right way to go: it's safer (no calling malloc() and free()), and it's more scalable in parallel (no need to interrupt or pause the GC).
Here's my argument why copying to malloc'ed memory is the most sensible compromise, after messing with this issue for a few years (and passing Lisp arrays directly):
* it uses the hard work that the authors of CFFI have done to understand various Lisp implementations. CFFI has a broad user base and is here to stay. Things like gitorious.org/iolib/static-vectors/ and FFA are impressive, but they depend on one or two dedicated people digging through many compilers' source code and often inferring unpublished internals.
* It's what the Lisp compiler writers expect you to do, by providing an FFI and malloc.
* In an age of 64 bit machines, insufficient address space for malloc should be much less of a problem. I just allocated and freed 1 billion bytes on my laptop in both CCL and SBCL.
* The penalty of copying isn't too bad. Copying is O(N), and the things you do in foreign memory are usually O(N^a) or AxO(N) for A>>1. For instance, matrix inversion is N^(3/2) for an N element array. The exception might be some sort of very simple IO with no processing, but why is that being done in FFI-land?
* If a implementation is slow to copy into foreign memory, then it will be slow for everything else, and the copying will be the least of one's worries.
-John