On Sat, Sep 11, 2010 at 19:22, JTK jetmonk@gmail.com wrote:
You might be disappointed (or pleased, perhaps) to see that someone (Tamas K. Papp, in particular) has already done this:
I feel a bit chagrined that I didn't find Papp's AAF before putting a lot of effort into my WAAF.
I've looked at the differences between FFA and my WAAF.
From my admittedly quick reading of the docs, FFA passes Lisp arrays to foreign land rather than copying for SBCL and maybe other Lisps. This involves suspending garbage collection. On the balance, I think that in an era of multithreaded 64 bit Lisps and relatively abundant memory, it is better not to mess with Lisp's internals (like suspending GC), and it is safer to allocate foreign memory and copy.
Allocating foreign memory is what Tamas calls "Approach 1" in this blog post. Rif's "fnv" (foreign numeric vectors) package (to which Tamas links in that post) does exactly this: space for vectors and matrices is only allocated in foreign memory, so you know the GC won't move it around or collect it. If I recall correctly, Tamas chose his Approach 2 because (a) supposedly, some Lisp implementations severely restrict the available heap for malloc() calls, and (b) Tamas was mainly interested in SBCL, on which Approach 2 is faster.
eg, SBCL will pin arrays if you ask it, but I understand that this is done by globally suspending GC, which is not necessarily good when other threads are running.
In fact, I was forced to write this after a multithreaded SBCL application froze mysteriously in foreign code with GC suspended.
You may find Tamas' experiences with various Lisp implementations useful:
http://tkpapp.blogspot.com/2010/05/upgraded-array-element-types-and-pinned.h...
Some Lisp implementations have an alternative to pinning arrays, which is instead to allocate the array in memory that the GC promises never to move ("static arrays"). The following library wraps that functionality:
http://gitorious.org/iolib/static-vectors/
My sense is that this is the right way to go: it's safer (no calling malloc() and free()), and it's more scalable in parallel (no need to interrupt or pause the GC). If you don't allocate your arrays in that way, it's reasonable to ask for payment of a performance tax (either copying into foreign memory, or pausing the GC). Anyway, that's just my opinion; I definitely don't speak for CFFI or any Lisp implementation :-)
mfh