You might be disappointed (or pleased, perhaps) to see that someone (Tamas K. Papp, in particular) has already done this:

http://tkpapp.blogspot.com/2007/12/announcement-ffa.html

I feel a bit chagrined that I didn't find Papp's AAF before putting a lot of effort into my WAAF.

I've looked at the differences between FFA and my WAAF.

From my admittedly quick reading of the docs, FFA passes Lisp arrays to foreign land rather than copying for SBCL and maybe other Lisps. This involves suspending garbage collection. On the balance, I think that in an era of multithreaded 64 bit Lisps and relatively abundant memory, it is better not to mess with Lisp's internals (like suspending GC), and it is safer to allocate foreign memory and copy. eg, SBCL will pin arrays if you ask it, but I understand that this is done by globally suspending GC, which is not necessarily good when other threads are running.

In fact, I was forced to write this after a multithreaded SBCL application froze mysteriously in foreign code with GC suspended.

FFA apparently uses its own knowledge about individual Lisp implementations to decide whether to copy or pass lisp array addresses directly to foreign code. WAAF, on the other hand, knows only about CFFI, and infers all its knowledge from information exported by CFFI (like pointer sizes, if you want a pointer array).

Most of the effort in WAAF is in the set of fast copying function macrology that minimizes the cost of copying to and from foreign memory.

At any rate, I think it would be very useful to have some form of this functionality in CFFI, or in a visible user contrib section of it, because passing arrays to foreign code is a common but somewhat arduous and tricky task.

John

On Sep 11, 2010, at 2:45 PM, Mark Hoemmen wrote:

On Sat, Sep 11, 2010 at 18:38, JTK jetmonk@gmail.com wrote:

...

I'm using cffi, and I need to perform the common task of passing Lisp arrays to foreign code.

This task comes up a lot in scientific code, matrix math, image processing, etc.

Normally, it involves a lot of boilerplate to allocate a pointer, copy data Lisp to foreign memory, copy the data back, and deallocate the memory. Getting the copying to run fast can be tricky.

Hence I wrote a package to automate it.

You might be disappointed (or pleased, perhaps) to see that someone (Tamas K. Papp, in particular) has already done this:

http://tkpapp.blogspot.com/2007/12/announcement-ffa.html

mfh

Thanks for all your links. I'm learning a lot.

You may find Tamas' experiences with various Lisp implementations useful:

http://tkpapp.blogspot.com/2010/05/upgraded-array-element-types-and-pinned.h...

This is indeed interesting, but the idea of keeping track of all these implementations is very daunting. That's CFFI's job ;)

Some Lisp implementations have an alternative to pinning arrays, which is instead to allocate the array in memory that the GC promises never to move ("static arrays"). The following library wraps that functionality:

http://gitorious.org/iolib/static-vectors/

That is especially interesting; I wish that I had know about it. After looking at SBCL's source, It appears that SBCL does real pinning, not just suspending GC (at least for x86, the only platform where you worry about threads).

My sense is that this is the right way to go: it's safer (no calling malloc() and free()), and it's more scalable in parallel (no need to interrupt or pause the GC).

Here's my argument why copying to malloc'ed memory is the most sensible compromise, after messing with this issue for a few years (and passing Lisp arrays directly):

* it uses the hard work that the authors of CFFI have done to understand various Lisp implementations. CFFI has a broad user base and is here to stay. Things like gitorious.org/iolib/static-vectors/ and FFA are impressive, but they depend on one or two dedicated people digging through many compilers' source code and often inferring unpublished internals.

* It's what the Lisp compiler writers expect you to do, by providing an FFI and malloc.

* In an age of 64 bit machines, insufficient address space for malloc should be much less of a problem. I just allocated and freed 1 billion bytes on my laptop in both CCL and SBCL.

* The penalty of copying isn't too bad. Copying is O(N), and the things you do in foreign memory are usually O(N^a) or AxO(N) for A>>1. For instance, matrix inversion is N^(3/2) for an N element array. The exception might be some sort of very simple IO with no processing, but why is that being done in FFI-land?

* If a implementation is slow to copy into foreign memory, then it will be slow for everything else, and the copying will be the least of one's worries.

-John

On Sat, Sep 11, 2010 at 21:49, JTK jetmonk@gmail.com wrote:

• If a implementation is slow to copy into foreign memory, then it will be slow

for everything else, and the copying will be the least of one's worries.

As long as people aren't tempted to call memcpy() on the Lisp array... ;-)

mfh

Actually, there is documentation, it's linked in CFFI's main page:

http://common-lisp.net/gitweb?p=projects/cffi/cffi.git;a=blob_plain;f=doc/sh...

http://common-lisp.net/gitweb?p=projects/cffi/cffi.git;a=blob_plain;f=doc/shareable-vectors.txtYou seem to need to use CFFI:MAKE-SHAREABLE-BYTE-VECTOR to create the array.

Em 12 de setembro de 2010 20:39, Gustavo gugamilare@gmail.com escreveu:

Hello,

What about CFFI:WITH-POINTER-TO-VECTOR-DATA? Or did you just miss it? Documentation is missing, but it pines the array within its body (which means the array won't be moved, but garbage collector might still run) and grabs the foreign pointer of the array, at least on SBCL.

Regards, Gustavo.

2010/9/11 JTK jetmonk@gmail.com

...

Hello,

I'm using cffi, and I need to perform the common task of passing Lisp arrays to foreign code.

This task comes up a lot in scientific code, matrix math, image processing, etc.

Normally, it involves a lot of boilerplate to allocate a pointer, copy data Lisp to foreign memory, copy the data back, and deallocate the memory. Getting the copying to run fast can be tricky.

Hence I wrote a package to automate it. It maintains a set of specialized copying functions to copy from array of Lisp type X to foreign memory of type Y. On SBCL, copying doesn't cons and is reasonably fast (~200 MB a second, there and back)

I've put the package here:

http://www.megaupload.com/?d=CX1XFMU6

Sorry for the megaupload, and its imposed delay, but I didn't want to spam the list with a 100K file.

The package includes a MANUAL.txt and a test package.

I'd be happy to have this included in a user contrib section of CFFI, if the maintainers think it would be useful.

Here's an example the main macro does:

(LET ((U (MAKE-ARRAY 100 :ELEMENT-TYPE 'SINGLE-FLOAT :INITIAL-ELEMENT 1.0))) ;; (WITH-ARRAY-AS-FOREIGN-POINTER (U PU :FLOAT ;; LISP-VAR POINTER-VAR CFFI-FOREIGN-TYPE :LISP-TYPE SINGLE-FLOAT ;; promise U is a single-float array :START 2 ;; begin at index 2 of of Lisp array :END 7 ;; last index used is 6 :COPY-TO-FOREIGN T ;; put contents of Lisp into foreign memory :COPY-FROM-FOREIGN T) ;; copy back from FFI space to lisp array ;; ;; at this point, PU is a foreign pointer containing data copied ;; from the array starting at element 1 and ending at 6, of type :FLOAT.

    (SOME-FOREIGN-FUNCTION PU))
  ;;
  ;; at end, all foreign memory is deallocated, and U has been copied
  ;; back from foreign space, but the 0th element of U is untouched
  ;; because START was 1, not 0
)

-John _______________________________________________ cffi-devel mailing list cffi-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel

Well, in this case you can use copy CFFI's code for that macro and that function to make your library work with more implementations. Or, wouldn't it be much better if you patch CFFI so that CFFI:MAKE-SHAREABLE-BYTE-VECTOR would be able to create arrays of different element types? You could create functions that transform between Lisp and C types (such as :int <-> '(signed-byte 32)).

Gustavo.

Em 12 de setembro de 2010 21:27, JTK jetmonk@gmail.com escreveu:

On Sep 12, 2010, at 1:57 PM, Gustavo wrote:

Actually, there is documentation, it's linked in CFFI's main page:

http://common-lisp.net/gitweb?p=projects/cffi/cffi.git;a=blob_plain;f=doc/sh...

You seem to need to use CFFI:MAKE-SHAREABLE-BYTE-VECTOR to create the array.

I saw this, but I assumed it was just a proposal, plus the use of shared vectors makes it hard to use with most lisp data, often leading the same copying overhead that would have arisen just malloc'ing.

I've been passing arrays to foreign libraries for a long time. I used to think that the right solution was to pass the Lisp data itself. Sometimes, for huge arrays on a 32 bit machine, it was actually necessary. Now, however, I've concluded that simply copying to malloc is good the vast majority of the time. There might be some truth to the adage that the perfect is the enemy of the good.

At any rate, I've put my WAAF copying package here:

http://www.cliki.net/WAAF-CFFI
http://sites.google.com/site/lithpthtuff/home/waaf-cffi

JK

http://common-lisp.net/gitweb?p=projects/cffi/cffi.git;a=blob_plain;f=doc/shareable-vectors.txt

Em 12 de setembro de 2010 20:39, Gustavo gugamilare@gmail.com escreveu:

...

Hello,

What about CFFI:WITH-POINTER-TO-VECTOR-DATA? Or did you just miss it? Documentation is missing, but it pines the array within its body (which means the array won't be moved, but garbage collector might still run) and grabs the foreign pointer of the array, at least on SBCL.

Regards, Gustavo.

2010/9/11 JTK jetmonk@gmail.com

...

Hello,

I'm using cffi, and I need to perform the common task of passing Lisp arrays to foreign code.

This task comes up a lot in scientific code, matrix math, image processing, etc.

Normally, it involves a lot of boilerplate to allocate a pointer, copy data Lisp to foreign memory, copy the data back, and deallocate the memory. Getting the copying to run fast can be tricky.

Hence I wrote a package to automate it. It maintains a set of specialized copying functions to copy from array of Lisp type X to foreign memory of type Y. On SBCL, copying doesn't cons and is reasonably fast (~200 MB a second, there and back)

I've put the package here:

http://www.megaupload.com/?d=CX1XFMU6

Sorry for the megaupload, and its imposed delay, but I didn't want to spam the list with a 100K file.

The package includes a MANUAL.txt and a test package.

I'd be happy to have this included in a user contrib section of CFFI, if the maintainers think it would be useful.

Here's an example the main macro does:

(LET ((U (MAKE-ARRAY 100 :ELEMENT-TYPE 'SINGLE-FLOAT :INITIAL-ELEMENT 1.0))) ;; (WITH-ARRAY-AS-FOREIGN-POINTER (U PU :FLOAT ;; LISP-VAR POINTER-VAR CFFI-FOREIGN-TYPE :LISP-TYPE SINGLE-FLOAT ;; promise U is a single-float array :START 2 ;; begin at index 2 of of Lisp array :END 7 ;; last index used is 6 :COPY-TO-FOREIGN T ;; put contents of Lisp into foreign memory :COPY-FROM-FOREIGN T) ;; copy back from FFI space to lisp array ;; ;; at this point, PU is a foreign pointer containing data copied ;; from the array starting at element 1 and ending at 6, of type :FLOAT.

    (SOME-FOREIGN-FUNCTION PU))
  ;;
  ;; at end, all foreign memory is deallocated, and U has been copied
  ;; back from foreign space, but the 0th element of U is untouched
  ;; because START was 1, not 0
)

-John _______________________________________________ cffi-devel mailing list cffi-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel

Em 13 de setembro de 2010 02:05, JTK jetmonk@gmail.com escreveu:

2010/9/12 Gustavo gugamilare@gmail.com

Well, in this case you can use copy CFFI's code for that macro and that

...

function to make your library work with more implementations.

Maybe I'm confused about what you mean, but WAAF already works with all implementations that work with CFFI. It's just that it uses foreign alloc and copying rather than tricks to make pointers to native Lisp arrays.

I meant to avoid copying.

...

Or, wouldn't it be much better if you patch CFFI so that CFFI:MAKE-SHAREABLE-BYTE-VECTOR would be able to create arrays of different element types?

I guess one could do this, but wouldn't this mean that only shareable arrays could be passed to foreign code? Not any lisp array? In return, for some implementations, you would avoid copying to and from malloc'ed memory.

It looks like some implementations need you to say that the array should be created in static memory so it doesn't get relocated. In your library, SBCL is somehow privileged, which is not bad but neither ideal. CFFI seems to have code for using pointers to native Lisp arrays in various implementations. I think the right answer would be to extend the specification of the proposal. CFFI should detect the cases where the implementation can't provide a pointer to the array so it copies the array instead.

Just a though.

...

You could create functions that transform between Lisp and C types (such as :int <-> '(signed-byte 32)).

Yes, this is what my code does. It constructs a library (a hash table, internally) of optimized lisp functions to do the copying, one for each LISP<-->CFFI type pair, plus a generic set at a higher safety level where the Lisp type is just T. But doesn't inline or macroexpand the copying because I don't think this buys much because most of the cycles are likely spent iterating, not funcallling.

There are a various approaches to take when passing arrays. What is important from my own perspective, as a user, is to have them publicized and accessible, so I don't end up reinventing them. Hence I put my solution on cliki's FFI page. Very likely, at some point cffi will have functionality that renders it redundant.