Hi,

David Lichteblau wrote: [I don't know if David joined cffi-devel, maybe people did not get his reply?]

But my impression was that CFFI, in contrast to UFFI, is willing to implement features even if not every Lisp supports them.

I had that impression too :)

So let's analyse further.

I'm going to argue for an -- if at all -- optional cffi:feature:shareable/vector. Don't provide a poor or problematic version!

CFFI would need to document that this API is *not* available everywhere, because the implementation cost of mimicking it is way to high (see below). Instead, code that wishes to benefit from this feature must either tell it's restricted to implementation XYZ (lousy), or use the #+/-cffi:feature to provide an *alternate*, portable implementation.

I say lousy, because as SW complexity grows, people will not remember all the places the pinnable vector is used, which will make later porting a challenge. Seems similar to the trouble with pointer aliasing in C.

If alternative code is provided, why not just avoid #+/- and use a single code? I believe it's fast enough for plenty of uses.

However, I don't think that the concept of WITH-POINTER-TO-VECTOR-DATA is entirely bogus just because some backends use a bad implementation.

Indeed, I advocate explicitly *not* providing a bogus or slow implementation. If some Lisps needs WITHOUT-GC to match it, better not provide the macro with them!

• The commercial Lisps: Don't know, but they'll implement it if their paying customers start asking for it. ;-)

I'm still convinced that the benefit is too low to add that, commercial or not. It's presumably hairy to add as an afterthought. There are many many thing that would provide more value to Lispers.

As an example, if foreign-memory where useable as Lisp arrays (and therefore as sequences), this would render many intended uses of with-pointer-to-vector-data superfluous. It leads to more Lisp-level code, instead of FFI stuff. And I believe it's easier to implement, and provides more benefit, in more situations.

interface would have to be changed so that [...] is guaranteed to work for any vector.

Indeed. That's a excellent requirement. Prefer consistency over restrictions. Generally, programmers should test "optimized" software with displaced vectors instead and witness the crashes :-( Not every array is simple...

BTW, I don't know whether it's appropriate to worry too much about efficiency in cl+ssl now. It uses Flexi-streams, of which Edi Weitz says "FLEXI-STREAMS is currently not optimized towards performance". It uses GRAY streams, which is the first thing I'd look to avoid in a quest for hopefully fast i/o. An interface built upon READ-CHAR cannot be fast. You need block operations. Simple streams and the like can get at it, not GRAY. This is a sad situation of I/O in Lisp, indeed.

Here's a sketch of my argumentation against providing with-pointer-to-vector-data for those Lisps that can't pin. Please excuse my lack of time for further development.

Consider general availabibility of such a macro. People may use it for 1MB large vectors. Consider calling the FFI GUI event loop inside its scope. This will cause callbacks to Lisp. Clearly, WITHOUT-GC would be an application killer (not killer application).

Now consider my :in/:out suggestion. I argue it's still not enough as follows:

(elided). [partial :in requires :out+in, so as not to kill already copied bytes, thus is a waste. Enter :start :end keywords for more precision about the touched area.]

Even if :in/:out plus :start + :end indices were added, it would not be enough in some cases. And there we notice that we've come extremely close to providing such precise information about what bytes move where, that we could, instead explicitly call memory-vector-copy operations. Why not do it right away?

Regards, Jorg Hohle