Hi, [I'm not member of cl-ssl@ and cross-posting is hairy, so I'd prefer this to remain in cffi-devel. I CC'ed to David Lichteblau, author of cl+ssl for comments.]
CL+SSL's using that macro caused me to think as follows:
CFFI-SYS says: ;;;# Shareable Vectors ;;; This interface is very experimental. WITH-POINTER-TO-VECTOR-DATA ;;; should be defined to perform a copy-in/copy-out if the Lisp ;;; implementation can't do this.
cffi-sys::with-pointer-to-vector-data, as is, is highly problematic. Trying to get the base address of a Lisp vector in memory is unportable and subject to subtle errors with a moving GC. I'll try and suggest a better API below.
On CMUCL, it's implementend using sys:without-gcing, which is sign enough of a problem.
This remembers me, 20 years ago, of similar bad use of SI::DISABLE-XYZ (I forgot the name) on Symbolics machines, and on the Amiga computer, where programmers were repeatedly told not to make wrong (ab)use of Forbid() (disable multitasking) or Disable() (disable interrupts).
I could understand it if it were restricted to CFFI internal use, for a couple of highly optimized vector copying operations. Everything else should be a big NO-NO. It's use is worse than the often repeated "EVAL is EVIL".
Sadly, its mere existance encourages some use. E.g. CL+SDL uses it :-(
This is a very bad idea. Let me explain. Cl+ssl:stream-read-byte (in stream.lisp) uses it for ssl-read and installs a callback handler. UNIX signaling seems alo involved, as the error codes and loop structure indicate.
Now consider what happens when a user has installed a SERVE-FD-EVENT handler (e.g. running SLIME). While cl+ssl is looping and waiting for ssl-read to complete, arbitrary Lisp code can be called by the signal handlers and the event dispatcher. All of this within the context of sys:without-gcing. This is calling for trouble. And I did not yet mention other treads getting to run.
Back on the Amiga computer, Forbid/Disable() were accepted in a few cases, and programmers were told to quickly exit the protected section. People used it mostly from assembly, but also C. OS calls within this section almost always are a bad idea, since it might either break out of Forbid() and reenable multitasking, or hang because important interrupts would not get served. This could break invariants of the OS or the application, which would cause random crashes some random time later.
As noted, I could understand use of gc:witout-gcing and with-pointer-to-data when kept strictly inside CFFI and used locally, e.g. to copy data from foreign to/from a Lisp array.
Any other use is leading to problems, typically hard to debug. I hope I made this clear.
There is IMHO no reason for CL+SSL to use such a function. If it wants objects at a fixed address, it should use foreign-alloc'ed memory.
Try to explain your C/Java/Perl/tcl programmer friend: "I can only use SSL when resorting to sys::without-gc". "Huh? In my language & environment, no such dirty hack is necessary. It's clearly superior".
BTW, cl+ssl:stream-read-sequence uses (replace thing buf :start1 start :end1 (+ start length)) anyway. So there's copying even when sharing. Instead it could copy from the foreign buffer to the user supplied "thing". This raises the question of whether copying to an arbitrary array-element-type is supported by CFFI's emerging memory<->vector block copy API.
Now let's move toward a better design:
From a CFFI perspective, the following comment:
;;; WITH-POINTER-TO-VECTOR-DATA should be defined to perform a ;;; copy-in/copy-out if the Lisp implementation can't do this. IMHO shows an inversed design.
With the interface and recommendation as is, nobody knows which of a copy-in and/or copy-out is needed. The macro would do both, just to be safe. Implementations would suffer a double speed penalty.
The need for copy-in or copy-out must be indicated by the programmer, similarly to :in and :out parameter modes.
I believe the design should be the opposite: an efficient copy-in or copy-out may resort to with-pointer-to-vector-data and possibly to si::without-gcing (is that thread-safe at all?) to quickly copy the vector and do nothing more than that (no callbacks, no signals, etc.).
To return to the CL+SSL example, I suggest to use a foreign-alloc'ed buffer for ssl-read etc., then copy that into a Lisp vector.
This copying could use CFFI's emerging memory block interface and rely on CFFI to be fast.
To implement that block copy, CFFI could use with-pointer-to-vector-data. However I believe it's now superfluous: In implementations were such a function is available at all (e.g. cmucl), the native code compiler can already translate Lisp code to an efficient loop from one array to the other. Safely.
I mean that what w-p-t-v-d does (sys:vector-SAP etc.) can be restricted to a few internal functions within cffi-{cmucl,*}.lisp and need no dangerous general macro wrapper.
I think this is the best one can achieve portably, without resorting to very specialised features like IIRC Allegro's ability to allocate Lisp vectors at non-moving locations. I haven't yet investigated how one could make transparent use of such a feature, even though any Allegro user would find it suboptimal if cl+ssl (or any other library) would not make best use of her/his Lisp implementation. (Oh well, I'll rant about portable libraries another time. :)
On a final note, cffi-cmucl contains
(let ((,ptr-var (sys:vector-sap ,vector)))
This won't work with displaced etc. (non simple) vectors. It's not the common case, but it should be supported, or the restriction documented (not that it matters if w-p-t-v-d is dropped anyway).
I always welcome comments.
Regards, Jörg Höhle