On Sun, Sep 4, 2011 at 3:23 PM, Luís Oliveira luismbo@gmail.com wrote:
On Sun, Sep 4, 2011 at 3:33 AM, Liam Healy lnp@healy.washington.dc.us wrote:
I've written a macro define-structure-conversion to flesh out what I see as the way to go. Eventually, I think this would be best as a couple of key arguments in the defcstruct, similar to FSBV's current :constructor and :deconstructor arguments.
I'm not convinced we need a macro like that. Certainly not one specific to structures. Defining translate-* methods is indeed a bit more verbose, but it's also plain CLOS, which makes it clear and flexible. (E.g., you easily can use inheritance and multiple dispatch when defining CFFI types if you already know CLOS.)
My target audience is people like me, scientists and engineers who want to make a foreign numeric library work and get on with their lives. They might not know what CLOS is, they might not ever have written a defmethod. Why should they? Even people who do know what these are should not have to rewrite boilerplate over and over; that is exactly what macros are designed for: to hide implementation details.
Here's what I have now in FSBV:
(defcstruct (complex :constructor complex :deconstructor (realpart imagpart)) (dat :double :count 2))
You can get a pretty good sense of what the :constructor and :deconstructor args do quickly, and I bet most people could write their own defcstruct and appropriate converter quickly with no trips to the manual. The expansion is quite ugly, but no one cares because no one looks at it, not even me any more. While your methods below are a lot simpler than what FSBV expands this defcstruct to, they still involve implementation detail irrelevant to the user that will involve (for me anyway) many trips to the manual to get right, or perhaps big chunks of cut-and-paste. I'd like to eliminate that. No one would be forced to use the macro, you could still write the defmethods by hand as you propose. Macros eliminate or minimize cut and paste.
I am stuck however on recursive conversion of slots. What I have works for the complex example, but if I built on that with a "real-and-complex" structure and try to convert to foreign, I get an error because the complex slot is already converted by the time it gets in the setf. Even with an existing pointer, as would be obtained from foreign-slot-pointer, the only function I have at my disposal for translating the slot is translate-to-foreign which creates a new foreign struct, it does not write to an existing struct. It seems I need something like a setf function.
(defcstruct (real-and-complex :class real-and-complex) (x :double) (c complex-double-c))
(define-structure-conversion value real-and-complex list (x c) ;; Make foreign ;;(setf x (first value) c (convert-to-foreign (second value) 'complex-double-c)) (setf x (first value) c (second value)) ;; Make CL (list x c))
I see a couple of issues raised by your "real-and-complex" example: (1) recursive translations. (2) differentiating between value/reference semantics for the translation of aggregate types. (3) reusing translation code for both semantics.
Regarding (1), at first sight, it seems to me that if we implement some sort of default translation from structures to CLOS objects, we can handle recursiveness there. User-provided translations are on their own, i.e. they should call convert-foreign-objects themselves. (We can export a way to iterate through the slots of a structure if we don't already.) BTW, it occurs to me that a default cstruct to CL conversion could use opaque objects whose accessors handle the translations (and possibly cache them).
I don't envision making CLOS objects the CL equivalent of foreign structures I need (I assume you mean of CLOS object of metaclass standard-class), but I have no objection to providing some default methods. My inclination would be to make the default CL equivalent a list. That's how it is for fsbv:defcstruct. It is unclear to me how making CLOS objects simplifies the recursive conversion task.
I don't understand your BTW statement.
Regarding (2), I agree that we need a translate-into-foreign-memory function that takes 3 arguments: source value, type, and a target pointer. (IIUC, this would be the setf function you were referring to.) This will also be useful to properly integrate the array type with structures, BTW.
Yes, this is what I meant by a setf function -- right now in FSBV, I have two functions, the :constructor which is analogous to CFFI's translate-from-foreign, and the :deconstructor which sets components of an existing foreign structure from the Lisp, unlike translate-to-foreign which creates a new foreign structure. This may be called by the (setf object) function; the former is called by the #'object function.
Having solved (2) that way, (3) should be straightforward. At some point, there'll always be a pointer to somewhere, so translate-to-foreign can call translate-into-foreign-memory, if applicable.
If CFFI had a translate-into-foreign-memory, then a translate-to-foreign could be an ordinary function: (translate-into-foreign-memory source type (foreign-alloc type)) Wouldn't that do everything it needs to?
(BTW, I get confused by foreign-alloc's :initial-element/contents arguments. Is this calling translate-to-foreign? It seems like it presumes some kind of translation facility. Or does it simply do nothing if type is a foreign structure?)
So, under this proposed scheme, your COMPLEX example would look something like:
(defcstruct (complex :class complex-type) (real :double) (imag :double))
(defmethod translate-into-foreign-memory ((value complex) (type complex-type) p) (with-foreign-slots ((real imag) p 'complex) (setf real (realpart value) imag (imagpart value))))
(defmethod translate-from-foreign (p (type complex-type)) (with-foreign-slots ((real imag) p 'complex) (complex real imag)))
This would be adequate to implement translation both when embedding the structure and when passing as a value. (We could also implement a TRANSLATE-TO-FOREIGN if we wanted to do automatic translation when passing by reference as well. I guess we probably would want want.)
I like that this transfers the responsibility of allocating a (temporary?) structure to libffi-specific code.
I don't understand why you call out libffi specifically. Certainly libffi isn't the only library that might want to refer to a foreign structure. And yes, the responsibility is on the user of the library to allocating it; the way I do it in FSBV is with-foreign-objects instead of something like the translate-to-foreign, free-translated-object pair because the lifetime is well-defined within one function call. Providing both is the best, however.
The REAL-AND-COMPLEX example would in turn look like this:
(defcstruct (real-and-complex :class real-and-complex-type) (x :double) (c complex))
(defmethod translate-into-foreign-memory (value (type real-and-complex-type) p) (setf (foreign-slot-value p 'real-and-complex 'x) (first value)) (convert-into-foreign-memory (second value) 'complex (foreign-slot-pointer p 'real-and-complex 'c)))
(defmethod translate-from-foreign (p (type real-and-complex-type)) (with-foreign-slots ((x c) p 'real-and-complex) (list x c)))
Does this look good to you plumbing-wise? Regarding the external API, do you still think we need some sort of convenience macro around this? (I guess the real-and-complex example would benefit from a WITH-FOREIGN-SLOT-PLACES along the lines of WITH-FOREIGN-PLACE described in https://bugs.launchpad.net/cffi/+bug/688532.)
Getting there; yes I definitely want the macro to hide implementation details like the calls to foreign-slot-pointer, with-foreign-slots parts, and the recursive *-into-foreign-memory, so that I can just give a pair:
(first value) (second value) (list x c)
or whatever would involve zero CFFI function/macro calls, and let the macro do the work for me, as with FSBV's :constructor and :deconstructor (which may be oversimplifying).
We'll want matching converto-into-foreign-memory and expand-into-foreign-memory functions for this new translate-into-foreign-memory.
I don't know what expand-into-foreign-memory is.
Cheers,
-- Luís Oliveira http://r42.eu/~luis/
Thanks,
Liam