On Sat, Aug 27, 2011 at 1:08 PM, Luís Oliveira luismbo@gmail.com wrote:

On Sat, Aug 27, 2011 at 5:05 PM, Liam Healy lnp@healy.washington.dc.us wrote:

...

My thought now is that I should define methods for translate-*-foreign

and

...

free-translated-object dispatching on the foreign-struct-type class.

That makes sense. The type object should have all the necessary info you need to construct whatever objects libffi requires. (I'm not familiar with the libffi API.) I.e., you shouldn't need to define specific translation methods for each struct type definition.

I think I do need specific translations; see below. I think a default would be appropriate though. libffi needs a pointer to a foreign structure.

...

The body will call the same method with the structure name, and

applications can

...

define methods eql-specialized on the name.

I don't think that's necessary. DEFCSTRUCT takes an undocumented class argument that allows for specialization based on class rather than using EQL-specializers. I'm guessing you're thinking about stuff like converting CLOS objects into C structs. That is something that hasn't been properly explored in CFFI. See the undocumented DEFINE-C-STRUCT-WRAPPER for a half-baked go at that problem.

I didn't know about the :class argument defcstruct, that seems right. I could define a default method for foreign-struct-type. I'm not thinking about CLOS objects, though that's interesting. However, I don't view foreign structs as always mapping into Lisp structures; my canonical example is complex numbers (see FSBV readme http://repo.or.cz/w/fsbv.git) because the original driver for FSBV was the use of complex numbers in GSLL. I convert Lisp complex numbers to and from the foreign structure consisting of two doubles or floats.

Does this sound like it will work correctly with CFFI as it exists now, so that I can then introduce

the

libffi interface for the calls to foreign functions?

What kind of translation do you need these methods to do on

FOREIGN-STRUCT-TYPE?

They should take the lisp object, allocate the foreign structure, then

define the slots correctly (recursively if necessary), and return the pointer. For the return pointer, they should read the contents and generate the lisp equivalent. I suppose the calls to libffi functions should invoke translate-*-foreign just like it now invokes the functions on the properties fsbv::foreign-object-components and fsbv::setf-foreign-object-components.

Cheers,

-- Luís Oliveira http://r42.eu/~luis/

Does this make sense?

Thanks,

Liam

On Sat, Aug 27, 2011 at 4:10 PM, Luís Oliveira luismbo@gmail.com wrote:

On Sat, Aug 27, 2011 at 7:39 PM, Liam Healy lhealy@common-lisp.net wrote:

...

...

What kind of translation do you need these methods to do on FOREIGN-STRUCT-TYPE?

They should take the lisp object, allocate the foreign structure, then define the slots correctly (recursively if necessary), and return the pointer. For the return pointer, they should read the contents and

generate

...

the lisp equivalent. I suppose the calls to libffi functions should

invoke

...

translate-*-foreign just like it now invokes the functions on the

properties

...

fsbv::foreign-object-components and fsbv::setf-foreign-object-components.

That seems to fit the translation-*-foreign pretty well. Moreover, it seems like there should be no default conversion, i.e., you could pass structures by value using the same code you currently use for passing structures by reference:

(with-foreign-object (foo 'some-struct-type) (bar foo))

The only difference is that BAR's first argument would be defined to be a structure argument passed by value. It's a bit subtle. Not sure how sensible it would be. An alternative would be represent structure values (as opposed to structure pointers) using some kind of wrapper object. Any thoughts?

So you're suggesting retaining the :constructor and :deconstructor arguments to defcstruct? It seems like a custom translate-*-foreign may be more compatible with existing CFFI, though I guess a compatibility expansion might be convenient.

For a default conversion assuming a structure on the lisp side, I'm not sure how to iterate over the structure. Maybe MOP has something? I do have iterate-foreign-structure. If I assume a list or vector on the lisp side, then I could iterate through it.

If I define a specializer for a particular cstruct, what should the class be? This looks like a type name, not the actual object that's passed as the second argument 'type, so I presume it should be an eql specializer on the name, but I admit I'm getting confused by this. In any case I think the :class option to defcstruct won't always work because it does a defclass, I need something like :existing-class that would just do a find-class.

Does any of this make sense?

Liam

--

Luís Oliveira http://r42.eu/~luis/

On Sat, Aug 27, 2011 at 5:56 PM, Luís Oliveira luismbo@gmail.com wrote:

On Sat, Aug 27, 2011 at 10:11 PM, Liam Healy lnp@healy.washington.dc.us wrote:

...

So you're suggesting retaining the :constructor and :deconstructor

arguments

...

to defcstruct? It seems like a custom translate-*-foreign may be more compatible with existing CFFI, though I guess a compatibility expansion might be convenient.

No, that's not what I meant. Let's take your COMPLEX example. That should make things clearer.

(defcstruct (complex :class complex-type) (real :double) (imag :double))

(defmethod translate-to-foreign ((value complex) (type complex-type)) (let ((p (foreign-alloc 'complex))) (with-foreign-slots ((real imag) p 'complex) (setf real (realpart value) imag (imagpart value))) (values p t)) ; second value is passed to FREE-TRANSLATED-OBJECT

(defmethod free-translated-object (value (p complex-type) freep) (when freep (foreign-free value)))

(defmethod translate-from-foreign (value (type complex-type)) ...)

(defcfun (complex-conjugate "gsl_complex_conjugate") complex (c (:by-value complex)) ; or something like that.

;; we can then call it like this:

(complex-conjugate #c(3.0d0 4.0d0)) #C(3.0 -4.0)

;; what I was proposing is that one could /also/ call it like this:

(with-foreign-object (p 'complex) (with-foreign-slots ((real imag) p 'complex) (setf real 3.0 imag -4.0)) (complex-conjugate p))

In summary, we're always dealing with struct pointers, but the function declaration will determine whether that struct will be passed by value or not. Does that make sense?

Yes. I'm not (yet) concerned about the function declaration or the libffi call, because my confusion was about the conversion structure and how it might apply to structures, even before any foreign functions are called.

My follow-up question was: is this too subtle? Should we make this more explicit? E.g.:

(with-foreign-object (p 'complex) (with-foreign-slots ((real imag) p 'complex) (setf real 3.0 imag -4.0)) (complex-conjugate (struct-pointer-value p)))

Doesn't look too pretty.

No, I don't like this either.

...

For a default conversion assuming a structure on the lisp side, I'm not

sure

...

how to iterate over the structure. Maybe MOP has something? I do have iterate-foreign-structure. If I assume a list or vector on the lisp

side,

...

then I could iterate through it.

If we add such a feature, I guess the Lisp structure (or class) should be defined by defcstruct itself.

Good point.

...

If I define a specializer for a particular cstruct, what should the class be? This looks like a type name, not the actual object that's passed as

the

...

second argument 'type, so I presume it should be an eql specializer on

the

...

name, but I admit I'm getting confused by this. In any case I think the :class option to defcstruct won't always work because it does a defclass,

I

...

need something like :existing-class that would just do a find-class.

Hopefully my example with COMPLEX-TYPE illustrates how that works. The type argument gets an instance of COMPLEX-TYPE, which is the result of (parse-type 'complex). Was it clear?

Yes, thanks. The subclassing of type names was confusing me. The example makes it clear. I think I will define a macro to make the creation of the translate-*-foreign and free-translated-object methods a little bit easier.

Liam

Cheers,

-- Luís Oliveira http://r42.eu/~luis/

On Sun, Aug 28, 2011 at 4:41 AM, Luís Oliveira luismbo@gmail.com wrote:

What kind of macro are you thinking of? In the COMPLEX case, do you want to abstract away the need to allocate (and free) the structure object explicitly?

I've written a macro define-structure-conversion to flesh out what I see as the way to go. Eventually, I think this would be best as a couple of key arguments in the defcstruct, similar to FSBV's current :constructor and :deconstructor arguments.

I am stuck however on recursive conversion of slots. What I have works for the complex example, but if I built on that with a "real-and-complex" structure and try to convert to foreign, I get an error because the complex slot is already converted by the time it gets in the setf. Even with an existing pointer, as would be obtained from foreign-slot-pointer, the only function I have at my disposal for translating the slot is translate-to-foreign which creates a new foreign struct, it does not write to an existing struct. It seems I need something like a setf function.

See the define-structure-conversion definition and test case exampleshttps://github.com/cffi/cffi/blob/4d475077a6c00dd12aa0fc72f8b36b8cb5b8c3d3/src/structures.lisp. Any thoughts on how to proceed?

Liam

-- Luís Oliveira http://r42.eu/~luis/

On Sun, Sep 4, 2011 at 3:23 PM, Luís Oliveira luismbo@gmail.com wrote:

On Sun, Sep 4, 2011 at 3:33 AM, Liam Healy lnp@healy.washington.dc.us wrote:

...

I've written a macro define-structure-conversion to flesh out what I see as the way to go.  Eventually, I think this would be best as a couple of key arguments in the defcstruct, similar to FSBV's current :constructor and :deconstructor arguments.

I'm not convinced we need a macro like that. Certainly not one specific to structures. Defining translate-* methods is indeed a bit more verbose, but it's also plain CLOS, which makes it clear and flexible. (E.g., you easily can use inheritance and multiple dispatch when defining CFFI types if you already know CLOS.)

My target audience is people like me, scientists and engineers who want to make a foreign numeric library work and get on with their lives.  They might not know what CLOS is, they might not ever have written a defmethod.  Why should they?  Even people who do know what these are should not have to rewrite boilerplate over and over; that is exactly what macros are designed for: to hide implementation details.

Here's what I have now in FSBV:

(defcstruct (complex :constructor complex :deconstructor (realpart imagpart))   (dat :double :count 2))

You can get a pretty good sense of what the :constructor and :deconstructor args do quickly, and I bet most people could write their own defcstruct and appropriate converter quickly with no trips to the manual. The expansion is quite ugly, but no one cares because no one looks at it, not even me any more. While your methods below are a lot simpler than what FSBV expands this defcstruct to, they still involve implementation detail irrelevant to the user that will involve (for me anyway) many trips to the manual to get right, or perhaps big chunks of cut-and-paste. I'd like to eliminate that. No one would be forced to use the macro, you could still write the defmethods by hand as you propose. Macros eliminate or minimize cut and paste.

I am stuck however on recursive conversion of slots.  What I have works for the complex example, but if I built on that with a  "real-and-complex" structure and try to convert to foreign, I get an error because the complex slot is already converted by the time it gets in the setf.  Even with an existing pointer, as would be obtained from foreign-slot-pointer, the only function I have at my disposal for translating the slot is translate-to-foreign which creates a new foreign struct, it does not write to an existing struct.  It seems I need something like a setf function.

(defcstruct (real-and-complex :class real-and-complex)     (x :double)     (c complex-double-c))

(define-structure-conversion value real-and-complex list (x c)      ;; Make foreign      ;;(setf x (first value) c (convert-to-foreign (second value) 'complex-double-c))      (setf x (first value) c (second value))      ;; Make CL      (list x c))

I see a couple of issues raised by your "real-and-complex" example:  (1) recursive translations.  (2) differentiating between value/reference semantics for the translation of aggregate types.  (3) reusing translation code for both semantics.

Regarding (1), at first sight, it seems to me that if we implement some sort of default translation from structures to CLOS objects, we can handle recursiveness there. User-provided translations are on their own, i.e. they should call convert-foreign-objects themselves. (We can export a way to iterate through the slots of a structure if we don't already.) BTW, it occurs to me that a default cstruct to CL conversion could use opaque objects whose accessors handle the translations (and possibly cache them).

I don't envision making CLOS objects the CL equivalent of foreign structures I need (I assume you mean of CLOS object of metaclass standard-class), but I have no objection to providing some default methods. My inclination would be to make the default CL equivalent a list. That's how it is for fsbv:defcstruct. It is unclear to me how making CLOS objects simplifies the recursive conversion task.

I don't understand your BTW statement.

Regarding (2), I agree that we need a translate-into-foreign-memory function that takes 3 arguments: source value, type, and a target pointer. (IIUC, this would be the setf function you were referring to.) This will also be useful to properly integrate the array type with structures, BTW.

Yes, this is what I meant by a setf function -- right now in FSBV, I have two functions, the :constructor which is analogous to CFFI's translate-from-foreign, and the :deconstructor which sets components of an existing foreign structure from the Lisp, unlike translate-to-foreign which creates a new foreign structure. This may be called by the (setf object) function; the former is called by the #'object function.

Having solved (2) that way, (3) should be straightforward. At some point, there'll always be a pointer to somewhere, so translate-to-foreign can call translate-into-foreign-memory, if applicable.

If CFFI had a translate-into-foreign-memory, then a translate-to-foreign could be an ordinary function: (translate-into-foreign-memory source type (foreign-alloc type)) Wouldn't that do everything it needs to?

(BTW, I get confused by foreign-alloc's :initial-element/contents arguments. Is this calling translate-to-foreign? It seems like it presumes some kind of translation facility. Or does it simply do nothing if type is a foreign structure?)

So, under this proposed scheme, your COMPLEX example would look something like:

(defcstruct (complex :class complex-type)  (real :double)  (imag :double))

(defmethod translate-into-foreign-memory ((value complex) (type complex-type) p)  (with-foreign-slots ((real imag) p 'complex)    (setf real (realpart value)          imag (imagpart value))))

(defmethod translate-from-foreign (p (type complex-type))  (with-foreign-slots ((real imag) p 'complex)    (complex real imag)))

This would be adequate to implement translation both when embedding the structure and when passing as a value. (We could also implement a TRANSLATE-TO-FOREIGN if we wanted to do automatic translation when passing by reference as well. I guess we probably would want want.)

I like that this transfers the responsibility of allocating a (temporary?) structure to libffi-specific code.

I don't understand why you call out libffi specifically. Certainly libffi isn't the only library that might want to refer to a foreign structure. And yes, the responsibility is on the user of the library to allocating it; the way I do it in FSBV is with-foreign-objects instead of something like the translate-to-foreign, free-translated-object pair because the lifetime is well-defined within one function call. Providing both is the best, however.

The REAL-AND-COMPLEX example would in turn look like this:

(defcstruct (real-and-complex :class real-and-complex-type)  (x :double)  (c complex))

(defmethod translate-into-foreign-memory (value (type real-and-complex-type) p)  (setf (foreign-slot-value p 'real-and-complex 'x) (first value))  (convert-into-foreign-memory (second value)                               'complex                               (foreign-slot-pointer p 'real-and-complex 'c)))

(defmethod translate-from-foreign (p (type real-and-complex-type))  (with-foreign-slots ((x c) p 'real-and-complex)    (list x c)))

Does this look good to you plumbing-wise? Regarding the external API, do you still think we need some sort of convenience macro around this? (I guess the real-and-complex example would benefit from a WITH-FOREIGN-SLOT-PLACES along the lines of WITH-FOREIGN-PLACE described in https://bugs.launchpad.net/cffi/+bug/688532.)

Getting there; yes I definitely want the macro to hide implementation details like the calls to foreign-slot-pointer, with-foreign-slots parts, and the recursive *-into-foreign-memory, so that I can just give a pair:

(first value) (second value) (list x c)

or whatever would involve zero CFFI function/macro calls, and let the macro do the work for me, as with FSBV's :constructor and :deconstructor (which may be oversimplifying).

We'll want matching converto-into-foreign-memory and expand-into-foreign-memory functions for this new translate-into-foreign-memory.

I don't know what expand-into-foreign-memory is.

Cheers,

-- Luís Oliveira http://r42.eu/~luis/

Thanks,

Liam

On Mon, Sep 5, 2011 at 10:16 AM, Luís Oliveira luismbo@gmail.com wrote:

On Mon, Sep 5, 2011 at 4:43 AM, Liam Healy lnp@healy.washington.dc.us wrote:

...

(defcstruct (complex :constructor complex :deconstructor (realpart imagpart))   (dat :double :count 2))

You can get a pretty good sense of what the :constructor and :deconstructor args do quickly, and I bet most people could write their own defcstruct and appropriate converter quickly with no trips to the manual.

[...]

...

No one would be forced to use the macro, you could still write the defmethods by hand as you propose.  Macros eliminate or minimize cut and paste.

My broader point, I guess, is that one must be careful not to mistake concision for abstraction and that the latter is more important. Anyway, I understand your point. Let's get the plumbing right first, and we'll get back to the macro sugar later.

Agreed.

...

I don't envision making CLOS objects the CL equivalent of foreign structures I need (I assume you mean of CLOS object of metaclass standard-class), but I have no objection to providing some default methods.   My inclination would be to make the default CL equivalent a list.   That's how it is for fsbv:defcstruct.   It is unclear to me how making CLOS objects simplifies the recursive conversion task.

I don't think using CLOS objects has any impact on handling recursiveness. I started thinking about conversion to CLOS and mixed things up, sorry.

OK

...

I don't understand your BTW statement.

I was just mentioning that a possible default translation could be to make an opaque object with accessors that handle translations on demand. Not a very important issue at this point. We can discuss it later.

OK

...

If CFFI had a translate-into-foreign-memory, then a translate-to-foreign could be an ordinary function: (translate-into-foreign-memory source type (foreign-alloc type)) Wouldn't that do everything it needs to?

Pretty much, yeah.

...

(BTW, I get confused by foreign-alloc's :initial-element/contents arguments.  Is this calling translate-to-foreign?  It seems like it presumes some kind of translation facility.  Or does it simply do nothing if type is a foreign structure?)

initial-element and contents will in fact go through translate-to-foreign, IIRC. Is that an issue?

On the contrary, it's welcome. Can we add these to with-foreign-object(s) too? Notice (http://repo.or.cz/w/fsbv.git) fsbv:with-foreign-objects has bindings ::= {(var type &optional initial-value)}* This is another macro sugar issue, so no problem deferring, but it's be nice to have ultimately.

The point of confusion for me is that when writing the translator, I'm calling a function which has an option to use the translator. Clearly, I need to avoid using the :initial-* arguments, which is what I had assumed, just wanted to confirm that.

...

I don't know what expand-into-foreign-memory is.

Have a look at: http://common-lisp.net/project/cffi/manual/html_node/Optimizing-Type-Translators.html.

OK, so it's an optimization issue. Can this be deferred until later?

Cheers,

-- Luís Oliveira http://r42.eu/~luis/

I've written some code to see how this approach would go: https://github.com/cffi/cffi/blob/cd744b78e13d2469f22cd1b7116c75499c63c8f5/s... Look under "Luis' approach". It all seems to work except for translation of real-and-complex back to Lisp; I can only get it to translate the complex if it's a macro. Any thoughts why?

Liam

Am 08.09.2011 um 12:20 schrieb Luís Oliveira:

On Thu, Sep 8, 2011 at 5:23 AM, Liam Healy lnp@healy.washington.dc.us wrote:

...

6. Finding that (parse-type 'complex) is an instance of neither

ENHANCED-TYPEDEF nor ENHANCED-FOREIGN-TYPE, the expand-from-foreignT T method is called, which just returns value (first arg).

So there's a gap here, and I'm not sure how to plug it. Should I be making complex-type a subclass of enhanced-foreign-type so that the compiler macro picks up the conversion to lisp?

Oh. That's right, unless it's an enhanced-foreign-type it won't go through the translation mechanism. Adding that as superclass should work.

I think we need a new DEFCSTRUCT-like macro, let's call it DEFCSTRUCT* for now. *sigh*

May I vote for DEFCSTRUCT-BY-VALUE as its name?

With a different macro we can (a) implement call-by-value semantics by default, (2) add the enhanced-foreign-type superclass by default, and probably (3) have a sane value for :CLASS, not sure.

Does that makes sense?

Yes!

Go, go, go ! ;-)

Cheers Frank

Hi again!

I'm still stuck with the following C code (simplified):

-X-X-X-

// Now, get the locationID of this device. In order to do this, we need to create an IOUSBDeviceInterface // for our device. This will create the necessary connections between our userland application and the // kernel object for the USB Device.

kr = IOCreatePlugInInterfaceForService(usbDevice, kIOUSBDeviceUserClientTypeID, kIOCFPlugInInterfaceID, &plugInInterface, &score);

// Use the plugin interface to retrieve the device interface. res = (*plugInInterface)->QueryInterface( plugInInterface, CFUUIDGetUUIDBytes( kIOUSBDeviceInterfaceID ), (LPVOID*) &privateDataRef->deviceInterface );

// Now done with the plugin interface. (*plugInInterface)->Release(plugInInterface);

-X-X-X-

I have translated this to:

-X-X-X-

(let ((kr nil) (res nil) (deviceName (cffi:foreign-alloc :char :count 128)) (deviceNameAsCFString nil) (privateDataRef nil) (plugInInterface (cffi:foreign-alloc 'IOCFPlugInInterface :count 1)) (score (cffi:foreign-alloc 'SInt32 :count 1)) (locationID nil))

(setq kr (io-registry-entry-get-name usb-device deviceName))

(when (not (eql kr KERN_SUCCESS)) (setf (cffi:mem-ref deviceName 'io_name_t 0) 0))

(setq deviceNameAsCFString (cf-string-create-with-c-string *kCFAllocatorDefault* (foreign-string-to-lisp deviceName) (foreign-enum-value '__kCFStringEncoding :kCFStringEncodingASCII)))

(setq privateDataRef (cffi:foreign-alloc 'MyPrivateData :count 1))

(setf (foreign-slot-value privateDataRef 'MyPrivateData 'deviceName) deviceNameAsCFString)

(setq kr (io-create-plug-in-interface-for-service usb-device kIOUSBDeviceUserClientTypeID kIOCFPlugInInterfaceID plugInInterface score))

(if (not (eql kr kIOReturnSuccess))

(format *debug-io* "~%*** ERROR: io-create-plug-in-interface-for-service: ~s." kr)

(progn ;; else

(let ((query-interface-ptr (foreign-slot-value plugInInterface 'IOCFPlugInInterface 'ccag.osx.iokit::QueryInterface)) (kIOUSBDeviceInterfaceID-uuid-bytes (cf-uuid-get-uuid-bytes kIOUSBDeviceInterfaceID)))

(format *debug-io* "~%DEVICE-ADDED: query-interface-ptr = ~s." query-interface-ptr)

(setq res (foreign-funcall-pointer query-interface-ptr () :pointer plugInInterface CFUUIDBytes kIOUSBDeviceInterfaceID-uuid-bytes :pointer (foreign-slot-pointer privateDataRef 'MyPrivateData 'deviceInterface))) ....

-X-X-X-

I get everything ok up until I am trying to call foreign-funcall-pointer ... - for which I have no way to create a call-by-value definition using FSBV.

I am sure I am not seeing the forest for the trees ... My mind seems trying to always do direct translations from C to Lisp where another approach is required ... I am eager to hear your thoughts... Thanks a lot !

Best wishes Frank

On Thu, Sep 8, 2011 at 9:49 AM, Luís Oliveira luismbo@gmail.com wrote:

On Thu, Sep 8, 2011 at 2:07 PM, Frank Goenninger frgo@me.com wrote:

...

I get everything ok up until I am trying to call foreign-funcall-pointer ... - for which I have no way to create a call-by-value definition using FSBV.

Have a look at FSBV:FOREIGN-FUNCALL's definition. It shouldn't be too hard to define a similar wrapper on top of CFFI:FOREIGN-FUNCALL-POINTER.

So I guess an enhanced foreign-funcall-pointer is a feature request for the future merged FSBV in CFFI. When we get to that point, I'll have to remember to put that in.

Liam

On Thu, Sep 8, 2011 at 11:36 AM, Attila Lendvai attila.lendvai@gmail.com wrote:

On Thu, Sep 8, 2011 at 13:15, Luís Oliveira luismbo@gmail.com wrote:

Oh. That's right, unless it's an enhanced-foreign-type it won't go through the translation mechanism. Adding that as superclass should work.

I think we need a new DEFCSTRUCT-like macro, let's call it DEFCSTRUCT* for now. *sigh*

With a different macro we can (a) implement call-by-value semantics by default, (2) add the enhanced-foreign-type superclass by default, and probably (3) have a sane value for :CLASS, not sure.

Does that makes sense?

...

Suggestions on how to best handle backwards-incompatible API changes like this are most welcome.

my 0.02: tag the repo, make a big fat warning in the commit message, and go on developing.

--  attila

I agree on the strategy where an incompatible change is necessary, but in this case in particular I don't see that a backwards-incompatible change is necessary. Can't we by default make defcstruct objects instances of enhanced-foreign-type by default without negatively affecting call-by-reference applications? If not, we can add an optional argument, but I'm concerned about cases where we need both call by ref and by value.

Liam

On Thu, Sep 8, 2011 at 5:30 PM, Liam Healy lnp@healy.washington.dc.us wrote:

Why not defcstruct (suitably enhanced, of course) for everything?  I don't want to preclude calling by reference those foreign structures I also need to call by value.  So something that's called exclusively by reference should work too.  If an application which only calls by reference doesn't want to use the translators, we can leave them undefined.

Right now (defcfun foo :void (x some-struct-type)) is identical to (defcfun foo :void (x (:pointer some-struct-type))). If you want to change the former to be call-by-value we need a backwards-incompatible change to DEFCSTRUCT such that a bare SOME-STRUCT-TYPE means call-by-value.

I think you're right that adding ENHANCED-FOREIGN-TYPE to the superclasses wouldn't be an issue.

On Thu, Sep 8, 2011 at 5:56 PM, Liam Healy lnp@healy.washington.dc.us wrote:

I see this is a simple change to the function call interface: naming the struct means call by value; if you want to call by reference, use :pointer.

That's Attila's suggestion. Break it and move on.

We can even make a warning if you call by reference with the struct name and don't have FSBV loaded (remember we are going to keep this a separate system so that users don't have to load libffi if they don't need call by value).

That's nicer, but fsbv might be loaded for some other reason and thus silently break existing code.

So this is an incompatible change to the call interface, essentially revoking CFFI's prior generosity in allowing structure name instead of :pointer.  I definitely don't like the idea of making different defcstructs, that will be too confusing and a kludge.

Just to be clear, the older defcstruct would become deprecated, removed from the user manual, etc.

On Fri, Sep 9, 2011 at 2:50 AM, Jean-Claude Beaudoin jean.claude.beaudoin@gmail.com wrote:

Has anyone of you given a look at what Franz did on this very subject for their Allegro 8.2 (http://www.franz.com/support/documentation/8.2/doc/contents.htm#foreign-func...) (http://www.franz.com/support/documentation/8.2/doc/foreign-functions.htm#str...) They added a keyword argument to their def-foreign-call form and also added a global variable ff:*pass-structs-by-value* that controls the default value of the keyword arg.

What do you think of that?

Cheers,

Jean-Claude Beaudoin

I like their approach, and I think we should follow a similar path with the argument passing determined by declaration (:pointer for call by reference and the stucture type name for call by value), but I would be a little stronger on the default behavior. I would make an "notification-interpretation" global variable that determines what the compiler will do (notification and possible conversion) if it encounters a structure name declaration to a function argument or return. Options could be (:warn :pointer), (:error), (:accept :pointer), (:accept :value), etc., :value meaning call by value and :pointer meaning call by reference. The current CFFI has (:accept :pointer). The default on release could be (:warn :pointer) if FSBV is not loaded, and (:warn :value) if it is. After a while (succeeding release, say), we should change this to (:warn :pointer) and (:accept :value), respectively. Ultimately, I think it should be (:error) and (:accept :value) respectively.

Liam

...

...

...

...

On Thu, 8 Sep 2011 17:48:57 +0100, LuÍs Oliveira said:

On Thu, Sep 8, 2011 at 5:30 PM, Liam Healy lnp@healy.washington.dc.us wrote:

...

Why not defcstruct (suitably enhanced, of course) for everything?  I don't want to preclude calling by reference those foreign structures I also need to call by value.  So something that's called exclusively by reference should work too.  If an application which only calls by reference doesn't want to use the translators, we can leave them undefined.

Right now (defcfun foo :void (x some-struct-type)) is identical to (defcfun foo :void (x (:pointer some-struct-type))). If you want to change the former to be call-by-value we need a backwards-incompatible change to DEFCSTRUCT such that a bare SOME-STRUCT-TYPE means call-by-value.

C has a separate namespace for struct/union tags, so a more complete alternative is to use (:struct some-struct-type) everywhere (the LispWorks FLI does this).

Passing the structure by value would be

(defcfun foo :void (x (:struct some-struct-type)))

and passing it by reference would be

(defcfun foo :void (x (:pointer (:struct some-struct-type))))

CFFI could retain the semantics of plain some-struct-type to mean (:pointer some-struct-type) at the top level of defcfun and (:struct some-struct-type) everywhere else, though it would still be broken by cases like this:

typedef double foo; struct foo { int x; }; void foofoo(foo x, struct foo y);

__Martin

Am 08.09.2011 um 18:30 schrieb Liam Healy:

On Thu, Sep 8, 2011 at 7:15 AM, Luís Oliveira luismbo@gmail.com wrote:

...

On Thu, Sep 8, 2011 at 12:06 PM, Frank Goenninger frgo@me.com wrote:

...

May I vote for DEFCSTRUCT-BY-VALUE as its name?

I agree that DEFCSTRUCT* is a lousy name, but DEFCSTRUCT-BY-VALUE is misleading, since there's more going on. How about DEFINE-FOREIGN-STRUCTURE. In any case, the idea is to deprecate DEFCSTRUCT.

Suggestions on how to best handle backwards-incompatible API changes like this are most welcome.

-- Luís Oliveira http://r42.eu/~luis/

Why not defcstruct (suitably enhanced, of course) for everything? I don't want to preclude calling by reference those foreign structures I also need to call by value. So something that's called exclusively by reference should work too. If an application which only calls by reference doesn't want to use the translators, we can leave them undefined.

Yep, true. KISS - it's always better to be as simple as possible.

Frank

On Fri, Sep 9, 2011 at 4:46 AM, Joerg-Cyril.Hoehle@t-systems.com wrote:

Hi,

Luís Oliveira wrote:

...

I agree that DEFCSTRUCT* is a lousy name, but DEFCSTRUCT-BY-VALUE is misleading, since there's more going on.

I recommend you look again at CLISP's FFI http://www.clisp.org/impnotes/dffi.html It is essentially call by value.  It will convert (de-/serialize) arbitrarily nested structures to and from Lisp, for instance a full linked list of objects given a single pointer (and hang on a circular list...). Passing stuff by reference (i.e. using the type (C-POINTER XYZ) was a late addition.

On a quick read, I can't discern how CLISP differentiates syntactically between call by reference and by value.

Note that automated return by value is not void of problems.  Old APIs use structs or &out pointers with flags indicating which fields are valid and you better not convert back an invalid char* (or random bits as a float). Modern APIs tend to do better here, esp. those that have been influenced by remote design thoughts (RPC, network transmission etc.).

The old Haskell/Direct had a domain specific language to express such dependencies and I've found myself suggesting some for CLISP (e.g. a :guard to prevent dereferencing some slots, as in (def-c-fun foobar (:return-type int) ... (errorstr (:type c-string) (:out :alloca) (:guard (zerop return))) => foobar returns type (values integer (or string null))  with integer return value being not 0 => NIL as second value, no string (or the converse, I forgot how I originally conceived the guard)

Regards,        Jörg Höhle

This is a good point; it would be nice to have some means to prevent autoconversion.

Liam

On Fri, Sep 9, 2011 at 3:30 PM, Liam Healy lnp@healy.washington.dc.us wrote:

I'm having trouble with this.  If I simply make it a superclass, I get an error: Must specify an ACTUAL-TYPE.  If I fill the slot when I try to define the class, I get an error that complex is not a CFFI type. I'm not knowledgeable about the various classes and what an alias type means, etc.  Can you fill me in on exactly how the new type class should be made when it is a subclass of both foreign-struct-type and enhanced-foreign-type?

Oh, ENHANCED-FOREIGN-TYPE assumes that the type will boil down to one of the primitive types. (:int, :long, etc.) I guess we have to yank that out into a new subclass. I'd be happy to some hacking along those lines later tonight or tomorrow if that gives you too much trouble.

Cheers,