Hi,
I'm finally able to spend some time on CFFI again, so I've been trying to get caught up with the excellent progress Luis has made on the CFFI type system.
I think I'm starting to understand how the new type translator stuff works, but I'm a little wary of the :TO-C translation as used from callbacks. For example:
(defcallback return-a-string :string () "Returns a string allocated in Lisp to C." (format nil "Hello, ~D." 42))
As Luis's comment in the C code portion of the callback tests indicates, C code that calls this callback has no reliable way to deallocate the memory allocated implicitly by the type translator. The only way to deallocate it is from Lisp via FOREIGN-STRING-FREE.
I think this is probably not too bad if we document it very clearly, but it worries me. Making type translators not apply to the return value of callbacks would remove the possibility for a non-obvious memory leak here.
Another option might be a :malloced-string type with a :TO-C translator that explicitly malloc's the buffer---or just specify that FOREIGN-STRING-ALLOC always uses malloc, I suppose. Obviously that wouldn't apply to possibly stack-allocated strings allocated by WITH-FOREIGN-STRING...
In any other situation, it'd probably be better to use a result-type of :POINTER.
Thoughts?
James
On 2005-nov-15, at 06:28, James Bielman wrote:
As Luis's comment in the C code portion of the callback tests indicates, C code that calls this callback has no reliable way to deallocate the memory allocated implicitly by the type translator. The only way to deallocate it is from Lisp via FOREIGN-STRING-FREE.
I think this is probably not too bad if we document it very clearly, but it worries me. Making type translators not apply to the return value of callbacks would remove the possibility for a non-obvious memory leak here.
Another option might be a :malloced-string type with a :TO-C translator that explicitly malloc's the buffer---or just specify that FOREIGN-STRING-ALLOC always uses malloc, I suppose. Obviously that wouldn't apply to possibly stack-allocated strings allocated by WITH-FOREIGN-STRING...
Or we could define yet another kind of translator, say :CALLBACK-TO- C, that would normally inherit from :TO-C and that in :string's case would use malloc. (hmm, this gives me some ideas on how to rewrite the type system.)
I think that WITH-FOREIGN-STRING, WITH-FOREIGN-OBJECT and WITH- FOREIGN-PTR should take a :MALLOC option, as should FOREIGN-ALLOC, etc... I have thought about this before, probably because CLISP does this IIRC.
Also, the same applies to foreign variables.
On Tue, 2005-11-15 at 15:17 +0000, Luís Oliveira wrote:
Or we could define yet another kind of translator, say :CALLBACK-TO- C, that would normally inherit from :TO-C and that in :string's case would use malloc. (hmm, this gives me some ideas on how to rewrite the type system.)
Also, the same applies to foreign variables.
Well, I don't see the point of using a separate translator for callback return values versus setting foreign variables. If I understand :TO-C correctly, it is used when we have a Lisp value we need to convert to a C value with indefinite extent, which is the situation in both those cases.
The more I think about it, the more I like just specifying the built-in type translators for aggregate objects (which is just :STRING, right?) allocate their memory using malloc in the :TO-C case. I can't see any real good reason not to.
If I had an existing string I wanted to pass from a callback I would just declare it to return :POINTER instead. (What happens if you return a pointer from a function with a result-type of :STRING? Should it pass it unmodified or malloc a copy?)
I think that WITH-FOREIGN-STRING, WITH-FOREIGN-OBJECT and WITH- FOREIGN-PTR should take a :MALLOC option, as should FOREIGN-ALLOC, etc... I have thought about this before, probably because CLISP does this IIRC.
Yes, that sounds like a good idea. Then we could document that the :TO-C translation for :STRING uses (FOREIGN-STRING-ALLOC :MALLOC T).
James
On 2005-nov-15, at 15:34, James Bielman wrote:
Well, I don't see the point of using a separate translator for callback return values versus setting foreign variables. If I understand :TO-C correctly, it is used when we have a Lisp value we need to convert to a C value with indefinite extent, which is the situation in both those cases.
Good point, that was the purpose of separating :TO-C from :TO-C-DYNAMIC.
The more I think about it, the more I like just specifying the built-in type translators for aggregate objects (which is just :STRING, right?) allocate their memory using malloc in the :TO-C case. I can't see any real good reason not to.
Right. Must not forget to document this well.
If I had an existing string I wanted to pass from a callback I would just declare it to return :POINTER instead. (What happens if you return a pointer from a function with a result-type of :STRING? Should it pass it unmodified or malloc a copy?)
Good question. My gut feeling is that it should be passed unmodified since if you're passing a pointer it probably means that you want to bypass the translator. But I'm not sure.
-
James Bielman -
Luís Oliveira