Hi,
I am forwarding you a question from one of our developers, who is working on updating CFFI to work with ECL.
His question is how you handle garbage collection and foreign variables. When setting these variables with, say, a copy of a string (setf *var-string* "hello") a region of memory has to be allocated. But what happens when we set this variable again afterwards? (setf *var-string* "world")
Does CFFI assume that foreign variables become part of the set of roots to be considered by the garbage collector? This seems a strong assumption, given the fact that not all variables will contain pointers, etc.
Or is the user expected to free the first string by herself?
Juanjo
On 24/out/2005, at 13:09, Juan Jose Garcia Ripoll wrote:
I am forwarding you a question from one of our developers, who is working on updating CFFI to work with ECL.
I read that, it's great news.
His question is how you handle garbage collection and foreign variables. When setting these variables with, say, a copy of a string (setf *var-string* "hello") a region of memory has to be allocated. But what happens when we set this variable again afterwards? (setf *var-string* "world")
Does CFFI assume that foreign variables become part of the set of roots to be considered by the garbage collector? This seems a strong assumption, given the fact that not all variables will contain pointers, etc.
Or is the user expected to free the first string by herself?
Right now the user should free the first string (or whatever) by himself. I plan to add some options to DEFCVAR to make this automatic (something like what CLISP has). For example the setter that DEFCVAR defines would automatically free the previous contents of the foreign var.
On the first SETF, CFFI will allocate a foreign string and set *var-string* to point to it. The second SETF will do the same, without deallocating the string "hello". To deallocate it, you would have to create a foreign-data object pointing to the same address as *var-string* and then free it, this looks tricky....
Well it is a bit tricky to free because you have to dodge the translator... but not *that* tricky.
(setf *fvar* "foo") (foreign-free (mem-ref (get-var-ptr '*fvar*) :pointer)) (setf *fvar* "new string")
Any suggestions on how to handle this better? Or is the automatic solution I mention before (regarding the setter) enough?
Btw, the test case Michael is talking about (foreign-globals.set.string) does indeed leak memory, I've fixed that.
On Mon, 2005-10-24 at 23:43 +0100, Luis Oliveira wrote:
Right now the user should free the first string (or whatever) by himself. I plan to add some options to DEFCVAR to make this automatic (something like what CLISP has). For example the setter that DEFCVAR defines would automatically free the previous contents of the foreign var. [...] Any suggestions on how to handle this better? Or is the automatic solution I mention before (regarding the setter) enough?
The point is that if the foreign var does not belong to you, some other component of the library might overwrite the variable you are setting. What can one then do if the pointer to the data that is to be freed is lost?
To make my case clear, suppose that you have a C variable
const char *error_message;
This is quite common in C, as the error_message is expected to be statically allocated. However, somewhere in your code you define *error-message* to be the foreign variable pointing to it and you do
(setf *error-message* "Unable to access temporary directory")
Now some other component of the library overwrites this variable because a different error condition happened
error_message = "File not found";
and then your lisp code tries to
(foreign-free (mem-ref (get-var-ptr '*error-message*) :pointer))
Kaboom! :-)
In ECL's implementation of UFFI, foreign variables are defined once and a property list is associated to the symbol. Whenever you set a foreign variable with some data, the foreign data generated by the lisp is stored in that property list, hence preventing garbage collection.
When you set the same variable a second time, the original lisp value is lost and then it is garbage collected. An alternative would be to free the value as soon as it is removed from the property list.
Regards
Juan Jose Garcia Ripoll lisp@arrakis.es writes:
Now some other component of the library overwrites this variable because a different error condition happened
error_message = "File not found";
and then your lisp code tries to
(foreign-free (mem-ref (get-var-ptr '*error-message*) :pointer))
Kaboom! :-)
Right. Well, at the momment the user will simply have to be careful in such a situation by saving the pointer elsewhere. I can imagine other situation where one would have to be careful; what if the foreign code free()s our data? I'm not sure if this is common though.
In ECL's implementation of UFFI, foreign variables are defined once and a property list is associated to the symbol. Whenever you set a foreign variable with some data, the foreign data generated by the lisp is stored in that property list, hence preventing garbage collection.
Is that the way UFFI works in other implementations though? I mean, in other implementations you have to free the memory yourself, right?
When you set the same variable a second time, the original lisp value is lost and then it is garbage collected. An alternative would be to free the value as soon as it is removed from the property list.
CFFI could certainly do the latter.
Ahem. I think it's good to have higher-level things that do more for you, but they should be layered portably on top of the low-level access primitives. Use symbol-macros, define-set-expander and suches if you want your variable access to magically manage memory. CFFI should start with the low-level.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Friedrich Hayek was the first object-oriented programmer. -- Bill Tulloh
On 26/out/2005, at 22:36, Faré wrote:
Ahem. I think it's good to have higher-level things that do more for you, but they should be layered portably on top of the low-level access primitives. Use symbol-macros, define-set-expander and suches if you want your variable access to magically manage memory. CFFI should start with the low-level.
Oh yes, of course. 'low-level' is a relative concept though. Do you think CFFI's current approach is too high-level?
1. foreign-symbol-ptr to get a pointer to the foreign variable.
2. defcvar to define a symbol macro for accessing a foreign variable (built on top of foreign-symbol-ptr).
3. eventually expand defcvar to do more for the user wrt to memory management.
On 26/10/05, Luis Oliveira luismbo@gmail.com wrote:
On 26/out/2005, at 22:36, Faré wrote:
Ahem. I think it's good to have higher-level things that do more for you, but they should be layered portably on top of the low-level access primitives. Use symbol-macros, define-set-expander and suches if you want your variable access to magically manage memory. CFFI should start with the low-level.
Oh yes, of course. 'low-level' is a relative concept though. Do you think CFFI's current approach is too high-level?
No, as far as I can tell (which is not that much), CFFI does about the right thing. The proposals by the ECL people (automatic management of alien pointer variables) seem to me like they could and should be layered on top of CFFI instead of being proposed as something that CFFI should use.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] It is now quite lawful for a Catholic woman to avoid pregnancy by a resort to mathematics, though she is still forbidden to resort to physics and chemistry. -- H.L. Mencken
-
Faré -
Juan Jose Garcia Ripoll -
Luis Oliveira