[cffi-devel] A question about memory management and garbage collection
Hi folks! I have the follwoing situation: A foreign function call returns a pointer to a "table" object. This is an opaque object that can only be manipulated by the APIs provided by the foreign library. To avoid memory leaks the "table" objects should by destroyed by a call to "destroy_table" function if they are not used anymore. I have wrapped the "table" object and its corresponding API functions in a CLOS object with appropriate methods. Instances of this CLOS object will be returned by function calls on the Lisp side. The CLOS instance contains a reference to a C "table" object (allocated on C-heap by foreign library). If a caller in a Lisp program ignores the returned "table" object, it should be garbage collected as expected. My questions are now: 1) Will it be really garbage collected? 2) If the answer is in affirmative, is it garbage collected correctly (no potential memory leaks)? Please note that the C API offers as mentioned above its own dedicated function for that purpose. What would be a correct solution to avoid memory leaks in this situation? Using a finalizer seems to be a possible way to go. But I'm not sure how this can be done by just using the finalizer parameters "object" and "function" where "function" can't reliablely access "object" (cf. documentation of finalizer in "trivial-garbage" package)!! Any help and feedback is very much appreciated! Regards Nik
On Sun, May 8, 2011 at 1:39 PM, Nitralime <nitralime@googlemail.com> wrote:
What would be a correct solution to avoid memory leaks in this situation? Using a finalizer seems to be a possible way to go. But I'm not sure how this can be done by just using the finalizer parameters "object" and "function" where "function" can't reliablely access "object" (cf. documentation of finalizer in "trivial-garbage" package)!!
Here's some pseudo-code to achieve what you're looking for: (let ((ptr (grab-your-table-pointer))) (tg:finalize your-wrapper-object (lambda () (table-free ptr)))) HTH, -- Luís Oliveira http://r42.eu/~luis/
Thanks a lot for your feedback! I have also intended to try finalization using a closure! But I wasn't that sure if it is an "idomatic" way to do that or there maybe an even more clever way for accomplishing it. By further googling yesterday I have also found similar code fragments at /http://code.google.com/p/lispbuilder/source/browse/trunk/lispbuilder-sdl/sdl... I'm just wondering about "cffi:finalize" which I couldn't find in my own installation of cffi package! Am I missing something? Regards Nik On 05/09/2011 12:16 AM, Luís Oliveira wrote:
On Sun, May 8, 2011 at 1:39 PM, Nitralime<nitralime@googlemail.com> wrote:
What would be a correct solution to avoid memory leaks in this situation? Using a finalizer seems to be a possible way to go. But I'm not sure how this can be done by just using the finalizer parameters "object" and "function" where "function" can't reliablely access "object" (cf. documentation of finalizer in "trivial-garbage" package)!! Here's some pseudo-code to achieve what you're looking for:
(let ((ptr (grab-your-table-pointer))) (tg:finalize your-wrapper-object (lambda () (table-free ptr))))
HTH,
On Mon, May 9, 2011 at 10:14 AM, nitralime <nitralime@googlemail.com> wrote:
I'm just wondering about "cffi:finalize" which I couldn't find in my own installation of cffi package!
trivial-garbage's finalizer code lived in CFFI for a brief period of time, but was moved into its own project a couple of years ago. Cheers, -- Luís Oliveira http://r42.eu/~luis/
On Sun, 08 May 2011 14:39:41 +0200, Nitralime said:
Using a finalizer seems to be a possible way to go. But I'm not sure how this can be done by just using the finalizer parameters "object" and "function" where "function" can't reliablely access "object" (cf. documentation of finalizer in "trivial-garbage" package)!!
Any help and feedback is very much appreciated!
IMHO, attempting to transparently wrap a foreign object with a CLOS object is not a good design. Using a finalizer is dangerous because it is difficult to guarantee that the Lisp wrapper object is retained for the correct lifetime. For example, consider (defstruct table foreign-pointer) (defun allocate-table () (make-table :foreign-pointer (foreign-allocate-table))) (defun allocate-and-munge-table () (let ((table (allocate-table))) (foreign-munge-table (table-foreign-pointer table)))) The compiler might not keep the variable "table" alive during the call to the foreign function foreign-munge-table. As a result, the Lisp table object might be gc'ed in another thread while the foreign code is accessing it. -- Martin Simmons LispWorks Ltd http://www.lispworks.com/
On 05/09/2011 01:49 PM, Martin Simmons wrote:
On Sun, 08 May 2011 14:39:41 +0200, Nitralime said: Using a finalizer seems to be a possible way to go. But I'm not sure how this can be done by just using the finalizer parameters "object" and "function" where "function" can't reliablely access "object" (cf. documentation of finalizer in "trivial-garbage" package)!!
Any help and feedback is very much appreciated! IMHO, attempting to transparently wrap a foreign object with a CLOS object is not a good design.
Using a finalizer is dangerous because it is difficult to guarantee that the Lisp wrapper object is retained for the correct lifetime.
For example, consider
(defstruct table foreign-pointer)
(defun allocate-table () (make-table :foreign-pointer (foreign-allocate-table)))
(defun allocate-and-munge-table () (let ((table (allocate-table))) (foreign-munge-table (table-foreign-pointer table))))
The compiler might not keep the variable "table" alive during the call to the foreign function foreign-munge-table. As a result, the Lisp table object might be gc'ed in another thread while the foreign code is accessing it.
Consider the following sample (pseudo-)code: ---------------------------------------------------------- (defclass table () ((table-pointer :initarg :handle :accessor handle-of))) (defmethod initialize-instance :after ((self table) &key) (let ((table-pointer (handle-of self))) (tg:finalize self (lambda () (foreign-destroy-table table-pointer))))) (defmethod get-row ((t table) index) (transform-to-lisp-vector (foreign-get-row (hadle-of t) index))) (defmethod move-to ((t table) index) (foreign-move-to (handle-of t) index)) ..... ---------------------------------------------------------- Now consider a function which creates a table instance and puts it into a result plist (and has probabely some side effects): (defun call-a-foreign-function (.....) (let (....) .... <call a foreign function which allocates a table object and returns a-foreign-table-pointer> .... (let ((result-table (make-instance 'table :handle a-foreign-table-pointer)) (...)) ... ... '(:id ... :tab result-table :msg ... ...)))) The question is now what will happen if you do just (call-a-foreign-function ...) or something like this (getf (call-a-foreign-function ...) :id) I assume that the unreachable object 'result-table' will be garbage collected, and the finalization will free the corresponding C table object! Do you see any misconception (resp. potential danger) here? Regards Nik
On Mon, 09 May 2011 16:46:20 +0200, nitralime said:
On 05/09/2011 01:49 PM, Martin Simmons wrote:
> On Sun, 08 May 2011 14:39:41 +0200, Nitralime said: Using a finalizer seems to be a possible way to go. But I'm not sure how this can be done by just using the finalizer parameters "object" and "function" where "function" can't reliablely access "object" (cf. documentation of finalizer in "trivial-garbage" package)!!
Any help and feedback is very much appreciated! IMHO, attempting to transparently wrap a foreign object with a CLOS object is not a good design.
Using a finalizer is dangerous because it is difficult to guarantee that the Lisp wrapper object is retained for the correct lifetime.
For example, consider
(defstruct table foreign-pointer)
(defun allocate-table () (make-table :foreign-pointer (foreign-allocate-table)))
(defun allocate-and-munge-table () (let ((table (allocate-table))) (foreign-munge-table (table-foreign-pointer table))))
The compiler might not keep the variable "table" alive during the call to the foreign function foreign-munge-table. As a result, the Lisp table object might be gc'ed in another thread while the foreign code is accessing it.
Consider the following sample (pseudo-)code: ---------------------------------------------------------- (defclass table () ((table-pointer :initarg :handle :accessor handle-of)))
(defmethod initialize-instance :after ((self table) &key) (let ((table-pointer (handle-of self))) (tg:finalize self (lambda () (foreign-destroy-table table-pointer)))))
(defmethod get-row ((t table) index) (transform-to-lisp-vector (foreign-get-row (hadle-of t) index)))
(defmethod move-to ((t table) index) (foreign-move-to (handle-of t) index)) ..... ----------------------------------------------------------
Now consider a function which creates a table instance and puts it into a result plist (and has probabely some side effects):
(defun call-a-foreign-function (.....) (let (....) .... <call a foreign function which allocates a table object and returns a-foreign-table-pointer> .... (let ((result-table (make-instance 'table :handle a-foreign-table-pointer)) (...)) ... ... '(:id ... :tab result-table :msg ... ...))))
The question is now what will happen if you do just
(call-a-foreign-function ...)
or something like this
(getf (call-a-foreign-function ...) :id)
I assume that the unreachable object 'result-table' will be garbage collected, and the finalization will free the corresponding C table object!
Do you see any misconception (resp. potential danger) here?
Your examples will be safe, but (get-row (getf (call-a-foreign-function ...) :tab) 0) might crash. -- Martin Simmons LispWorks Ltd http://www.lispworks.com/
On 05/09/2011 06:44 PM, Martin Simmons wrote:
Your examples will be safe, but
(get-row (getf (call-a-foreign-function ...) :tab) 0)
might crash.
I got it now!! I thought that an object in a nested call like (get-row (getf (call-a-foreign-function ...) :tab) 0) must be considered as live/reachable until the outer most call is complete. Otherwise, you will have the same problem even with pure Lisp object as well. I'm not very familiar with the delicate technical aspects of runtime systems of Lisp implementations and maybe I make a fundamental mistake at that point!! Regards Nik
participants (4)
-
Luís Oliveira -
Martin Simmons -
Nitralime -
nitralime