To prevent any confusion beforehand: The mail below talks about class objects, objects which store class definitions, not the actual instances of those classes, which use those class definitions and store class-allocated or instance values.
We have fixed a few bugs where memory would be allocated during (package creation or) FASL loading, but would not be freed upon package deletion. That was an issue because it's an explicit use-case for Blake.
Last weekend I came up with another "leak" in ABCL. The word "leak" is between quotes, because depending on how you look at it, not all of it may technically be leaky.
My point: it's easy to create a class; simply use a DEFCLASS form. However, unlike package creation, which can be "undone" through DELETE-PACKAGE, there's no DELETE-CLASS function to undo the result of a DEFCLASS.
I started to think about supporting the use-case of deleting a class. Digging around in our implementation, I found there are several reasons why a class won't be automatically cleaned up when the package containing its naming symbol is deleted:
1. Every parent maintains a list of subclasses The list contains class objects, meaning that the actual class-object is referenced in this list and for that reason can't be GC-ed 2. Every generic function maintains a list of specializing methods The methods in the list refer to the dispatch types, meaning that the class-objects are strongly referenced from there too 3. Our FIND-CLASS implementation contains strong references to both the naming symbols as well as the class-objects. 4. Our implementation uses a cache to speed up the calls to generic functions. This cache uses strong references too.
The issue in item (3) can be solved (relatively) easy by developing a weak hash table which contains both weak keys and values. This hash table could even be made available as regular Lisp objects.
Item (4) currently uses an explicitly specialized hash table implemented on the Java side. The hash table could be adapted to use weak references as well - the effort wouldn't be as generally usable, but that's just tough luck.
Items (1) and (2) are something to think about though: Is it wise to implement a function which removes the methods and the class from its parent? What if the class itself has direct subclasses? Do we signal an error? Do we provide a parameter forcing the deletion of a whole tree? Or maybe we should just document how potential users of our CLOS should proceed if they have the intention to do this, since it won't be portable anyway?
Myself, I'm currently contemplating "just document" on one hand and "do whatever we can, even if they're whole trees" on the other...
What do other people think?
Bye,
Erik.
Greetings,
First of all, thanks so much for help with this issue. I would not have been able to use ABCL without the fantastic support I've gotten. It is really appreciated!
I did a little checking and CL does seem to localize class definitions to packages as it does for regular functions. Not too surprising. So, it does make sense for a class definition to be totally eradicated when a package is explicitly deleted.
The serious down side of not eliminating a class definition is that if a package is load, deleted, and reloaded many times, the act of defining the same class (presumably in the same package name but will actually be creating a new package structure) causes (in essence) a memory leak.
Thanks.
Blake McBride
On Thu, Jan 27, 2011 at 1:19 PM, Erik Huelsmann ehuels@gmail.com wrote:
To prevent any confusion beforehand: The mail below talks about class objects, objects which store class definitions, not the actual instances of those classes, which use those class definitions and store class-allocated or instance values.
We have fixed a few bugs where memory would be allocated during (package creation or) FASL loading, but would not be freed upon package deletion. That was an issue because it's an explicit use-case for Blake.
Last weekend I came up with another "leak" in ABCL. The word "leak" is between quotes, because depending on how you look at it, not all of it may technically be leaky.
My point: it's easy to create a class; simply use a DEFCLASS form. However, unlike package creation, which can be "undone" through DELETE-PACKAGE, there's no DELETE-CLASS function to undo the result of a DEFCLASS.
I started to think about supporting the use-case of deleting a class. Digging around in our implementation, I found there are several reasons why a class won't be automatically cleaned up when the package containing its naming symbol is deleted:
- Every parent maintains a list of subclasses The list contains class objects, meaning that the actual class-object is referenced in this list and for that reason can't be GC-ed
- Every generic function maintains a list of specializing methods The methods in the list refer to the dispatch types, meaning that the class-objects are strongly referenced from there too
- Our FIND-CLASS implementation contains strong references to both the naming symbols as well as the class-objects.
- Our implementation uses a cache to speed up the calls to generic functions. This cache uses strong references too.
The issue in item (3) can be solved (relatively) easy by developing a weak hash table which contains both weak keys and values. This hash table could even be made available as regular Lisp objects.
Item (4) currently uses an explicitly specialized hash table implemented on the Java side. The hash table could be adapted to use weak references as well - the effort wouldn't be as generally usable, but that's just tough luck.
Items (1) and (2) are something to think about though: Is it wise to implement a function which removes the methods and the class from its parent? What if the class itself has direct subclasses? Do we signal an error? Do we provide a parameter forcing the deletion of a whole tree? Or maybe we should just document how potential users of our CLOS should proceed if they have the intention to do this, since it won't be portable anyway?
Myself, I'm currently contemplating "just document" on one hand and "do whatever we can, even if they're whole trees" on the other...
What do other people think?
Bye,
Erik.
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
On Thu, Jan 27, 2011 at 9:26 PM, Blake McBride blake@mcbride.name wrote:
Greetings, First of all, thanks so much for help with this issue. I would not have been able to use ABCL without the fantastic support I've gotten. It is really appreciated!
You're welcome. We hope that our support is of mutual benefit: I'm sure the issues you have experienced so far will be or would have been experienced in the future by others too. Your continued efforts help shake up ABCL for better large-scale systems development. I hope I'm speaking for the other developers when I say I appreciate the trust you put in us by embarking on this effort.
By the way, you've seen little activity on this issue because I'm trying to clean up some CLOS code to address an issue reported by Alan Ruttenberg in december. There were two ways to solve his report; one by quick hacking, the other by diving in deep and trying to use the solution to move our CLOS a bit closer to the AMOP spec. If you know me, you'll know I chose the latter :-) Once Alan's issue is resolved, I'll follow up with more work on the issue this thread is about.
I did a little checking and CL does seem to localize class definitions to packages as it does for regular functions. Not too surprising. So, it does make sense for a class definition to be totally eradicated when a package is explicitly deleted.
Well, strictly speaking, it's only the *name* of the function or class which is located in a package: they're both tied to symbols in a package indeed.
There is a big difference between functions and classes though: classes are part of a hierarchical structure of classes which all store references to each other through slot values in their metaclasses. One of the properties of the graph of classes is that it's rooted in the CL package (through the classes T, STANDARD-CLASS and STANDARD-OBJECT). When a package is deleted, the class structure (not its name) will still be referenced from the "other" classes in the class-graph.
Although functions are part of a graph of cross references too, the graph doesn't go back to any packages which will never be deleted: while packages will be using the CL package, the CL package will never incorporate function calls to user-defined functions. This doesn't happen in your situation, but if you have 2 packages which cross reference each other, it could be that part of a package which is being deleted can't be cleaned yet, because it's still referenced by the "other" package.
The serious down side of not eliminating a class definition is that if a package is load, deleted, and reloaded many times, the act of defining the same class (presumably in the same package name but will actually be creating a new package structure) causes (in essence) a memory leak.
Right. That's why I started the thread. Of course if you have intimite knowledge of an implementation, you could figure out what all needs to be removed in order to disentangle the graph. I'm sure most users aren't in such a position.
Possibly, it would make sense to create a facility which allows execution of "cleanup functions" upon package deletion. Such a facility could then be used to "disentangle" the web around the class, causing the class to be collected as soon as the package is deleted and with it the symbol naming the class. That would achieve roughly what you describe above.
Regards,
Erik.
armedbear-devel@common-lisp.net
-
Blake McBride -
Erik Huelsmann