The other day, Ville and I were discussing the issue with recovery of "special slots" after package deletion on IRC. Let me start by describing the problem for you first. After that I'll describe the solution I see, however, your feedback on the solution is very much appreciated.
Special symbol slots? ABCL - like SBCL and CCL - uses a thread-local array containing the 'current' values of its special variables. All symbols which have been declared or declaimed special at one point or another have been assigned an index in this array. This method allows very quick access to the special value (and hence the symbol value) with only a single level of added indirection: the symbols hold the number of their index in the array, meaning that having a reference to the symbol and the current thread is enough to find its value. When I implemented it, it was a huge difference with respect to the method we were using before.
Do those need release? Since packages can be deleted, symbols be garbage collected. In the current implementation, GC-ing a special symbol will leave the slot it was assigned unused. There's no way the slot gets reassigned, because nothing keeps track of these 'dead' slots. To make matters worse: there may be huge lisp structures referenced in the dead slot, because it's also not cleared out. In the very long run, this will cause issues in scenarios as Blake describes where one creates, deletes and re-creates packages. The symbols in the packages are new symbols on every re-creation. Yet there are only 4096 spots in the special-symbols table.
The solution? Well, the solution I see is to use the Object.finalize() method to release the index back to a freelist. There's a downside here though: the 4096 slots may run out before a multi-gb system sees its first GC run. There are two ways out here: grow the table once we run out of free slots (even though many potentially free slots may be pending GC). Or force a gc/runFinalization() run when running out of indices. Surely a combination of both would also be an option.
Ville partially objects to the above because he's used to the pattern "you can't depend on finalize() to be invoked". Reading its documentation and googling the topic a bit, I see that finalize() can't be depended upon especially on JVM termination. However, that particular case isn't interesting here: we're looking at releasing "memory" related resources, which seems to be the exact purpose of finalize() as described in all the web docs I found.
Can you comment? Is my approach as sane as I think it is?
Bye,
Erik.
On Mon, Jan 10, 2011 at 11:38 PM, Erik Huelsmann ehuels@gmail.com wrote:
The other day, Ville and I were discussing the issue with recovery of "special slots" after package deletion on IRC. Let me start by describing the problem for you first. After that I'll describe the solution I see, however, your feedback on the solution is very much appreciated.
[snip very thorough explanation]
Can you comment? Is my approach as sane as I think it is?
I have no objections on the use of finalize(). The common wisdom is that finalize() should be avoided, but that's for ordinary code, where indeed finalize() is most of the time the wrong answer. Our situation sounds like a typical use case for finalize(). However, I have a concern. I didn't research this at all - I should really be already in bed sleeping now - but I fear the consequences of generational garbage collection. Symbols are typically long-lived objects and likely to end up in the "elder" generation. I don't think a System.gc() is guaranteed to clean that up, so using it as a mean to obtain new indexes when all 4096 have been allocated might not work. Extending the table will do the trick.
Bye, Alessio
On Mon, Jan 10, 2011 at 5:38 PM, Erik Huelsmann ehuels@gmail.com wrote:
Since packages can be deleted, symbols be garbage collected. In the current implementation, GC-ing a special symbol will leave the slot it was assigned unused. There's no way the slot gets reassigned, because nothing keeps track of these 'dead' slots. To make matters worse: there may be huge lisp structures referenced in the dead slot, because it's also not cleared out.
Would it not make sense to have delete-package at least clear out the relevant values in the array. -Alan
On Tue, Jan 11, 2011 at 3:21 PM, Alan Ruttenberg alanruttenberg@gmail.com wrote:
On Mon, Jan 10, 2011 at 5:38 PM, Erik Huelsmann ehuels@gmail.com wrote:
Since packages can be deleted, symbols be garbage collected. In the current implementation, GC-ing a special symbol will leave the slot it was assigned unused. There's no way the slot gets reassigned, because nothing keeps track of these 'dead' slots. To make matters worse: there may be huge lisp structures referenced in the dead slot, because it's also not cleared out.
Would it not make sense to have delete-package at least clear out the relevant values in the array.
No, because symbols are first-class objects, so a given symbol can continue to be referenced even after its home package has been deleted.
Bye, Alessio
armedbear-devel@common-lisp.net
-
Alan Ruttenberg -
Alessio Stalla -
Erik Huelsmann