Hello,
as I have already informally discussed with Erik and David Miles, I have been working on a branch which changes how functions are compiled in FASLs (and by consequence how they are loaded from FASLs). The purpose was to reduce the use of reflection by using a single classloader per FASL (as opposed to one per function) plus a master "loader" function that will initialize all other function classes using the new operator, rather than passing through reflection. Now such loading is done eagerly, as soon as the FASL is loaded; but with a bit more work, I should be able to restore autoloading semantics, so that a given function object is only instantiated the first time it is called (but the class representing it will always be loaded eagerly). I suspect it won't change much, though.
The overall performance results (reported below) are not very encouraging. However, I do think that the branch has a better design than our current approach, and goes towards the objective of having a per-fasl object holding various things (e.g. a FASL constant pool). In the long run, not having functions rely on loadCompiledFunction in their static initialization code is a nice step towards serializable closures, if we'll ever want to have them.
The results on load times and overall runtime speed have been insignificant. Apparently by large the most important factor in our load time is I/O and the branch doesn't change anything about that. It just saves ~1% of the loading time spent on reflection.
The results on memory consumption are mixed: they do show a great decrease in the number of classloaders, as expected, but the numbers are not tremendously life-changing: 56 objects (live and allocated, for a total of 4032B, 0.1% of total live objects) vs 789 live objects of 1014 allocated, 44.184B, 1.1% of total live objects. There are also less reflection-related objects, but more Lisp objects, presumably due to eager loading. The numbers for those from a quick glance more or less balance out with each other.
Doing these profiles showed that a lot of our memory is actually made of the stored byte[] containing class data for compiled functions. Those amount to ~4M on trunk and almost 8M on my branch (probably due the eager loading of compiled functions). Given that I now load all functions with a classloader that knows where functions have been loaded from, I think I should not store the byte[] in the function at all, but rely on the fasl classloader to re-extract them from the FASL on demand. However, I haven't still implemented this.
For those who are interested, I'm attaching the memory profiles done with NetBeans. I left out the CPU profiles because they're fatter (~6M) and I'm on a slow connection now; but I can send them separately.
Cheers, Alessio