Hello all,
since my merge of the less-reflection branch with trunk[1], several of you have been bitten by the new limitation in the number of functions that can be contained in a FASL. The reflection-less scheme requires creating a loader class because it needs to spell out in the bytecode all the class names corresponding to compiled Lisp functions in the FASL. Functions are numbered from 1 onwards and a method is created that is a huge switch (actually a chain of if's) that returns the right function given its index:
if(index >= 1 && index < 1024) { if(index == 1) return new function_1(); if(index == 2) return new function_2(); ... } else if(index >= 1024 ...
In the presence of many functions, this method grows very large, eventually surpassing the JVM's 64k method size limit.
A couple of days ago Erik and I discussed a bit about the issue and how to solve it. One option is to implement method splitting for the big FASL loader method. Method splitting in general is hard (you'd need some form of CPS conversion) but in this case it's quite easy. The current scheme, though, has drawbacks: it's more complicated than it used to be, and it's not that beneficial performance-wise - from a profiling I did a while ago, its performance gains amounted to ~1% of our loading time, which is completely dominated by I/O and UTF-8 decoding. So, there's another option: go back to reflection-based loading. Using reflection has the benefit of being simpler. On certain platforms (e.g. mobile JVMs or similar, like Dalvik), reflection has a higher cost, but we currently don't run satisfactorily on those platforms anyway and we'd require multiple changes to do so. After my discussion with Erik, a further refinement of this second option came to my mind. As part of the no-reflection effort, the compiler was changed to stop emitting calls to loadCompiledFunction(className) in favor of generating `new className()` bytecode for local functions, in accordance with the goal of avoiding reflection as much as possible. A special classloader is used to both resolve those local function references, and to load the FASL loader class (and thus transitively all top-level functions). I propose keeping the compilation of local function references as it is, and thus the special classloader; and use it directly (via reflection) to load top-level functions, instead of generating one big loader method. This would result in slightly less reflection, but more importantly in a clearer function loading scheme, and less development effort compared to returning completely to the reflection-based scheme also for local functions.
What do you think? If it's not clear, both Erik and I sponsor the solution of returning to a reflection-based approach, but we'd like to hear everyone's opinions.
Bye, Alessio