- Because lisp functions have become Java functions, we can't
directly expose them to the Lisp world anymore. This basically creates the option to have an 'internal function signature' and an 'external function signature'. SBCL has this: XEPs -- eXternal Entry Points. These entry points into a function sort the arguments into the right order before calling the internal entry point. Code which is compiled into the same fasl pre-sorts the arguments at compile time (when possible) and calls the internal entry point -- eliminating the need to sort keyword parameters.
That's awesome. To clarify, do you intend these XEPs to be method overloads in the same FASL-class?
Yes: the XEPs would be methods accepting an array argument, just like we accept now. The XEP would use the Closure.processArgs() code to figure out the optional and keyword arguments, just like today. Unlike today, the XEP would then forward the call unpacking the argument array into individual parameters and call the internal entry point.
[...]
c. We need to find a way to correctly handle the interaction between the successive IN-PACKAGE, DEFPACKAGE, EVAL-WHEN, etc, forms appearing in the input file and the initialization of fields in the resulting class file.
Off the top of my head (but the standard might imply otherwise) the only problem is IN-PACKAGE (and DEFPACKAGE) and that can be solved by
- adding a static initializer to the fasl-class that pre-installs all
the packages and 2) serializing all symbols explicitly as package::symbol. Everything else (mostly EVAL-WHEN combined with stuff that affects the reader) is handled by isolating the reader used to parse serialized stuff in the class from the reader used to read forms in the fasl, and I believe it's already like that - isn't it?
Yes, the reader fasl reader is indeed separated from the compile-time reader. One thing that got us in the past when serializing symbols was EXPORT, which was first evaluated and then serialized. When reading the resulting fasl, the export command tries to export a non-existing symbol.
However, serializing all symbols with their package prefix breaks a use case I know of to be actually in use - which may not be explicitly supported by the CLHS: when loading our current fasls while having a different current package than during compilation, we simply load everything into the new current package. This is because the preconditions before loading are assumed to be the same as the preconditions before compilation.
What I'm thinking now is that I might try to compile the top level forms into an initialization function in the class. That initialization function could interleave class field initialization with "lisp world initialization". That way, I think, we should be able to get our dependencies ordered correctly.
d. We need a way to expose the external entry points to the lisp world.
We could lazily construct (by generating bytecode at runtime) a LispFunction instance that calls the right method. It would be generated the first time someone calls, or otherwise references, a function in the fasl, and then associated with the symbol as its symbol-function. Hopefully these runtime-generated classes will be in much smaller numbers than if we compiled each and every function to a separate class. Additionally with invokedynamic the LispFunction will only need to be generated when the function is reified (basically the first time someone directly reads a symbol's function slot), while regular function calls could be directed to the target method.
This is one method. The other - which might have roughly equal performance characteristics - that I was thinking about is to create a class which uses introspection to look up the right method to call. The initial introspection lookup has bad performance drawbacks. However, after the initial lookup, I've been told, the performance drawbacks have been fixed for quite some time (Java 1.4?).
That way, we would not even need to increase the number of classes in the system when the number of entry points into the fasl grows. The same approach could apply when a function reference is to be returned: the function reference could simply be an encapsulating function object.
Thanks for your comments!
Bye,
Erik.