Hi all,
Sorry for the bumpy ride while I reorganized the auto loader system.
Thanks for taking the time to respond to my earlier questions on auto loaders. This is the time to explain what I did with your answers (and what I did in general).
To reiterate what this is all about: the autoloader is an important part of the start up efficiency and internal file dependency resolution of ABCL: it establishes function and macro bindings on symbols without loading the actual code to bind to. Instead, loading the FASL is being delayed until some part of the system (or a user) needs it. There are two variants of autoloaders: Java class autoloaders and lisp (fasl) autoloaders. Their purpose is the same and they operate roughly the same way, but one operates by forcing a class to be loaded while the other causes a FASL to be loaded.
Most parts of the system don't know about autoloaders and simply evaluate code. The autoloaders take special action when evaluated: their purpose is not to execute the intended action, but instead, they load the fasl in which the actual code is located. Then, when the actual code is loaded, the function (or macro) call is forwarded to the loaded code. The FASL loader process replaces the content of the function symbol slot with the true function so the autoloader gets garbage collected after replacement.
This entire system works pretty well, that is: *if* you know which symbols you have to install an autoloader on. Before my changes, we didn't know which symbols we had to install autoloaders on. Instead, we maintained a list of symbols by hand through trial and error. When a change caused total havoc, you'd look for the cause and - hopefully - debugged the problem to a missing autoloader. That's when you would add an autoloader to autoloads.lisp. That works as long as our only purpose is to build ABCL. However, we actually intend for ABCL to be used by our users.
Users may not be aware of what autoloaders do for them and choose to start calling some random function in the AMOP definition. Since we hand-maintain a list of functions to be decorated with autoloaders, that probably doesn't work. So, in order to provide more stability for our users we need to decorate all functions with autoloaders.
What's more: having all functions decorated with autoloaders -if done well- should also help ABCL development quite a bit: the chances of a change causing havoc because a non-decorated function is being used have been reduced to next-to-none.
What then did I do?
The change involves a few steps:
1. Extraction of symbols to be decorated =============================
The file compiler now generates (when compiling our own system, that is) two extra output files: basename.funcs and basename.macs. These files contain a single s-exp with all the symbols which have been DEFUNed or DEFMACROed respectively. With this information, we can generate decoration information.
Analysis of the extracted symbols leads to the conclusion we can't assign an autoloader to each and every extracted symbol for 2 reasons: a. There is a built-in function associated with the symbol to help bootstrapping b. There are multiple DEFUNs or DEFGENERIC/DEFMETHODs
2. Determination of what to decorate symbols with ====================================
Autoloaders - at this time - can only be used to load a single FASL. So, when a symbol is in multiple files, we need to choose which file it load the FASL for (and hence not the others).
Most symbols which occur in multiple files are DEFMETHODs which is perfectly fine. Most symbols with double locations stem from the EXTENSIBLE-SEQUENCES implementation. We want to exclude those FASLs, which reduces the number of multi-homed symbols to only a few.
All single-homed symbols excluding those which have bootstrap functions attached can be decorated with autoloaders.
3. Generate code to install the autoloaders ===============================
We just used to have autoloads.lisp (and Autoload.java). But that file will contain only the manual bits which can't be auto-generated. So there's now a second file autoload-gen.lisp. This file is being generated during the build and populated with all symbols which need an autoloader. Today's autoload.lisp has been stripped to the bare minimum of AUTOLOAD and AUTOLOAD-MACRO forms.
With the stripped autoload.lisp and the fact that the autoloaders are bootstrapping helpers, I had to take additional measures to help us bootstrap again. The result is autoloads-gen.lisp in the source directory. It is loaded at build time to help bootstrapping. It's an unmodified version I took from the build directory: build/classes/org/armedbear/lisp/autoloads-gen.lisp - the one which is generated on each build.
4. Exporting the right symbols ======================
Not a problem related to autoloading per se, but definitely related to the fact that we don't load all fasls at boot time: autoload.lisp - as the canonical location being loaded at startup - had grown to be a listing of EXPORT forms. Having EXPORT forms and actual code widely apart is counter-intuitive: to the occasional developer it's inexplicable how some symbols become exported and some don't. (We had files with additional EXPORT forms.)
So, when the above was "solved", I created another facility which collects all the symbols in the second argument of EXPORT throughout our source tree. Those symbols are added to the autoload-gen file and the EXPORT forms executed at start up as well. Even though the developer still doesn't know what's going on, the system should be working transparently.
All of the above has happened over the last week(s) - last week, mostly.
Then there is the question:
As it turns out, the macroexpansion for DEFSTRUCT happens in a way such that the file compiler can't detect the slot accessors as being defined as functions. The advantage is that we're *not* seeing a whole slew of symbols to be attached to autoloaders and hence limiting the number of autoloaders (the contructor function *is* detected).
Now: what do others think: should we depend on the constructor being called before any of the accessors to make sure the autoloaders have been run? Or should we attach an autoloader to each accessor function just to be sure? (This requires small changes to the file compiler and defstruct macro expansion.)
Bye,
Erik.