Hi all,
This has bugged me for years, and recent traffic brings it back to the forefront. I apologize that this email is more a teaser than a complete solution, but that's what time allows for right now.
In my opinion, a lot of the complexity in ASDF is caused by its conflation of two separate goals: a "build system" and a "loader".
There are a wide variety of build tools available for C/C++ and many other languages: make, imake, cmake, (s)cons, (b)jam, ninja, redo, etc. While the added overhead of installing a build system before compiling a project has drawbacks, many of these tools have compatibility with make, and most users don't compile most code. The compatibility of files produced by these diverse tools allows a lot of experiments to be run, and successful features often get absorbed into tools like GNU make.
This compatibility is allowed because the loader (dynamic linker in C speak, class loader in Java) has a simple, well-defined protocol for telling it where to find things, independent of the build system. Look at the linux manpages for ld.so and ldconfig...
If CL is to grow from a dev-centric ecosystem to have a large base of "normal users", I think we need to start finding ways to enable better long-term stability. Pre-tested, "binary" releases are exactly what an end user wants. Applications need to "just work" and not break whenever a dependency changes its API.
What does this mean for ASDF?
Since many CL developers want features like load-time recompilation, the simple static decoupling in C between link and load is not possible. A richer interface is required.
While deprecated, the CL PROVIDE and REQUIRE API is very close to ideal for a loader. Obvious extensions include protocols for adding versioning checks/negotiation, search paths, re-build hooks, something better than *MODULES*, and implementation independent behavior. This API should be as small as possible, truly abstracting the different activities, and allowing multiple implementations to coexist.
Then tools like ASDF could produce "installs" that fit into this framework. The core loader would still be integrated into each CL distro, but the build tool, with its dependency tracking, source file searching, and other features could be distributed separately, and multiple tools could coexist in a single image.
Wherever possible, the simplest loader solution should be chosen, leaving sysadmins and/or build tools to handle any "difficult" configuration (such as hidden directories). For example, the loader could require that all libraries be installed using some UUID or hash-based scheme in a common location, and hooks inside these installs are used to activate fancy builder-specific features.
I apologize, what I just wrote looks way too complicated. For example, it is possible that versioning, searching, rebuilding, and loading could be a builder-specific behavior inside a single try-load function. However it works out, I do believe that a simple decoupled API that breaks ASDF into a simple, stable loader core and a more featureful, faster changing piece, would be good for everyone in the long term.
Loaders have a simpler required feature set than build systems. Loaders just need to allow for a few types of search scoping (per system, per user, and per program). Build systems require configuration, searching, incremental rebuilds, installation, unit testing, etc.
Later, Daniel