: dherring

In my opinion, a lot of the complexity in ASDF is caused by its conflation of two separate goals: a "build system" and a "loader".

Actually, in my asdf3 article, which I invite you to review, I actually list that as an asset that ASDF has compared to C systems: by having the same component path at compile-time and runtime, you minimize a lot of the complexity that C software distribution systems have. C systems need to ensure coherency between .h files and .so files via a complex mix of technical and social rules, and umpteen different configuration managers (autoconf, gconf, kdeconfig, etc., etc.). C code needs to be installed twice, once as source, another time as libraries. [Source] https://github.com/fare/asdf3-2013/blob/master/asdf3-2014.scrbl [HTML] http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html [PDF] http://fare.tunes.org/files/tmp/asdf/asdf3-2014.pdf

This compatibility is allowed because the loader (dynamic linker in C speak, class loader in Java) has a simple, well-defined protocol for telling it where to find things, independent of the build system. Look at the linux manpages for ld.so and ldconfig...

The CL_SOURCE_REGISTRY was used by XCVB as well as ASDF, so there. But admittedly there is only one codebase: I originally wrote the code for XCVB, ebzzry wrote an initial port to ASDF, I hacked the code a lot as part of ASDF, and XCVB is reusing most of the code directly from ASDF. cl-launch also uses the registry via ASDF, and can be passed options to override it. XCVB usually produces self-contained executables; cl-launch can produce either self-contained executables or scripts that rely on the source-registry.

If CL is to grow from a dev-centric ecosystem to have a large base of "normal users", I think we need to start finding ways to enable better long-term stability. Pre-tested, "binary" releases are exactly what an end user wants. Applications need to "just work" and not break whenever a dependency changes its API.

If a user installs libraries in ~/common-lisp/ or wherever that override system libraries, he probably means it. If a programmer really wants to not have his libraries overridden by users, he can either fixate the source-registry in his script, or dump a self-contained executable. I think ASDF is already good enough in this regard.

What does this mean for ASDF?

Since many CL developers want features like load-time recompilation, the simple static decoupling in C between link and load is not possible. A richer interface is required.

While deprecated, the CL PROVIDE and REQUIRE API is very close to ideal for a loader. Obvious extensions include protocols for adding versioning checks/negotiation, search paths, re-build hooks, something better than *MODULES*, and implementation independent behavior. This API should be as small as possible, truly abstracting the different activities, and allowing multiple implementations to coexist.

They are not ideal at all; indeed, they work completely differently on each implementation (or sometimes not at all, e.g. in GCL). And there is no way to portably extend this API... and/or the closest thing there is to it is ASDF itself, which manages to hook itself into require on all seven maintained free software CL implementations.

Then tools like ASDF could produce "installs" that fit into this framework. The core loader would still be integrated into each CL distro, but the build tool, with its dependency tracking, source file searching, and other features could be distributed separately, and multiple tools could coexist in a single image.

I'm not sure what you mean that cl-launch doesn't do. Are you suggesting a shared "big image" with everything pre-loaded, and what more one that maintains a buildapp-like and extensible list of dispatched-entry main functions depending on what the image is invoked under?

Wherever possible, the simplest loader solution should be chosen, leaving sysadmins and/or build tools to handle any "difficult" configuration (such as hidden directories). For example, the loader could require that all libraries be installed using some UUID or hash-based scheme in a common location, and hooks inside these installs are used to activate fancy builder-specific features.

ASDF has an extensible *system-definition-search-functions*, courtesy of the far-seeing Dan Barlow. You can certainly adapt it some way. Or you can preload systems and use the new *immutable-systems* feature to prevent reloading.

I apologize, what I just wrote looks way too complicated. For example, it is possible that versioning, searching, rebuilding, and loading could be a builder-specific behavior inside a single try-load function. However it works out, I do believe that a simple decoupled API that breaks ASDF into a simple, stable loader core and a more featureful, faster changing piece, would be good for everyone in the long term.

Loaders have a simpler required feature set than build systems. Loaders just need to allow for a few types of search scoping (per system, per user, and per program). Build systems require configuration, searching, incremental rebuilds, installation, unit testing, etc.

I'm not sure at all what you mean, and what that would look like. ASDF is pretty minimal within constraints — certainly, the mission creep has certainly led to a tenfold size increase in size since the Dan Barlow days (only fivefold and a half since I took over), but I'll argue that extending the mission was justified. Being able to write a #!/usr/bin/cl script that doesn't have to be modified specially depending on the implementation used and local installation paths — that's worth it. Oh, and not having any of the many bugs in ASDF1. That said, if you want to release a "load-only" version of ASDF that just hooks into require, only does the loading part assuming fasl-op output, go ahead. That might be interesting. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org God is Dead — Nietzsche Nietzsche is Dead — God Nietzsche is God — The Dead

On Mar 13, 2014, at 15:42, Robert P. Goldman <rpgoldman@sift.info> wrote: […]

I believe Faré and I discussed this issue in our first ASDF paper at ILC. For an even more radical proposal, see McDermott's earlier ILC paper about his chunking mechanism, which allows programmers to specify integrity constraints even for individual data structures.

Suggestion: please put URI de-referencable bibliographic resources like this in the ASDF documentation. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."