This email arises from a discussion at the ECL mailing list about how ASDF needs an urgent replacement and that is should be just discontinued and replaced with something else. Autoconf, extended REQUIREs and other stuff was suggested.
I do not agree. I think ASDF can be fixed and made easier to use.
Before explaining my view, please let me disgress a bit. I use autoconf a lot. There is no way around it if one works in Unix environments with moderately complicated software [1,2]. The need of autoconf is motivated and justified because of certain reasons
1. All build environments in Unix work the same: all C/C++ compilers are similar in the way they process their input, arguments they take and what they produce: object files. 2. Components are found in rather standard locations (/usr/include, /usr/lib, ...) and Autoconf provides information about that. 3. Components may or may not be present. Autoconf helps us determining that. 4. We normally have a simple build process with a list of files to be compiled and linked together. Dependencies are also simple forming an open graph, without loops. Automake takes this graph and produces makefiles which build the system 5. Autoconf installs your software via the generated files. We know where to place them. If the user chooses a nonstandard location he will be punished but it has a simple solution (set an environment variable).
What about lisp?
1. Each lisp behaves differently. ECL is not at all like SBCL. Currently this is solved by imposing a rather outmodish way of building the software based on compile-file and load. No notion of standalone binary files, no way of producing executables, no way of shipping things.
2. Each lisp has a different notion of how to locate "required" components. Some have search paths (SBCL), some do not (ECL). There is not a standard for installation.
3. There is an additional problem in the lisp world: we do not use the shared library loader! In order to locate components we may either rely on crappy things like REQUIRE or rely on an external agent that seeks those components prior to loading our software. ASDF behaves in that sense as a shared library loader that takes into account version information!
4. System definitions provide the description of our system and how pieces bind together. This has always been like that, even before ASDF, with mk:defsystem. It is a simple, mostly declarative description. What we do with it is up to the software that manipulates it, just like automake may evolve at some point and behave differently.
5. Installation is again a problem because there is not a well defined way to look for software. Furthermore, since current system descriptions do not provide information about "resources" that the software may need, such as bitmaps, icons, databases, etc, we have no information about installing them and instead things are left in place.
On comparing both lists one must reach the conclusion that the situation is bad. Indeed. All of the problem revolves around two things
* We need some way to build our system taking into account all dependencies * There is no standard way to produce standalone libraries and install them in a way that just works.
ASDF tries to solve BOTH problems, that is BUILDING and LOADING/EXECUTING. The problem is that development has over the time forgotten the distinction between both and one finds amazing things such as users that place some of the execution logic (creating packages, classes, etc) IN THE SYSTEM DEFINITION!
However, while we are in a position of say, disadvantage, w.r.t the Unix community at large, I believe we have a good starting point to get things right.
My vision is to *progressively* decouple the building and loading components in ASDF. I do not want to push anything down anybody's throat, but even less take all the existing database of knowledge about all the libraries that build and work with ASDF and presenting myself as an illuminated that has the solution to everything and that this solution has to be something orthogonal to ASDF.
How can we achieve these things and make everybody's life simpler? I am not going to produce the roadmap, but what I present below can be done progressively, and examples for that are the bits I have been contributing to the asdf-devel list.
* First of all understand that we have two different targets here. One is people building standalone software (programs) and the other one is people building libraries. In both cases the system definitions we have are good enough to describe them and their dependencies. Let's call the component in ASDF that handles these systems ASDF-system. This includes classes and the parsing logic. This is very tiny and should be easy to bootstrap.
* Understand how different lisps locate software. Realize that not all lisps can do that cleverly and that this is one of the reasons for ASDF to exist. Produce a minimalistic ASDF-search that reuses existing logic or implements new one for lisps that miss that part.
* Realize that we need a better way of loading software. I do not care if that guy is improving REQUIRE, he is building a new component that people will have to install or ship with their lisps and autoconfed scripts are not going to solve it.
* As a solution to the previous point realize that system definitions already have enough information for an ASDF-loader facility that builds on ASDF-search and ASDF-system. Even for standalone systems it is possible to create stripped down *.asd files with version information and dependencies in them. This is already present in the patch I sent to the list and ECL can also make it: it is the binary-op operation I introduced. The result is a file (defsystem :my-system :depends-on (:cl-ppcre) :components ((:compiled-file "my-component"))) which contains just the minimal information for our "shared library" loader to work. I personally find this better (and potentially more extensible) than Unix version numbers in file names (libecl.so.10.2.1 ?!?)
* ASDF-loader can be a very transparent component and it may even be possible to ship it together with _every_ library you have around, just like libtool is shipped with every library. As a compatibility layer over REQUIRE it should be possible to produce three files for each system - the system definition my-system.asd - a simple-minded loader my-system.fas that sets up paths, requires asdf-loader and loads the last file which is... - the component of the library itself, my-component.fas
* We still need a building facility, say ASDF-builder, that uses a system definition to compile everything. This would be the COMPILE-OP we have right now, glorified and improved, but it would be augmented with facilities to pack a library in a single file, building standalone programs, etc.
* The interesting part about this decoupling is that standalone programs do not need any of it. I can just use ASDF-builder as a glorified makefile and produce a program that just works by concatenating or linking pieces of precompiled software and inserting a simple logic at the beginning of the program that sets up the environment that the components expect. No need for ASDF-loader nor ASDF-search nor ASDF-system. Since I already removed all dependency on ASDF from the executable code itself by having logical pathnames, things just work. Compare this with a hand-built software where I abuse REQUIRE here and there.
Say you are a developer of a library, MY-LIBRARY, and want the user to just build it and install it. There are various alternatives to achieve this.
One is to continue as now. The user loads ASDF and from the prompt she is able to compile, build and install the software with existing operations.
But from the point of view of the user things can be improved a lot by means of scripting.
* Assume that ASDF is preinstalled by the user. This needs not be done in complicated fashion. ASDF itself can be "autoconf'ed" and taught to find out the existing lisps, install itself, and create a set of scripts that the application is going to use: "asdf-build" and "asdf-install".
* Alternatively you may ship a version of ASDF with your library, hidden in an asdf directory just like libtool hides itself in the m4 or config directory of a Unix library. The scripts asdf-build and asdf-install are then placed on the top of your library source tree, where the *.asd definitions live.
Now the user only has to do something like ./asdf-build --lisps=sbcl,ecl,acl ./asdf-install These two scripts will loop over lisps loading them with ASDF, compiling and installing stuff, reusing the knowledge this library has about the systems themselves. If the lisp environment does not have ASDF installed, a copy may be installed together with the library or we may use the logic explained above by which each library carries a copy of ASDF-loader just in case there is none.
All this can be coded on top of the existing ASDF, better factoring the code, and with a bits of magic here and there.
In short
* Except for standalone software, we a lisp loader that -- Takes into account dependencies -- Searches for the software at standard installation paths * ASDF may coexist with software built using other tools. * ASDF may build software for various distribution purposes. * ASDF can hide itself from the user as much as we want. * ASDF can produce software that does not rely on a configured ASDF to load * We can write libraries that do not care much whether ASDF is there or not * There is always going to be some amount of configuration on the side of the user -- Either using scripts and assuming default installation paths -- Either specifying search paths SBCL_SEARCH_PATH=... * System definitions are useful, dependencies scattered over your sources are not.
Juanjo
[1] ECL itself [2] Libraries I have to ship to clusters all around the world (AIX, Linux, SGI...) See signature below.