This email arises from a discussion at the ECL mailing list about how ASDF needs an urgent replacement and that is should be just discontinued and replaced with something else. Autoconf, extended REQUIREs and other stuff was suggested.
I do not agree. I think ASDF can be fixed and made easier to use.
Before explaining my view, please let me disgress a bit. I use autoconf a lot. There is no way around it if one works in Unix environments with moderately complicated software [1,2]. The need of autoconf is motivated and justified because of certain reasons
1. All build environments in Unix work the same: all C/C++ compilers are similar in the way they process their input, arguments they take and what they produce: object files. 2. Components are found in rather standard locations (/usr/include, /usr/lib, ...) and Autoconf provides information about that. 3. Components may or may not be present. Autoconf helps us determining that. 4. We normally have a simple build process with a list of files to be compiled and linked together. Dependencies are also simple forming an open graph, without loops. Automake takes this graph and produces makefiles which build the system 5. Autoconf installs your software via the generated files. We know where to place them. If the user chooses a nonstandard location he will be punished but it has a simple solution (set an environment variable).
What about lisp?
1. Each lisp behaves differently. ECL is not at all like SBCL. Currently this is solved by imposing a rather outmodish way of building the software based on compile-file and load. No notion of standalone binary files, no way of producing executables, no way of shipping things.
2. Each lisp has a different notion of how to locate "required" components. Some have search paths (SBCL), some do not (ECL). There is not a standard for installation.
3. There is an additional problem in the lisp world: we do not use the shared library loader! In order to locate components we may either rely on crappy things like REQUIRE or rely on an external agent that seeks those components prior to loading our software. ASDF behaves in that sense as a shared library loader that takes into account version information!
4. System definitions provide the description of our system and how pieces bind together. This has always been like that, even before ASDF, with mk:defsystem. It is a simple, mostly declarative description. What we do with it is up to the software that manipulates it, just like automake may evolve at some point and behave differently.
5. Installation is again a problem because there is not a well defined way to look for software. Furthermore, since current system descriptions do not provide information about "resources" that the software may need, such as bitmaps, icons, databases, etc, we have no information about installing them and instead things are left in place.
On comparing both lists one must reach the conclusion that the situation is bad. Indeed. All of the problem revolves around two things
* We need some way to build our system taking into account all dependencies * There is no standard way to produce standalone libraries and install them in a way that just works.
ASDF tries to solve BOTH problems, that is BUILDING and LOADING/EXECUTING. The problem is that development has over the time forgotten the distinction between both and one finds amazing things such as users that place some of the execution logic (creating packages, classes, etc) IN THE SYSTEM DEFINITION!
However, while we are in a position of say, disadvantage, w.r.t the Unix community at large, I believe we have a good starting point to get things right.
My vision is to *progressively* decouple the building and loading components in ASDF. I do not want to push anything down anybody's throat, but even less take all the existing database of knowledge about all the libraries that build and work with ASDF and presenting myself as an illuminated that has the solution to everything and that this solution has to be something orthogonal to ASDF.
How can we achieve these things and make everybody's life simpler? I am not going to produce the roadmap, but what I present below can be done progressively, and examples for that are the bits I have been contributing to the asdf-devel list.
* First of all understand that we have two different targets here. One is people building standalone software (programs) and the other one is people building libraries. In both cases the system definitions we have are good enough to describe them and their dependencies. Let's call the component in ASDF that handles these systems ASDF-system. This includes classes and the parsing logic. This is very tiny and should be easy to bootstrap.
* Understand how different lisps locate software. Realize that not all lisps can do that cleverly and that this is one of the reasons for ASDF to exist. Produce a minimalistic ASDF-search that reuses existing logic or implements new one for lisps that miss that part.
* Realize that we need a better way of loading software. I do not care if that guy is improving REQUIRE, he is building a new component that people will have to install or ship with their lisps and autoconfed scripts are not going to solve it.
* As a solution to the previous point realize that system definitions already have enough information for an ASDF-loader facility that builds on ASDF-search and ASDF-system. Even for standalone systems it is possible to create stripped down *.asd files with version information and dependencies in them. This is already present in the patch I sent to the list and ECL can also make it: it is the binary-op operation I introduced. The result is a file (defsystem :my-system :depends-on (:cl-ppcre) :components ((:compiled-file "my-component"))) which contains just the minimal information for our "shared library" loader to work. I personally find this better (and potentially more extensible) than Unix version numbers in file names (libecl.so.10.2.1 ?!?)
* ASDF-loader can be a very transparent component and it may even be possible to ship it together with _every_ library you have around, just like libtool is shipped with every library. As a compatibility layer over REQUIRE it should be possible to produce three files for each system - the system definition my-system.asd - a simple-minded loader my-system.fas that sets up paths, requires asdf-loader and loads the last file which is... - the component of the library itself, my-component.fas
* We still need a building facility, say ASDF-builder, that uses a system definition to compile everything. This would be the COMPILE-OP we have right now, glorified and improved, but it would be augmented with facilities to pack a library in a single file, building standalone programs, etc.
* The interesting part about this decoupling is that standalone programs do not need any of it. I can just use ASDF-builder as a glorified makefile and produce a program that just works by concatenating or linking pieces of precompiled software and inserting a simple logic at the beginning of the program that sets up the environment that the components expect. No need for ASDF-loader nor ASDF-search nor ASDF-system. Since I already removed all dependency on ASDF from the executable code itself by having logical pathnames, things just work. Compare this with a hand-built software where I abuse REQUIRE here and there.
Say you are a developer of a library, MY-LIBRARY, and want the user to just build it and install it. There are various alternatives to achieve this.
One is to continue as now. The user loads ASDF and from the prompt she is able to compile, build and install the software with existing operations.
But from the point of view of the user things can be improved a lot by means of scripting.
* Assume that ASDF is preinstalled by the user. This needs not be done in complicated fashion. ASDF itself can be "autoconf'ed" and taught to find out the existing lisps, install itself, and create a set of scripts that the application is going to use: "asdf-build" and "asdf-install".
* Alternatively you may ship a version of ASDF with your library, hidden in an asdf directory just like libtool hides itself in the m4 or config directory of a Unix library. The scripts asdf-build and asdf-install are then placed on the top of your library source tree, where the *.asd definitions live.
Now the user only has to do something like ./asdf-build --lisps=sbcl,ecl,acl ./asdf-install These two scripts will loop over lisps loading them with ASDF, compiling and installing stuff, reusing the knowledge this library has about the systems themselves. If the lisp environment does not have ASDF installed, a copy may be installed together with the library or we may use the logic explained above by which each library carries a copy of ASDF-loader just in case there is none.
All this can be coded on top of the existing ASDF, better factoring the code, and with a bits of magic here and there.
In short
* Except for standalone software, we a lisp loader that -- Takes into account dependencies -- Searches for the software at standard installation paths * ASDF may coexist with software built using other tools. * ASDF may build software for various distribution purposes. * ASDF can hide itself from the user as much as we want. * ASDF can produce software that does not rely on a configured ASDF to load * We can write libraries that do not care much whether ASDF is there or not * There is always going to be some amount of configuration on the side of the user -- Either using scripts and assuming default installation paths -- Either specifying search paths SBCL_SEARCH_PATH=... * System definitions are useful, dependencies scattered over your sources are not.
Juanjo
[1] ECL itself [2] Libraries I have to ship to clusters all around the world (AIX, Linux, SGI...) See signature below.
By and large I agree with Juanjo's statement of principles, but I would like to discuss the reason /why/ ASDF is different from tools like Autoconf.
Lisp development --- at least for people like me --- is primarily an interactive process, and this is why building and loading/executing are /inextricably/ linked.
It is, IMO, /not/ a handicap that ASDF performs both building and loading tasks, because Lisp software is not like conventional software.
In particular, this apparent duality ceases to become a source of confusion/dismay, if one stops thinking of "building" and "loading" and thinks instead about maintaining the integrity of a running lisp image.
Building and loading are just pieces of that integrity maintenance process.
I am with Juanjo in wanting to see the process of integrity maintenance become more declarative, and less cluttered with arbitrary procedural code. But I don't think that decoupling building and loading is particularly desirable in general.
For an extreme perspective on this issue, I'd encourage interested readers to look at Drew McDermott's paper on his integrity maintenance approach based on arbitrary chunks of code (a finer granularity than source files), where, for example, one can define dependencies needed to compose a hash table, so that its correctness (integrity) can be restored when data structures on which it depends are mutated:
http://cs-www.cs.yale.edu/homes/dvm/papers/lisp05.pdf
This is admittedly an extreme, but it shows a purpose that systems like ASDF can serve --- maintaining the integrity of a running lisp image over extremely long periods of time.
Autoconf and make are not the tools for people who want to be able to keep their lisp sessions running for days, weeks, or even months at a time. To do that one must integrate building and loading.
To borrow an analogy from Mikel Evins, the lisp session is a world that one creates, and which one can occupy. In this way it is radically different from building software in C, for example.
Best, r
On 2010-03-31, at 15:46 , Robert Goldman wrote:
By and large I agree with Juanjo's statement of principles, but I would like to discuss the reason /why/ ASDF is different from tools like Autoconf.
Lisp development --- at least for people like me --- is primarily an interactive process, and this is why building and loading/executing are /inextricably/ linked.
It is, IMO, /not/ a handicap that ASDF performs both building and loading tasks, because Lisp software is not like conventional software.
In particular, this apparent duality ceases to become a source of confusion/dismay, if one stops thinking of "building" and "loading" and thinks instead about maintaining the integrity of a running lisp image.
Building and loading are just pieces of that integrity maintenance process.
I am with Juanjo in wanting to see the process of integrity maintenance become more declarative, and less cluttered with arbitrary procedural code. But I don't think that decoupling building and loading is particularly desirable in general.
i am at a loss as to how it would be possible with the level of information in an asdf system definition. i admit there is this class named asdf:compile-op, but i have never made an explicit reference to it.
if the system is small enough, compilation speed is sufficiently extreme that there is no case to track internal component dependencies. for systems which find themselves with thousands of load->compile dependencies, the only hope is for progress in dvm's direction.
every production lisp can save an image. maybe there's a commercial perspective which would see value in versioned fasl components, but it i have yet to.
For an extreme perspective on this issue, I'd encourage interested readers to look at Drew McDermott's paper on his integrity maintenance approach based on arbitrary chunks of code (a finer granularity than source files), where, for example, one can define dependencies needed to compose a hash table, so that its correctness (integrity) can be restored when data structures on which it depends are mutated:
ok, but it leaves me wishing even more for the paper which really really attends to just the "what".
This is admittedly an extreme, but it shows a purpose that systems like ASDF can serve --- maintaining the integrity of a running lisp image over extremely long periods of time.
it's not extreme. it is just an unfortunately neglected direction.
On Wed, Mar 31, 2010 at 3:46 PM, Robert Goldman rpgoldman@sift.info wrote:
Lisp development --- at least for people like me --- is primarily an interactive process, and this is why building and loading/executing are /inextricably/ linked.
Please note that image driven development is NOT the only paradigm that exists in the lisp world. ECL works very well at creating standalone executables from compiled files, but it can not dump binary images of its full environment because that can not be implemented portably across all platforms we support -- specially not when one mixes shared libraries, different memory management paradigms (C, lisp, C++, operating system) and other stuff.
The problem is that people who are used to think on image driven development are those who have pushed forward also the development of ASDF as it is right now and indeed, with this mentality of an image where everything is loaded there is no distinction between placing things an the *.asd file or in the compiled files. In other words, the flexibility has propagated to the users with an absolute lack of organization in how things are done right now.
I even found out some libraries that ave problems in the dependencies, because the image driven development has side effects, such as the creation of macros, etc, which persist beyond compilation and can lead the developer to think that a load-op succeeds when it will not in a fresh new image.
What I am trying to say is that ASDF "as is" need not stop working. We may keep ASDF as a large database which you can use to power interactive development and automatic recompilation as things change. I believe this is a wonderful feature it is implicit in the long email I wrote.
However, ASDF has to seek other fields and in particlar enforce a discipline such that existing systems can be built from scratch using different paradigms -- for instance, building standalone programs from individual components.
In other words, the image-driven system can not be the model on which ASDF is built and it can not be part of its semantics because it prevents other models, but it has to remain an option for those who want to use it.
I strongly believe that both things can and should coexist.
Juanjo
On 3/31/10 Mar 31 -12:25 PM, Juan Jose Garcia-Ripoll wrote:
On Wed, Mar 31, 2010 at 3:46 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:
Lisp development --- at least for people like me --- is primarily an interactive process, and this is why building and loading/executing are /inextricably/ linked.Please note that image driven development is NOT the only paradigm that exists in the lisp world. ECL works very well at creating standalone executables from compiled files, but it can not dump binary images of its full environment because that can not be implemented portably across all platforms we support -- specially not when one mixes shared libraries, different memory management paradigms (C, lisp, C++, operating system) and other stuff.
The problem is that people who are used to think on image driven development are those who have pushed forward also the development of ASDF as it is right now and indeed, with this mentality of an image where everything is loaded there is no distinction between placing things an the *.asd file or in the compiled files. In other words, the flexibility has propagated to the users with an absolute lack of organization in how things are done right now.
I even found out some libraries that ave problems in the dependencies, because the image driven development has side effects, such as the creation of macros, etc, which persist beyond compilation and can lead the developer to think that a load-op succeeds when it will not in a fresh new image.
What I am trying to say is that ASDF "as is" need not stop working. We may keep ASDF as a large database which you can use to power interactive development and automatic recompilation as things change. I believe this is a wonderful feature it is implicit in the long email I wrote.
However, ASDF has to seek other fields and in particlar enforce a discipline such that existing systems can be built from scratch using different paradigms -- for instance, building standalone programs from individual components.
In other words, the image-driven system can not be the model on which ASDF is built and it can not be part of its semantics because it prevents other models, but it has to remain an option for those who want to use it.
I strongly believe that both things can and should coexist.
With all due respect, this seems contentious and unsupported. I don't see any particular reason to believe that a tool for the coherent maintenance of a long-running image would also be a good fit for more conventional development.
There are already tools for the non-image-driven development. If ECL is not aiming at image-driven development, maybe a tool like make, or make + autoconf would be a better fit.
That said, if you wish to try to make ASDF work for conventional build-and-exit system construction, and that doesn't break ASDF for image integrity maintenance, more power to you!
Best, R
On Wed, Mar 31, 2010 at 7:43 PM, Robert Goldman rpgoldman@sift.info wrote:
With all due respect, this seems contentious and unsupported. I don't see any particular reason to believe that a tool for the coherent maintenance of a long-running image would also be a good fit for more conventional development.
You do not see it, but there other users out there that do. I do not need to keep my image up and runnign throughout my development process and I can still achieve the same thing. I am not disputing your development model, I am disputing that anyone wishes to impose anything like that on anybody else.
There are already tools for the non-image-driven development. If ECL is not aiming at image-driven development, maybe a tool like make, or make
- autoconf would be a better fit.
I do not see why and how. Compiling lisp systems can only be done from lisp environments. This is not like you write "ecl my-file.lisp" and get something else precisely because that lisp file depends on a lot of other things.
That said, if you wish to try to make ASDF work for conventional build-and-exit system construction, and that doesn't break ASDF for image integrity maintenance, more power to you!
This is what I am claiming that it is possible to achieve, but hey this is like walking on a 1km lane covered with velcro and me having socks, friction everywhere :-) With all due respect, I can understand some people polarize against ASDF.
Juanjo
good evening;
On 2010-03-31, at 19:25 , Juan Jose Garcia-Ripoll wrote:
On Wed, Mar 31, 2010 at 3:46 PM, Robert Goldman rpgoldman@sift.info wrote: Lisp development --- at least for people like me --- is primarily an interactive process, and this is why building and loading/executing are /inextricably/ linked.
Please note that image driven development is NOT the only paradigm that exists in the lisp world. ECL works very well at creating standalone executables from compiled files, but it can not dump binary images of its full environment because that can not be implemented portably across all platforms we support -- specially not when one mixes shared libraries, different memory management paradigms (C, lisp, C++, operating system) and other stuff.
in general, "image-driven" development does not preclude a release which comprises more that one file, but if you were more specific about requirements, it would be easier understand where the problems arise for ecl.
The problem is that people who are used to think on image driven development are those who have pushed forward also the development of ASDF as it is right now and indeed, with this mentality of an image where everything is loaded there is no distinction between placing things an the *.asd file or in the compiled files.
this sentence implies either a contingent development or a logical consequence which issues from "image driven development". my experience contradicts the former and logic does not support the latter.
In other words, the flexibility has propagated to the users with an absolute lack of organization in how things are done right now.
yes, one finds among asdf users programmers who have not taken care to make worthwhile distinctions. their practice is not inherent in an "image-driven" development paradigm.
I even found out some libraries that ave problems in the dependencies, because the image driven development has side effects, such as the creation of macros, etc, which persist beyond compilation and can lead the developer to think that a load-op succeeds when it will not in a fresh new image.
then they were sloppy. even in "image-driven" development one must attend to the various environments. the distinction between and the consequences of source-only, binary-only, and mixed builds is not arcane.
What I am trying to say is that ASDF "as is" need not stop working. We may keep ASDF as a large database which you can use to power interactive development and automatic recompilation as things change. I believe this is a wonderful feature it is implicit in the long email I wrote.
However, ASDF has to seek other fields and in particlar enforce a discipline such that existing systems can be built from scratch using different paradigms -- for instance, building standalone programs from individual components.
In other words, the image-driven system can not be the model on which ASDF is built and it can not be part of its semantics because it prevents other models, but it has to remain an option for those who want to use it.
you assert a logical necessity for which i do not share your experience. you will need to argue that with more examples and more care before i can follow it.
as i have been able to comprehend the implied requirements for ecl's delivery use cases from the messages in this venue, while they do extend beyond asdf's present abilities, i have yet to have read any which are at odds with the (limited) dependency-directed mechanics at asdf's core.
yes, strictly declarative system definitions would be good. i posted a note some time back which discussed the changes to asdf which i observed to be sufficient to handle the population of .asd's in the wild with limited exceptions.
there was a concern expressed that a program be able to locate components at all stages in its life-cycle. the concern predates asdf. the mechanism to accomplish it as well. they can be integrated with asdf, used along side it, or used independently. it is demonstrably possible to do any and all.
neither of these issues is necessarily related with a "image-driven" development paradigm.
On Wed, Mar 31, 2010 at 8:10 PM, james anderson james.anderson@setf.dewrote:
in general, "image-driven" development does not preclude a release which comprises more that one file, but if you were more specific about requirements, it would be easier understand where the problems arise for ecl.
I have written a very very long email after this one. Read it. Software can be built with ASDF without needing it to run. We can deliver FASL files that can be used for people that do not use or do not have ASDF installed
there was a concern expressed that a program be able to locate components at all stages in its life-cycle. the concern predates asdf. the mechanism to accomplish it as well. they can be integrated with asdf, used along side it, or used independently. it is demonstrably possible to do any and all.
Yes, frack! Have I said that I want to replace ASDF? No! You all seem to read just words and not get the global idea.
I have written probably now 5000 words claiming that ASDF can accomodate ALL PARADIGMS. I have repeated this about 50 times now. I have provided extensions to include THOSE ways of working. I have suggested minor changes that help in the task -- logical pathnames for program delivery. I have even suggested extensions to integrate ASDF with other build processes -- asdf-install, asdf-build
And all I get is a request for more more and more writing.
neither of these issues is necessarily related with a "image-driven" development paradigm.
It is related, in the sense that you are all so focused on the load, load, load, maybe dump process that that nobody here cares a penny about other models. And this causes a lot of friction and resistance against anything that may break your workflow -- which I insist is not my goal.
This is getting so tiresome that I may just as well quit and forget.
Juanjo
-
james anderson -
Juan Jose Garcia-Ripoll -
Robert Goldman