March 2010 - asdf-devel - mailman3.common-lisp.net

[asdf-devel] :logical-hostname
by Juan Jose Garcia-Ripoll 10 Apr '10

10 Apr '10

One of the common problems that ASDF users faced when introducing the new binary location system is finding out where things reside. Another problem we will eventually find is locating the resources associated to an ASDF system which has been replaced by a precompiled version. In general it would be desirable for a system to be able to locate its resources (additional files, icons, bitmaps, etc) using logical pathnames. For that purpose I have coded a small patch that adds a new keyword, :logical-host, to DEFSYSTEM. The keyword defines a logical pathname translation from "**;*.*.*" to "path/to/asdf/or/pathname/argument/**/*.*" This way we may have (defsystem :cl-unicode :logical-host "cl-unicode" ...) and in the sources load the databases such as "cl-unicode:uppercase-table.dat" https://bugs.launchpad.net/asdf/+bug/551222 -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

4 33

[asdf-devel] Patches for ABCL against asdf-1.641
by Mark Evenson 05 Apr '10

05 Apr '10

Attached are patches in 'abcl-asdf.patch' to get asdf-1.641 to work against ABCL for the run-tests.sh. Unfortunately, you will need to build [ABCL from trunk][1] using at least svn r12550, because I had to patch ABCL to work with ASDF. And you'll need to apply the 'abcl-translate-pathname.patch'. [1]: svn://common-lisp.net/project/armedbear/svn/trunk/abcl Notes: 1. The fugliness of the conditional around ASDF:GET-UID works around a bug in the ABCL compiler. We are in the process of [analyzing the error][2]. [2]: http://trac.common-lisp.net/armedbear/ticket/89 2. There is some undiagnosed problem in translating the binary location for the new ASDF2 "~/.cache" conventions, as ABCL seems to "collapses" everything into a single directory. Somehow, the default *output-translations* have a "/**/**/*.*" where there should be "/**/*.*". I'm not sure if this is a problem in "abcl-translate-pathname.patch", which is why I didn't apply this to ABCL trunk. 3. ASDF's run-tests.sh seems to ignore the "flags" setting, which seems to be broken on the ASDF side. 4. The changes for ASDF look quite interesting. However I would advocate that you should make ASDF2 work *exactly* like ASDF1 out of the box, meaning you shouldn't do any of the subsumed ASDF-BINARY-LOCATIONS stuff without being asked to by configuration. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."

6 26

[asdf-devel] ASDF 2 issues
by Faré 01 Apr '10

01 Apr '10

There are many things to clarify about logical pathnames and the use thereof or not, but I don't think this should block a release of ASDF 2. Should it? And so I'd like to declare it an ASDF 2.1 or ASDF 3 issue, and invite you to focus on blocking issues for ASDF 2. (Of course, if someone has a working, portable, solution that makes everyone happy, I'll gladly merge and commit it.) I think the asdf-bundle extension is more important, because underlying it is the structure of how we deliver extensions, and how we play well with ECL, especially as it might entail some slight backwards incompatibility for ECL that I'd rather not happen *after* we release ASDF 2. Can we focus on getting ASDF 2 out before we add new features? [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Superstition brings bad luck. — Paul Carvel On 31 March 2010 14:00, Juan Jose Garcia-Ripoll <juanjose.garciaripoll(a)googlemail.com> wrote: > On Wed, Mar 31, 2010 at 4:39 PM, james anderson <james.anderson(a)setf.de> > wrote: >> >> i simply don't understand your requirements. on one hand both with and >> without asdf, but then it is objected, that a mechanism which requires asdf >> does not work without it. please explain. > > What I do not understand is what YOU do not understand. > > In an standalone system there is no ASDF. That is all. > > I want to be able to take an ASDF system describing my program and its > dependencies on other libraries and compile it and produce a standalone > executable. That program will not have ASDF inside, it will not have the > notion of a central registry it will have nothing, but it still has to work. > > Please understand that the program I am talking about is not a dumped lisp > image that contains everything needed to build the system and even some > extra stuff that was there (database of ASDF systems and other trash). The > programs I produce are just a set of compiled files, one per ASDF system, > linked together with a core library that is the Common Lisp environment ECL > offers. > > When the program starts, the linked files are "loaded" one after another, as > if LOAD was used but actually much faster. But I insist again ASDF is not > part of the things that is loaded simply because there is no need. > > The only thing that one might need ASDF for is to locate the data associated > to the program, but if we provide logical pathname translations (one host > per library), then a small piece of code can be inserted at the beginning of > the program which locates the data and sets up the appropriate translations. > >> if the requirement is that exact same exact formulation be used with and >> without asdf, then >> a) the additional system initialization argument is excluded >> b) the second option in my earlier message provides this path. > > > I only find "b" acceptable because I still do not understand how "a" is > going to work without a central registry and a lot of complicated logic that > has to propagate to those distributed binaries. > > Having "b", which I recall was one logical host name per system, one may > simply prepend to the compiled set of files a tiny bit of lisp that looks as > follows > > (defvar *program-data-location* > #+windows ;; Data and binaries live together > (program-location) > #+unix ;; Data is installed somewhere standard > "/usr/share/program/" > #+osx ;; Assume this is an application bundle > (merge-pathnames "Content/Resources/" (program-location))) > > (defun set-host-translation (root extra) > ...) > > (set-host-translation "cl-ppcre" "cl-ppcre/") > (set-host-translation "cl-unicode" cl-unicode/") > > With this simple, automatically generated code I can pack the given systems > and let them work anywhere, even though ASDF is not present. > > The same or a similar procedure can be used to link many systems into a > single distributable binary fasl + resources, install it in a standard > location that can be found by certain lisps and prepend a simple logic for > finding out its data... Without using ASDF in the file itself. > > There is even a third model in which installation paths are known > beforehand. Say that a package maintainer from Debian has the role of > generating binaries for 10 libraries out there and they have complex > inter-dependencies. Libraries can be compiled in the machine of the > maintainer and then installed by ASDF in their standard locations. ASDF can, > in the process, hard-code the pathname translations when installing the > files. > > For instance, if we produce binaries for cl-ppcre-1.0.1 and cl-unicode-2.0.1 > and the latter depends on the former, and we know that they are going to > live in > /usr/lib/common-lisp/cl-ppcre-1.0.1 > /usr/lib/common-lisp/cl-unicode-2.0.1 > when "asdf-build" compiles the files and "asdf-installs" them we may get > > ;; Stripped down version of a system definition > ;; which only contains the precompiled files > ;; and which can be searched and used by ASDF > /usr/lib/common-lisp/cl-ppcre-1.0.1/cl-ppcre.asd > /usr/lib/common-lisp/cl-unicode-1.0.1/cl-unicode.asd > > ;; The binaries themselves > /usr/lib/common-lisp/cl-ppcre-1.0.1/cl-ppcre-1.0.1.fas > /usr/lib/common-lisp/cl-ppcre-1.0.1/cl-unicode-1.0.1.fas > > ;; Proxy binaries that can be loaded using "require" > ;; even if ASDF is not present. These files define the logical > ;; pathname translations based on hardcoded locations. > /user/lib/common-lisp/cl-ppcre-1.0.1/cl-ppcre.fas > /user/lib/common-lisp/cl-ppcre-1.0.1/cl-unicode.fas > > ;; Data used by the library > /user/lib/common-lisp/cl-ppcre-1.0.1/cl-unicode.dat > > ;; Symbolic link to the last version > /usr/lib/common-lisp/cl-ppcre -> cl-ppcre-1.0.1 > > The FASL without version numbers files can explicitly load the required > versions of the libraries because they know the hardcoded paths of their > files. In doing so they can also set up the appropriate pathname > translations that are used by the other files. > > I have listed just two examples with different paradigms on how to build and > distribute software, and which go beyond current use in which the user has > to set up a collection of registries, in your case you even define pathname > translations and have magic to make ASDF understand them, sources are always > present, things are recompiled... > > But there are other possibilities. ASDF can be used to produce programs and > libraries that can be distributed, ASDF can itself be used without people > noticing it (see the automatically generated FASL files), it can integrate > with other build processes (via some simple scripts asdf-build, asdf-install > which I may offer to write), we can achieve better system descriptions, and > a lot of things that make people's life easier because they can forget that > they use ASDF at all -- just like when writing C I am free to forget that I > am using "make" at some point. > > Juanjo

2 2

[asdf-devel] The future of ASDF (was logical pathnames)
by Juan Jose Garcia-Ripoll 31 Mar '10

31 Mar '10

This email arises from a discussion at the ECL mailing list about how ASDF needs an urgent replacement and that is should be just discontinued and replaced with something else. Autoconf, extended REQUIREs and other stuff was suggested. I do not agree. I think ASDF can be fixed and made easier to use. Before explaining my view, please let me disgress a bit. I use autoconf a lot. There is no way around it if one works in Unix environments with moderately complicated software [1,2]. The need of autoconf is motivated and justified because of certain reasons 1. All build environments in Unix work the same: all C/C++ compilers are similar in the way they process their input, arguments they take and what they produce: object files. 2. Components are found in rather standard locations (/usr/include, /usr/lib, ...) and Autoconf provides information about that. 3. Components may or may not be present. Autoconf helps us determining that. 4. We normally have a simple build process with a list of files to be compiled and linked together. Dependencies are also simple forming an open graph, without loops. Automake takes this graph and produces makefiles which build the system 5. Autoconf installs your software via the generated files. We know where to place them. If the user chooses a nonstandard location he will be punished but it has a simple solution (set an environment variable). What about lisp? 1. Each lisp behaves differently. ECL is not at all like SBCL. Currently this is solved by imposing a rather outmodish way of building the software based on compile-file and load. No notion of standalone binary files, no way of producing executables, no way of shipping things. 2. Each lisp has a different notion of how to locate "required" components. Some have search paths (SBCL), some do not (ECL). There is not a standard for installation. 3. There is an additional problem in the lisp world: we do not use the shared library loader! In order to locate components we may either rely on crappy things like REQUIRE or rely on an external agent that seeks those components prior to loading our software. ASDF behaves in that sense as a shared library loader that takes into account version information! 4. System definitions provide the description of our system and how pieces bind together. This has always been like that, even before ASDF, with mk:defsystem. It is a simple, mostly declarative description. What we do with it is up to the software that manipulates it, just like automake may evolve at some point and behave differently. 5. Installation is again a problem because there is not a well defined way to look for software. Furthermore, since current system descriptions do not provide information about "resources" that the software may need, such as bitmaps, icons, databases, etc, we have no information about installing them and instead things are left in place. On comparing both lists one must reach the conclusion that the situation is bad. Indeed. All of the problem revolves around two things * We need some way to build our system taking into account all dependencies * There is no standard way to produce standalone libraries and install them in a way that just works. ASDF tries to solve BOTH problems, that is BUILDING and LOADING/EXECUTING. The problem is that development has over the time forgotten the distinction between both and one finds amazing things such as users that place some of the execution logic (creating packages, classes, etc) IN THE SYSTEM DEFINITION! However, while we are in a position of say, disadvantage, w.r.t the Unix community at large, I believe we have a good starting point to get things right. My vision is to *progressively* decouple the building and loading components in ASDF. I do not want to push anything down anybody's throat, but even less take all the existing database of knowledge about all the libraries that build and work with ASDF and presenting myself as an illuminated that has the solution to everything and that this solution has to be something orthogonal to ASDF. How can we achieve these things and make everybody's life simpler? I am not going to produce the roadmap, but what I present below can be done progressively, and examples for that are the bits I have been contributing to the asdf-devel list. * First of all understand that we have two different targets here. One is people building standalone software (programs) and the other one is people building libraries. In both cases the system definitions we have are good enough to describe them and their dependencies. Let's call the component in ASDF that handles these systems ASDF-system. This includes classes and the parsing logic. This is very tiny and should be easy to bootstrap. * Understand how different lisps locate software. Realize that not all lisps can do that cleverly and that this is one of the reasons for ASDF to exist. Produce a minimalistic ASDF-search that reuses existing logic or implements new one for lisps that miss that part. * Realize that we need a better way of loading software. I do not care if that guy is improving REQUIRE, he is building a new component that people will have to install or ship with their lisps and autoconfed scripts are not going to solve it. * As a solution to the previous point realize that system definitions already have enough information for an ASDF-loader facility that builds on ASDF-search and ASDF-system. Even for standalone systems it is possible to create stripped down *.asd files with version information and dependencies in them. This is already present in the patch I sent to the list and ECL can also make it: it is the binary-op operation I introduced. The result is a file (defsystem :my-system :depends-on (:cl-ppcre) :components ((:compiled-file "my-component"))) which contains just the minimal information for our "shared library" loader to work. I personally find this better (and potentially more extensible) than Unix version numbers in file names (libecl.so.10.2.1 ?!?) * ASDF-loader can be a very transparent component and it may even be possible to ship it together with _every_ library you have around, just like libtool is shipped with every library. As a compatibility layer over REQUIRE it should be possible to produce three files for each system - the system definition my-system.asd - a simple-minded loader my-system.fas that sets up paths, requires asdf-loader and loads the last file which is... - the component of the library itself, my-component.fas * We still need a building facility, say ASDF-builder, that uses a system definition to compile everything. This would be the COMPILE-OP we have right now, glorified and improved, but it would be augmented with facilities to pack a library in a single file, building standalone programs, etc. * The interesting part about this decoupling is that standalone programs do not need any of it. I can just use ASDF-builder as a glorified makefile and produce a program that just works by concatenating or linking pieces of precompiled software and inserting a simple logic at the beginning of the program that sets up the environment that the components expect. No need for ASDF-loader nor ASDF-search nor ASDF-system. Since I already removed all dependency on ASDF from the executable code itself by having logical pathnames, things just work. Compare this with a hand-built software where I abuse REQUIRE here and there. Say you are a developer of a library, MY-LIBRARY, and want the user to just build it and install it. There are various alternatives to achieve this. One is to continue as now. The user loads ASDF and from the prompt she is able to compile, build and install the software with existing operations. But from the point of view of the user things can be improved a lot by means of scripting. * Assume that ASDF is preinstalled by the user. This needs not be done in complicated fashion. ASDF itself can be "autoconf'ed" and taught to find out the existing lisps, install itself, and create a set of scripts that the application is going to use: "asdf-build" and "asdf-install". * Alternatively you may ship a version of ASDF with your library, hidden in an asdf directory just like libtool hides itself in the m4 or config directory of a Unix library. The scripts asdf-build and asdf-install are then placed on the top of your library source tree, where the *.asd definitions live. Now the user only has to do something like ./asdf-build --lisps=sbcl,ecl,acl ./asdf-install These two scripts will loop over lisps loading them with ASDF, compiling and installing stuff, reusing the knowledge this library has about the systems themselves. If the lisp environment does not have ASDF installed, a copy may be installed together with the library or we may use the logic explained above by which each library carries a copy of ASDF-loader just in case there is none. All this can be coded on top of the existing ASDF, better factoring the code, and with a bits of magic here and there. In short * Except for standalone software, we a lisp loader that -- Takes into account dependencies -- Searches for the software at standard installation paths * ASDF may coexist with software built using other tools. * ASDF may build software for various distribution purposes. * ASDF can hide itself from the user as much as we want. * ASDF can produce software that does not rely on a configured ASDF to load * We can write libraries that do not care much whether ASDF is there or not * There is always going to be some amount of configuration on the side of the user -- Either using scripts and assuming default installation paths -- Either specifying search paths SBCL_SEARCH_PATH=... * System definitions are useful, dependencies scattered over your sources are not. Juanjo [1] ECL itself [2] Libraries I have to ship to clusters all around the world (AIX, Linux, SGI...) See signature below. -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

3 7

[asdf-devel] Error loading closure-html
by Futu Ranon 30 Mar '10

30 Mar '10

Hello, I wanted to test closure-html so I installed and attempted to load it in LispWorks 6 on Windows. CL-USER 3 > (asdf:load-system 'closure-html) > > Error: External format (:UTF-8 :EOL-STYLE :LF) produces characters of type > SIMPLE-CHAR, which is not a subtype of the specified element-type BASE-CHAR. > 1 (abort) Return to level 0. > 2 Return to top loop level 0. > > Type :b for backtrace or :c <option number> to proceed. > Type :bug-form "<subject>" for a bug report template or :? for other > options. > > CL-USER 4 : 1 > :b > Call to ERROR > Call to STREAM::EXTERNAL-FORMAT-STREAM-DEFAULTED-ELEMENT-TYPE > Call to STREAM::EXTERNAL-FORMAT-OPEN > Call to OPEN > Call to ASDF:FIND-SYSTEM > Call to ASDF:OPERATE > Call to ASDF:LOAD-SYSTEM > Call to EVAL > Call to CAPI::CAPI-TOP-LEVEL-FUNCTION > Call to CAPI::INTERACTIVE-PANE-TOP-LOOP > Call to MP::PROCESS-SG-FUNCTION > > CL-USER 5 : 1 > > This diff corrects it for me. I don't have another lisp to test it immediately to see if this needs to be conditional. --- a/system/asdf/asdf.lisp +++ b/system/asdf/asdf.lisp @@ -1022,7 +1022,7 @@ to `~a` which is not a directory.~@:>" (< (car in-memory) (safe-file-write-date on-disk)))) (let ((package (make-temporary-package))) (unwind-protect - (with-open-file (asd on-disk :if-does-not-exist nil) + (with-open-file (asd on-disk :if-does-not-exist nil :element-type :default) (if asd (let ((*package* package)) (asdf-message Is this correct or is there something broken on my end?

4 8

[asdf-devel] SBCL port of ECL's extensions
by Juan Jose Garcia-Ripoll 29 Mar '10

29 Mar '10

I just cut & paste what I submitted to the appropriate record in launchpad https://bugs.launchpad.net/asdf/+bug/542583 where the port can be downloaded and given a try. I attach a straightforward port of two operations that ECL supports: - Building a unified FASL file that contains either a system, or a system plus all its dependencies. - Building an ASDF system that, using the previous feature, can replace an existing one (this include *.asd file) The implementation tries to be done with future extensions in mind, such as bundling all executables plus resources into an application, dumping images, etc, but all is constrained into new files that are not required for booting -- should simplify the process of reloading ASDF. The patch need not be accepted as is. I will submit it for discussion at the mailing list. -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

4 16

[asdf-devel] ABCL testing issue
by Faré 28 Mar '10

28 Mar '10

http://ec2-174-129-63-37.compute-1.amazonaws.com/test/log/20100316T233422/a… > > Testing: test-force.script > Armed Bear Common Lisp 0.18.1 > Java 1.6.0_0 Sun Microsystems Inc. > OpenJDK Client VM > Low-level initialization completed in 1.985 seconds. > Startup completed in 3.591 seconds. > > Caught ERROR while processing --eval option "(load "test- > force.script")": > Failed to find asdf-abcl.lisp in /ebs/test/asdf/tmp/asdf-abcl.abcl. > Using abcl , test-force.script failed > This looks like a failure in ABCL's make-pathname or merge-pathnames mechanism. Mark, can you comment? >> In run-tests.sh I currently use >> export ASDF_OUTPUT_TRANSLATIONS="${asdfdir}:${asdfdir}/tmp/fasls/$ >> (basename >> $command):" > > the problem arises when running run-tests.sh from a web server. > > would it controvert the purpose fo the tests to set it to a single > directory which i know the web server process can use? > Not at all. Make the variable unmodified if it still exists. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The Constitution may not be perfect, but it's a lot better than what we've got!

3 3

[asdf-devel] question for API
by Robert Goldman 24 Mar '10

24 Mar '10

One thing I find myself doing over and over (and I don't think I'm alone in this) in .asd files is to capture a pathname for the directory of the system definition file. This is particularly important when one has code that wants to load data files, in the presence of FASL file relocation (since it means that source code files can't effectively use *load-truename*). Bits of code like this seem like a good candidate for being added to the ASDF API. Any thoughts? R

2 4

[asdf-devel] Enforcing pure *.asd files
by Juan Jose Garcia-Ripoll 24 Mar '10

24 Mar '10

This is an idea that has been long floating in the back of my mind, and was brought back to life by these comments http://groups.google.com/group/comp.lang.lisp/msg/f99a69797eda1caf The problem is that many people use *.asd files to do things like building up packages, creating operations, defining methods, etc. That makes it impossible to ship those libraries because the side effects are not recorded in the system itself. So if ECL traverses the system to build a standalone program it will know which sources are required, but those sources may for instance rely on a package that was created in the *.asd definition. My suggestion is a gradual move towards enforcing that *.asd files become mere descriptions of the system and that side effects needed for building and loading are listed in the system itself. - System definitions should be read and recorded by ASDF in pure form, that is as S expressions, with a hash that maps system names to s-expressions. - Instead of using LOAD we use a LOOP and READ. If a form which is not a DEFSYSTEM is found, we have two options: if we enforce strict mode, an error is signaled, otherwise the form is evaluated after emitting a warning. - Warnings could also be emitted when a DEFSYSTEM clause does not match the name of the file, or when more than one such clauses is found. - The system definition should be scanned for a class name and for a keyword option, :asdf-support If the :asdf-support option is found, it should name a file with side effects that are needed for the ASDF definition to be used. The file should be loaded using LOAD. - The original S expression is now parsed and the system definition is created. The warning could be really self-explanatory, stating the following: - Dependencies that are needed for processing the system definition must be listed in :asdf-support - Dependencies that are needed for load/compile-op should be listed in :depends-on I can think on even more elegant ways to do this. For instance having proxy classes, such as proxy-system and proxy-component which are used to store the S-expression for the system and using change-class on these objects to transform them into valid systems and components on demand, that is only when the systems are needed. It should also be possible to write an utility that grovels existing ASDF systems looking for potential problems and suggesting changes. What the solution above does not address is the following sentence from Daniel's comment: As to the "no". Simply doing asdf:find-component may result in side effects, both in the package of interest and in random other packages before it. These side effects include dumping fasls. There are several asd files that force things to load early in order to extend ASDF by defining custom classes." In other words, the side effects in :asdf-support can be really harmful, or costly. However this proposal could help us identify what libraries do have side effects and also help enforce further restrictions in the side effects they may have. I mean, there are many many other ways to do this, but I think you get the idea. Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

7 24

[asdf-devel] maintaining the ASDF web site
by Robert Goldman 23 Mar '10

23 Mar '10

OK, I've looked at this, and I'm afraid I am not going to take this on in its current form. If someone else would be so kind as to wrangle the markdown and grok the scripts, that's fine. If after some time, no one else expresses a willingness to do this, then I will try to do some of it, but I hereby put the community on notice: I don't know markdown, and I'm not going to learn markdown. If this is left to me, the markdown will go away, and HTML will gradually take its place. HTML is ugly, annoying, difficult, etc., but it has the immense advantage, to me, that I know it, and I know the toolchain (which is basically emacs + my browser). So there it is --- anyone interested in taking this over? Best, r

2 1