- asdf-devel - mailman3.common-lisp.net

Re: [Asdf-devel] startup times and initialize-source-registry
by Ben Hyde 02 Sep '14

02 Sep '14

Would changing (:tree <path>) so it accepts an optional argument be less adhoc? Possibly (:tree <path> &key (depth nil) (don-not-recure-after-asdf-encountered nil)) That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those. - ben

4 15

[Asdf-devel] Minor minimakefile nit
by Robert P. Goldman 02 Sep '14

02 Sep '14

The minimakefile help lists: archive alias for command make-and-publish-archive but make-and-publish-archive is not listed as a command in the help, nor in the makefile. Is this description out of date? Question: can we do a long help for individual commands, like make help archive? As an aside, I would say that if one is interested in making CL a serious scripting contender, some equivalent of perl's Getopt::Long library for command line parsing (perhaps Didier's Clon would fit the bill?), and pod inline documentation is going to be critical. I pushed a comment on the makefile to explain at the top how to get help. Cheers, r

3 2

[Asdf-devel] Problem with Minimakefile branch -- was Re: Merging experimental-submodules
by Robert P. Goldman 02 Sep '14

02 Sep '14

Kambiz Darabi wrote: > I don't know how many people are devs and how many people just 'users' > in the sense that they clone the repo and use it without caring about > the dependencies of asdf. > > But your point is definitively valid. Actually, Kambiz's point is a good one, and it illustrates a real problem with the minimakefile branch. With the old makefile, all someone needed to do if they were to checkout and test ASDF was to grab the git repo, and type make test l=<my lisp> Now this process is substantially more complicated because now the user will have to go to substantially more work to get our not extensively tested CL infrastructure up and running. We should seriously consider revising the minimakefile so that testing does not require CL scripting, but instead goes back to using make and bash alone. Best, r

2 3

[Asdf-devel] Merging experimental-submodules
by Robert P. Goldman 02 Sep '14

02 Sep '14

Now is the time to speak up if there's any reason I should *not* merge the experimental-submodules branch, which will make ASDF freestanding by pulling in its build dependencies. I'll probably merge this and push sometime tomorrow. Cheers, r

3 4

[Asdf-devel] Time to make ASDF more hookable
by Robert P. Goldman 28 Aug '14

28 Aug '14

ASDF is getting bigger and bigger. Faré and others wish to do some more exciting things with ASDF, like make it the basis for scripting, etc. BUT: ASDF is the "make" of Common Lisp. It must be stable and solid. The whole community depends upon it. So... How can we have our cake and eat it, too? If people want to do more exciting things with ASDF, and don't want to develop their own build system from scratch (the Java folks seem to do this every 15 minutes), it's time to allow the community better hooks to personalize ASDF, instead of making it more and more complicated and, inevitably, more brittle. Then, if you hot rod your ASDF, you can make it do exactly what you want, like your emacs or your linux box. And you won't mess up everyone's reliable but boring commuter sedan in the process. The flexibility of Lisp should let us provide the hooks to let people build a playground on top of ASDF. Let's figure out how to do this.

2 1

[Asdf-devel] request for test: experimental-submodules branch
by Robert P. Goldman 27 Aug '14

27 Aug '14

I have tried to pull all the dependencies for ASDF building into the repo. I have done this primarily using git submodules, but also had to pull in cl-ppcre as source, since it doesn't seem to be available as a git repo. If you can, please pull an update, checkout the experimental-submodules branch, and see if it configures properly. All the dependencies should pop up in the ext/ subdirectory (this may not be the best place; I'm open to suggestions for improvement). Please report any problems and Thanks, Robert

3 17

[Asdf-devel] startup times and initialize-source-registry
by Faré 25 Aug '14

25 Aug '14

At ILC 2014, one discussed show-stopper for using CL as a scripting language was startup time. Indeed, right now, when used as a script rather than as a dumped image, CL takes a lot of time to start: time ( sbcl --noinform --eval '(require :asdf)' --eval '(progn (asdf:initialize-source-registry) (uiop:writeln (hash-table-count asdf::*source-registry*)) (uiop:quit))' ) 711 ( sbcl --noinform --eval '(require :asdf)' --eval ; ) 0.66s user 0.17s system 99% cpu 0.832 total time cl '(hash-table-count asdf::*source-registry*)' 711 cl '(hash-table-count asdf::*source-registry*)' 1.20s user 0.25s system 99% cpu 1.456 total That's because it will recursively walk all the directories under the registered source-registry trees, and there can be a lot of them. Ben Hyde tells me it's much worse on his machine. The two following slightly incompatible changes divide that startup time by three, and promises to divide it further if people adhere to some discipline. (defun collect-sub*directories (directory collectp recursep collector) "Given a DIRECTORY, call-function the COLLECTOR function designator on the directory if COLLECTP returns true when CALL-FUNCTION'ed with the directory, and recurse each of its subdirectories on which the RECURSEP returns true when CALL-FUNCTION'ed with them." (when (call-function collectp directory) (call-function collector directory) (dolist (subdir (subdirectories directory)) (when (call-function recursep subdir) (collect-sub*directories subdir collectp recursep collector))))) This nests the dolist into the when, which is backward compatible as far as uiop and asdf internal usage is concerned, but not as far as other users might be concerned; however, the collectp function is a bit redundant and useless without this nesting. (defun collect-sub*directories-asd-files (directory &key (exclude *default-source-registry-exclusions*) collect (stop-at-asd t)) (collect-sub*directories directory #'(lambda (dir) (let ((asds (directory-asd-files dir))) (map () collect asds) (not (and asds stop-at-asd)))) #'(lambda (x) (not (member (car (last (pathname-directory x))) exclude :test #'equal))) (constantly nil))) The trick here is in this new stop-at-asd flag, which here defaults to t and isn't configurable, but which should default to nil and be configurable, for backward compatibility. Its effect is that recursing into subdirectories stops if a .asd file is found in the toplevel directory. This saves a lot of recursing, and would save even more if a .asd file of symlink to one exists at the top of a git hierarchy. But this is incompatible with a lot of existing code, and so the transition will be long and painful if this is adopted. With these changes, I get: time ( sbcl --noinform --eval '(require :asdf)' --eval '(progn (asdf:initialize-source-registry) (uiop:writeln (hash-table-count asdf::*source-registry*)) (uiop:quit))' ) 534 ( sbcl --noinform --eval '(require :asdf)' --eval ; ) 0.24s user 0.05s system 99% cpu 0.293 total time cl '(hash-table-count asdf::*source-registry*)' 534 cl '(hash-table-count asdf::*source-registry*)' 0.54s user 0.13s system 99% cpu 0.665 total That's much better timewise (about 3x speedup), but it's obviously missing a lot of .asd files. To recover them, I had to: for i in */ ; do ( cd $i ; setopt NULL_GLOB ; A=( */**/*.asd ) ; echo $i $#A ; if [ $#A -gt 0 ] ; then ln -s $A . ; fi ) ; done |&tee /tmp/a Then I get: time ( sbcl --noinform --eval '(require :asdf)' --eval '(progn (asdf:initialize-source-registry) (uiop:writeln (hash-table-count asdf::*source-registry*)) (uiop:quit))' ) 711 ( sbcl --noinform --eval '(require :asdf)' --eval ; ) 0.24s user 0.09s system 99% cpu 0.335 total time cl '(hash-table-count asdf::*source-registry*)' 711 cl '(hash-table-count asdf::*source-registry*)' 0.50s user 0.07s system 100% cpu 0.567 total And... oops, I realize I failed to do the scripting at the SLIME REPL. Mea culpa. Of course, if you want instant startup without any search, you can eschew ASDF2 style autoconfiguration, and go the sysadmin way of ASDF1. I still think there is value in combining autoconfiguration with somewhat faster startup time than we have now. Of course, implementing such a plan with a multi-year backward-compatible migration strategy is the prerogative of the current and future maintainers, if they wish to undertake it: it would take implementing (and testing) the code, but disabling it by default, enabled with suitable (backward-incompatible) flags in configuration files. Then, while waiting for all implementations to eventually adopt the new release, pushing users to reform the way they layout their directories (and maybe adopt package-inferred-system, while we're at it — it could help systems that have a lot of one-file subsystems. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org As for poverty, no one need be ashamed to admit it: the real shame is in not taking practical measures to escape from it. — Perikles

4 8

Re: [Asdf-devel] ASDF
by Faré 20 Aug '14

20 Aug '14

Dear Geoff, >> If you are still interested in becoming ASDF maintainer, which I hope >> you are, or even just developer, here is a post linking to some >> suggested things to read about the state of ASDF: >> http://fare.livejournal.com/176185.html > > I am still interested, even enthusiastic. I will read that and see > how I feel with respect to time commitment and ambition. That said, > I'm very much interested in taking part in ASDF, even if not in the > capacity of maintainer. > [NB: continuing this previously private mail thread in asdf-devel] If you're looking at where the action is, there are two branches that still haven't been merged into master: * the minimakefile branch, where I translated all the release and test code from shell scripts to CL scripts, with various improvements, paving the way for possible future improvements. I believe this branch is ready, but Robert is just lacking time to fully assess it; also, he was hoping for Kambiz Darabi's promised use of git submodules to manage all the dependencies of ASDF — though I don't believe this branch makes the issue any worse. * the syntax-control branch, where I introduce control for the *readtable* variable, allowing for safe compilation at a REPL (or other) where the *readtable* was overridden (to e.g. use fare-quasiquote or reader-interception, etc.) Unhappily, this branch requires more development: it seems that at least *package* needs the same treatment; and ideally all the variables of with-standard-io-syntax, though that might cause "interesting" backward compatibility issues. Also, Robert mentions that some large software he is using is somehow failing with this branch, for reasons he hasn't pinpointed. If and when this branch is merged into master (if ever), it deserves a version bump to 3.2 or some such. Even if the default changes to something safer than now, some backward compatibility mode is probably required to cater to the needs of all users. Robert may have further comments or clarifications. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org There is neither early nor late to fall in love. Only the right time.

2 1

[Asdf-devel] ASDF and generated files
by Ralf Mattes 15 Aug '14

15 Aug '14

Hello, dear ASDF developers, I'm currently trying to port some old lisp code (Common Music) to ASDF3[1]. Common Music's source is written in Scheme, the Common Lisp version is generated from the Scheme code. Unfortunately CM's current asdf file is littered with executable code (not only compiling inoking the Scheme->Lisp compiler but also calling (non-portable) shell scripts etc. This is unfortunate since many asdf operations will trigger this code. Hence my attempt to rewrite the asdf and transfer the scheme->lisp part into a second asdf system. Unfortunately I can't find any documentation on how to implement such a code transformation. I can create a new component class scheme-file but what methods do I need to implement? Where can I call the function that actually compiles a scheme file to lisp (Common Music's scheme->lisp compiler works on files). TIA Ralf Mattes Footnotes: [1] my main goal is to make Common Music distributable by quicklisp.

5 4

[Asdf-devel] Bug in operate :around t t method?
by Jan Moringen 11 Aug '14

11 Aug '14

Hi, I think there is a bug in the operate :around t t method. I looked at this in the SBCL source tree and couldn't find ASDF's upstream git repository on common-lisp.net, so this may already be fixed there. Sorry for the noise in that case. The problem is #'(lambda () (make-operation name :original-initargs initargs initargs)) which, I think, should be #'(lambda () (apply #'make-operation name :original-initargs initargs initargs)) Kind regards, Jan

3 3