asdf-devel

asdf-devel@common-lisp.net

5 participants
1375 discussions

[Asdf-devel] startup times and initialize-source-registry
by Faré 25 Aug '14

25 Aug '14

At ILC 2014, one discussed show-stopper for using CL as a scripting language was startup time. Indeed, right now, when used as a script rather than as a dumped image, CL takes a lot of time to start: time ( sbcl --noinform --eval '(require :asdf)' --eval '(progn (asdf:initialize-source-registry) (uiop:writeln (hash-table-count asdf::*source-registry*)) (uiop:quit))' ) 711 ( sbcl --noinform --eval '(require :asdf)' --eval ; ) 0.66s user 0.17s system 99% cpu 0.832 total time cl '(hash-table-count asdf::*source-registry*)' 711 cl '(hash-table-count asdf::*source-registry*)' 1.20s user 0.25s system 99% cpu 1.456 total That's because it will recursively walk all the directories under the registered source-registry trees, and there can be a lot of them. Ben Hyde tells me it's much worse on his machine. The two following slightly incompatible changes divide that startup time by three, and promises to divide it further if people adhere to some discipline. (defun collect-sub*directories (directory collectp recursep collector) "Given a DIRECTORY, call-function the COLLECTOR function designator on the directory if COLLECTP returns true when CALL-FUNCTION'ed with the directory, and recurse each of its subdirectories on which the RECURSEP returns true when CALL-FUNCTION'ed with them." (when (call-function collectp directory) (call-function collector directory) (dolist (subdir (subdirectories directory)) (when (call-function recursep subdir) (collect-sub*directories subdir collectp recursep collector))))) This nests the dolist into the when, which is backward compatible as far as uiop and asdf internal usage is concerned, but not as far as other users might be concerned; however, the collectp function is a bit redundant and useless without this nesting. (defun collect-sub*directories-asd-files (directory &key (exclude *default-source-registry-exclusions*) collect (stop-at-asd t)) (collect-sub*directories directory #'(lambda (dir) (let ((asds (directory-asd-files dir))) (map () collect asds) (not (and asds stop-at-asd)))) #'(lambda (x) (not (member (car (last (pathname-directory x))) exclude :test #'equal))) (constantly nil))) The trick here is in this new stop-at-asd flag, which here defaults to t and isn't configurable, but which should default to nil and be configurable, for backward compatibility. Its effect is that recursing into subdirectories stops if a .asd file is found in the toplevel directory. This saves a lot of recursing, and would save even more if a .asd file of symlink to one exists at the top of a git hierarchy. But this is incompatible with a lot of existing code, and so the transition will be long and painful if this is adopted. With these changes, I get: time ( sbcl --noinform --eval '(require :asdf)' --eval '(progn (asdf:initialize-source-registry) (uiop:writeln (hash-table-count asdf::*source-registry*)) (uiop:quit))' ) 534 ( sbcl --noinform --eval '(require :asdf)' --eval ; ) 0.24s user 0.05s system 99% cpu 0.293 total time cl '(hash-table-count asdf::*source-registry*)' 534 cl '(hash-table-count asdf::*source-registry*)' 0.54s user 0.13s system 99% cpu 0.665 total That's much better timewise (about 3x speedup), but it's obviously missing a lot of .asd files. To recover them, I had to: for i in */ ; do ( cd $i ; setopt NULL_GLOB ; A=( */**/*.asd ) ; echo $i $#A ; if [ $#A -gt 0 ] ; then ln -s $A . ; fi ) ; done |&tee /tmp/a Then I get: time ( sbcl --noinform --eval '(require :asdf)' --eval '(progn (asdf:initialize-source-registry) (uiop:writeln (hash-table-count asdf::*source-registry*)) (uiop:quit))' ) 711 ( sbcl --noinform --eval '(require :asdf)' --eval ; ) 0.24s user 0.09s system 99% cpu 0.335 total time cl '(hash-table-count asdf::*source-registry*)' 711 cl '(hash-table-count asdf::*source-registry*)' 0.50s user 0.07s system 100% cpu 0.567 total And... oops, I realize I failed to do the scripting at the SLIME REPL. Mea culpa. Of course, if you want instant startup without any search, you can eschew ASDF2 style autoconfiguration, and go the sysadmin way of ASDF1. I still think there is value in combining autoconfiguration with somewhat faster startup time than we have now. Of course, implementing such a plan with a multi-year backward-compatible migration strategy is the prerogative of the current and future maintainers, if they wish to undertake it: it would take implementing (and testing) the code, but disabling it by default, enabled with suitable (backward-incompatible) flags in configuration files. Then, while waiting for all implementations to eventually adopt the new release, pushing users to reform the way they layout their directories (and maybe adopt package-inferred-system, while we're at it — it could help systems that have a lot of one-file subsystems. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org As for poverty, no one need be ashamed to admit it: the real shame is in not taking practical measures to escape from it. — Perikles

4 8

Re: [Asdf-devel] ASDF
by Faré 20 Aug '14

20 Aug '14

Dear Geoff, >> If you are still interested in becoming ASDF maintainer, which I hope >> you are, or even just developer, here is a post linking to some >> suggested things to read about the state of ASDF: >> http://fare.livejournal.com/176185.html > > I am still interested, even enthusiastic. I will read that and see > how I feel with respect to time commitment and ambition. That said, > I'm very much interested in taking part in ASDF, even if not in the > capacity of maintainer. > [NB: continuing this previously private mail thread in asdf-devel] If you're looking at where the action is, there are two branches that still haven't been merged into master: * the minimakefile branch, where I translated all the release and test code from shell scripts to CL scripts, with various improvements, paving the way for possible future improvements. I believe this branch is ready, but Robert is just lacking time to fully assess it; also, he was hoping for Kambiz Darabi's promised use of git submodules to manage all the dependencies of ASDF — though I don't believe this branch makes the issue any worse. * the syntax-control branch, where I introduce control for the *readtable* variable, allowing for safe compilation at a REPL (or other) where the *readtable* was overridden (to e.g. use fare-quasiquote or reader-interception, etc.) Unhappily, this branch requires more development: it seems that at least *package* needs the same treatment; and ideally all the variables of with-standard-io-syntax, though that might cause "interesting" backward compatibility issues. Also, Robert mentions that some large software he is using is somehow failing with this branch, for reasons he hasn't pinpointed. If and when this branch is merged into master (if ever), it deserves a version bump to 3.2 or some such. Even if the default changes to something safer than now, some backward compatibility mode is probably required to cater to the needs of all users. Robert may have further comments or clarifications. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org There is neither early nor late to fall in love. Only the right time.

2 1

[Asdf-devel] ASDF and generated files
by Ralf Mattes 15 Aug '14

15 Aug '14

Hello, dear ASDF developers, I'm currently trying to port some old lisp code (Common Music) to ASDF3[1]. Common Music's source is written in Scheme, the Common Lisp version is generated from the Scheme code. Unfortunately CM's current asdf file is littered with executable code (not only compiling inoking the Scheme->Lisp compiler but also calling (non-portable) shell scripts etc. This is unfortunate since many asdf operations will trigger this code. Hence my attempt to rewrite the asdf and transfer the scheme->lisp part into a second asdf system. Unfortunately I can't find any documentation on how to implement such a code transformation. I can create a new component class scheme-file but what methods do I need to implement? Where can I call the function that actually compiles a scheme file to lisp (Common Music's scheme->lisp compiler works on files). TIA Ralf Mattes Footnotes: [1] my main goal is to make Common Music distributable by quicklisp.

5 4

[Asdf-devel] Bug in operate :around t t method?
by Jan Moringen 11 Aug '14

11 Aug '14

Hi, I think there is a bug in the operate :around t t method. I looked at this in the SBCL source tree and couldn't find ASDF's upstream git repository on common-lisp.net, so this may already be fixed there. Sorry for the noise in that case. The problem is #'(lambda () (make-operation name :original-initargs initargs initargs)) which, I think, should be #'(lambda () (apply #'make-operation name :original-initargs initargs initargs)) Kind regards, Jan

3 3

[Asdf-devel] COERCE-CLASS error format string
by Matt Niemeir 06 Aug '14

06 Aug '14

Hello. COERCE-CLASS's error reporting has an invalid format string, which becomes evident in the error reporting of a call like: (asfd:operate 'cl-user::load-op 'alexandria) I have (tried to) attach a patch to this email.

2 3

[Asdf-devel] Some ASDF patches
by Faré 05 Aug '14

05 Aug '14

Dear Robert, I have two local branches with these patches. May I merge into master? The input-string function was added for symmetry while hacking cl-launch, though I ended up not using it, to remain compatible with older versions of uiop. The uiop:getenv bug was discovered while fixing cl-launch regressions on LispWorks. The fasl-op thing is what he had discussed before (only minimally manually tested). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you could kick the person in the pants responsible for most of your trouble, you wouldn't sit for a month. — Theodore Roosevelt

2 4

[Asdf-devel] Call for Testers (just quick bootstrap test for now)
by Dave Cooper 01 Aug '14

01 Aug '14

Dear All, We are releasing Gendl (our free, open-source product) as a pre-built distribution, based on CCL, on MacOS, Linux, and Windows (I'm giving a tutorial on Gendl at Montreal ILC, so this is part of the reason for making convenient self-contained builds available). At this point we'd like to request anyone who is interested to download one or more of these bundles, and simply test the start-up. The respective downloads for current builds are here: http://downloads.genworks.com/gendl1587p004-devo-ccl.dmg (Mac) http://downloads.genworks.com/gendl1587p004-windows-oss.exe (Windows) http://downloads.genworks.com/gendl1587p004-ccl-linux.zip (Linux) The Mac and Windows versions are self-contained including Emacs and enough of Quicklisp/Slime to get bootstrapped (quicklisp and asdf are not built-in to the image). The Linux distribution assumes that you have "emacs" installed and in your path. The start-up is as follows: Mac: simply drag-to-install and click the application icon. Windows: run the setup executable and run one of the .bat files in the toplevel directory (or from Start menu if you have one) Linux: Unzip the .zip file, cd into the new directory, and ./run-gdl This should start emacs, Slime, and give you a GDL-USER> prompt. On Linux, we have had a report that the Slime tries to load ASDF. I'm not sure why this is, and I'm not able to replicate the issue on any Linux systems I have access to. So especially if anyone is able to replicate this ASDF loading error on the Linux version, it will be greatly appreciated (especially if you can see why it's trying to load asdf!) By the way, these distributions should work regardless of any other Lisp or Emacs configuration you currently have on your machine (and shouldn't interfere with them). Note that I still have to fix up the Quicklisp copyright statements in the startup banner for the included quicklisp libraries (this is done in the Linux distro but the other distros still have to be rebuilt with the proper statements). My plan is to make available a "stable" and "nightly" for each of these builds at all times. -- My Best, Dave Cooper, Genworks Support david.cooper(a)genworks.com, dave.genworks.com(skype) USA: 248-327-3253(o), 1-248-330-2979(mobile)

1 0

[Asdf-devel] (asdf:output-file 'asdf:monolithic-fasl-op ...) no longer working
by Dave Cooper 29 Jul '14

29 Jul '14

I'm not sure how much this matters because it's mostly a backward compatibility issue with something which is a relatively new feature anyway, but (asdf:output-file[s] 'asdf:monolithic-fasl-op ...) no longer works in 3.1.3 (at least on CCL/Linux). The new nomenclature is apparently: (asdf:output-file[s] 'asdf:monolithic-compile-bundle-op ...) is 'asdf:monolithic-compile-bundle-op indeed the exact replacement for 'asdf:monolithic-fasl-op? FWIW, (asdf:operate 'asdf:monolithic-fasl-op ...) still appears to work. I apologize for not running tests on my own stuff before the release of 3.1.3; I'll try to make a point of doing that for future releases. -- My Best, Dave Cooper, Genworks Support david.cooper(a)genworks.com, dave.genworks.com(skype) USA: 248-327-3253(o), 1-248-330-2979(mobile) UK: 0191 645 1699

3 5

[Asdf-devel] minimakefile branch
by Faré 29 Jul '14

29 Jul '14

I hadn't tested the minimakefile branch with quicklisp, so of course it wasn't working. Fixed. All you need to bootstrap the asdf-tools is a recent quicklisp (and possibly removing antique cl-ppcre packages from debian that might take precedence). I just realized that with quicklisp, asdf:load-system seems to load already-installed quicklisp packages, but not install new ones, for which you need ql:quickload. Xach, is that on purpose? —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I'll start exercising as soon as I'm into shape.

3 3

[Asdf-devel] ASDF version 3.1.3 released: Important bug-fix release
by Robert P. Goldman 24 Jul '14

24 Jul '14

Today I have released (with much help from Fare) ASDF 3.1.3. We urge implementations that are currently shipping with 3.1.2 to move forward to 3.1.3. 3.1.3 has no API incompatibilities that we know of, and contains significant bug fixes. Most significantly, 3.1.3 fixes bugs that impede hot-upgrade from 3.1.2 to a later version of ASDF, which is critical to ASDF development. In addition, we have fixed multiple bugs in ASDF system search caching, that impeded correct use of restarts involving adding new systems to a running CL. Regards, Robert Goldman

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

asdf-devel