Re: [Asdf-devel] startup times and initialize-source-registry
Would changing (:tree <path>) so it accepts an optional argument be less adhoc? Possibly (:tree <path> &key (depth nil) (don-not-recure-after-asdf-encountered nil)) That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those. - ben
On Tue, Aug 26, 2014 at 1:40 PM, Ben Hyde <bhyde@pobox.com> wrote:
Would changing (:tree <path>) so it accepts an optional argument be less adhoc?
Possibly (:tree <path> &key (depth nil) (don-not-recure-after-asdf-encountered nil))
I of course thought about this, but it's the wrong thing: the person who knows best what's in a tree is not the end-user, but the developer, package-manager or system-administrator, and that's who should be doing this configuring. Therefore, this configuration option is ultimately no help unless the developer, package-manager or system-administrator gets to write configuration files, which is essentially my proposal, at which point this option is not needed, because it can be done better without, with a simple script builds a list of all asd file paths under the tree. Proposal: * :tree checks for cl-source-registry.conf or .cl-source-registry.conf and recurses on that rather than on a directory search, when available. * :file is accepted, in addition to :directory and :tree * a :source-registry entry can have a :cache entry (or be followed by a :cache entry? or have a .cache file equivalent?) that lists all relevant .asd files as a sorted list of relative unix-namestrings. * Some script can regenerate the cache from the registry.
That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those.
My proposal would solve your issue, at the cost of re-running the script when you install, deinstall or update lisp code. Or you could split your active CL source in a separate directory. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org It's better to know nothing that to know what ain't so. — Josh Billings
On Aug 26, 2014, at 2:40 PM, Faré <fahree@gmail.com> wrote:
* a :source-registry entry can have a :cache entry (or be followed by a :cache entry? or have a .cache file equivalent?) that lists all relevant .asd files as a sorted list of relative unix-namestrings. * Some script can regenerate the cache from the registry.
Something of this kind - A disk cache of *source-registry* - is my preference de jour. It’s trivial to dump the *source-registry* to a file and reload it. My hack here along those lines is almost nothing more than a list of pathnames. The code which builds the source registry fascinates my inner archeologist; I wouldn’t want to write that in bash. It is easier to explain to users that there is one disk cache, v.s. N.
… re-running the script when you install, deinstall or update lisp code …
I often have very long running lisp processes. So I need to do exactly that via asdf:clear-source-repository etc. Who knows when to update/rebuild the disk cache? Sometimes only the developer. But quicklisp knows when it (un)installes things. I’d be tempted to update the disk cache when that happens. - ben ---- http://enthusiasm.cozy.org tel:+1-781-791-3054 I forecast sunny weather! "Communication rests on a foundation of mutual perspective-taking, the mechanics of which are poorly understood."
My cl-launch scripts can take one second. I can trim a half second off that with a disk-cache[2] of *source-registry*. Tedious details follow. My cl-launch install defaults to using ccl. My ccl install is currently HEAD. My current setup has 3.8K working directories, which compute-source-registry scans. Ok I have three little bash functions[1] to time things. try-ccl- — run ccl with —no-init try-cl — run cl-launch, so no init file; but it will load asdf. try-ccl — run ccl with standard quicklisp init file Without the disk cache. try-ccl- 0.03 seconds try-cl 1 seconds try-ccl 1 seconds With the disk cache: try-ccl- 0.03 seconds try-cl 0.5 seconds try-ccl 0.5 seconds These #’s are gleaned from batches of 10 runs so things can settle. Theories for that remaining half a second need looking into. - ben [1] The three bash functions try-ccl-(){ (time ccl --no-init --eval '(progn (print (lisp-implementation-type)) (quit))') 2>&1 | grep real ; } try-ccl(){ (time ccl --eval '(progn (print (lisp-implementation-type)) (quit))') 2>&1 | grep real ; } try-cl(){ (time cl -ip '(lisp-implementation-type)') 2>&1 | grep real ; } [2] A proof of concept disk cache... https://gist.github.com/bhyde/526c5acb38f341d60dba So, anytime somebody rearranges the set of asdf systems they are expected to invoke reset-quickasdf. quickasdf horrible name - but it bemused me at the time. Calling initialize-source-registry directly from the repl takes 0.47 seconds without the diskcache and .004 seconds with it.
On Wed, Aug 27, 2014 at 1:11 PM, Ben Hyde <bhyde@pobox.com> wrote:
My cl-launch scripts can take one second.
I can trim a half second off that with a disk-cache[2] of *source-registry*.
I'm jealous. Your machine is almost 25% faster than mine.
[2] A proof of concept disk cache...
The cache should ideally be per source-registry entry; and managed by the same entity that manages said entry. Thus, I was thinking of an optional second form in cl-source-registry.conf file. Or a separate .cl-source-registry.cache file.
So, anytime somebody rearranges the set of asdf systems they are expected to invoke reset-quickasdf.
quickasdf horrible name - but it bemused me at the time.
Ouch.
Calling initialize-source-registry directly from the repl takes 0.47 seconds without the diskcache and .004 seconds with it.
Yes, that's the general thing I'm thinking about: kind of back to the concept of link farms of ASDF 1, but only as an optimization, without any additional user configuration required, and leaving system administration to system administrators. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org To surrender to ignorance and call it God has always been premature, and it remains premature today. — Isaac Asimov
On Aug 27, 2014, at 1:35 PM, Faré <fahree@gmail.com> wrote:
On Wed, Aug 27, 2014 at 1:11 PM, Ben Hyde <bhyde@pobox.com> wrote:
My cl-launch scripts can take one second.
I can trim a half second off that with a disk-cache[2] of *source-registry*.
I'm jealous. Your machine is almost 25% faster than mine.
weird, it’s nothing special. I bought something I could hold it open with one hand while hitting staff with the other. http://j.mp/1lfhMwY
[2] A proof of concept disk cache...
The cache should ideally be per source-registry entry; and managed by the same entity that manages said entry. Thus, I was thinking of an optional second form in cl-source-registry.conf file. Or a separate .cl-source-registry.cache file.
I saw that. i’m ambivalent about the complexity.
So, anytime somebody rearranges the set of asdf systems they are expected to invoke reset-quickasdf.
quickasdf horrible name - but it bemused me at the time.
Ouch.
I love achieving my goals.
Calling initialize-source-registry directly from the repl takes
0.47 seconds without the diskcache and .004 seconds with it.
Yes, that's the general thing I'm thinking about: kind of back to the concept of link farms
Any insta-theories for where other half second comes from? - ben "Since English is a mess, it maps well onto the problem space, which is also a mess, which we call reality.” - Larry Wall
The cache should ideally be per source-registry entry; and managed by the same entity that manages said entry. Thus, I was thinking of an optional second form in cl-source-registry.conf file. Or a separate .cl-source-registry.cache file.
I saw that. i’m ambivalent about the complexity.
Checking for .cl-source-registry.cache in collect-sub*directories-asd-files should be straightforward. NB: If you're going to hack these things, please use the faster-registry branch. You don't have an account on common-lisp.net yet, but it looks like, after a year, it's still the same machine that some people were angry about and calling for the replacement of, and I happen to be root there, so I can open you an account and give you write access to the asdf repository (if Robert agrees) — send me your ssh public key.
Any insta-theories for where other half second comes from?
If you mean the second half of cl-launch's startup time, I fear it might be a combination of shell and CL compilation overhead, but I admit I haven't tried timing where the time is going. You're welcome to investigate. At some point, I was thinking of translating cl-launch to CL and having it output a bootstrap script to make itself. Then it would have a "fast" mode that loads code onto the cl-launch image, rather than start a new process (with maybe a different CL implementation) to implement the requested software. At that point, maybe Xach's buildapp is a better starting point. Or not. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org None are more hopelessly enslaved than those who falsely believe they are free. — Johann Wolfgang von Goethe
Faré wrote:
The cache should ideally be per source-registry entry; and managed by the same entity that manages said entry. Thus, I was thinking of an optional second form in cl-source-registry.conf file. Or a separate .cl-source-registry.cache file. I saw that. i’m ambivalent about the complexity.
Checking for .cl-source-registry.cache in collect-sub*directories-asd-files should be straightforward. NB: If you're going to hack these things, please use the faster-registry branch.
I'm having a little trouble keeping track of this discussion. Is there a clear argument for this caching to be a core piece of ASDF instead of a contrib? Could the disk cache be handled by code that massages *system-definition-search-functions* in a contrib? Certainly not as currently defined, but with some rejiggering? Or could we add a means to hook the source registry initialization process? I note also that the quickasdf proposal has no room for people who have multiple *different* lisp configurations. Currently, if you work on different projects, with different library sets, you can change the environment variable to change contexts, you can point your ASDF at different configuration files, or you can use code that changes the value of asdf:*central-registry*. For quickasdf to be an acceptable integration into ASDF proper, it would have to be able to be reconfigurable in exactly the same ways, and guarantee that when users change their current working configuration *in any one of these ways*, they cannot get a some other configuration's disk cache. So: for the moment, no. Quickasdf's disk cache will NOT be integrated into ASDF. There are way too many open questions. See my earlier email manifesto as ASDF maintainer, and new follow on. Best, r
… argument for this caching to be a core piece of ASDF … ?
My apologies, I should have known that saying “proof of concept” wasn’t sufficient to signal that - yeah! , it’s a vile hack not ready to be integrated into anything, including my own day to day usage. In fact I put forward code because that’s how I test that my ideas have any grounding in reality. Good point about how people are likely to be switching configs, not one of they many open issues I’d already noticed.
Any insta-theories for where other half second comes from?
If you mean the second half of cl-launch's startup time, I fear it might be a combination of shell and CL compilation overhead, but I admit I haven't tried timing where the time is going. You're welcome to investigate. At some point, I was thinking of translating cl-launch to CL and having it output a bootstrap script to make itself. Then it would have a "fast" mode that loads code onto the cl-launch image, rather than start a new process (with maybe a different CL implementation) to implement the requested software. At that point, maybe Xach's buildapp is a better starting point. Or not.
The other half of cl-launch's startup time is spent in CL, and is definitely a performance bug: % time env SBCL=/bin/true cl -ip 1 env SBCL=/bin/true cl -ip 1 0.03s user 0.02s system 102% cpu 0.045 total % time env CL_SOURCE_REGISTRY="/does/not/exist" cl -ip 1 env CL_SOURCE_REGISTRY="/does/not/exist" cl -ip 1 0.43s user 0.06s system 98% cpu 0.494 total Inserting a few (time) forms and enabling verbose loading, I find that the cl:load'ing of cl-launch itself is taking most of the time, and the most time in that is loading then upgrading asdf, i.e. loading the fasl's for it. I'm not sure how to speed THAT up; ASDF is big, and not likely to shrink significantly while preserving backward compatibility. SBCL also seems to be spending some notable fraction of the time compiling the cl-launch code itself, though most of the time is the fasl loading. Conclusion: FASt Loading is only so "fast". PS: your topology.sh file has a ":" ; # ... comment as if it were a bilingual script, but it isn't bilingual. Also, the path /usr/local/bin/cl is not standard and neither is the -S argument. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "The Army is a place where you get up early in the morning to be yelled at by people with short haircuts and tiny brains." — Dave Barry
get’n off topic … scripting isn’t asdf ... should find a room. On Aug 28, 2014, at 11:17 AM, Faré <fare@tunes.org> wrote:
PS: your topology.sh file has a ":" ; # ... comment as if it were a bilingual script, but it isn't bilingual.
right. vistigal. cl-shim has arrived. life is getting better.
Also, the path /usr/local/bin/cl is not standard
true, i blame the makefile. look'n forward teaching emacs the mode of these files.
and neither is the -S argument.
not long for this world. cleaning shaved 4 seconds of the run time soon I can remove that damn link farm. standard - snort. tis tacky. thanks for proof reading! new revision[1] - ben [1] https://github.com/bhyde/yed-lisp/blob/master/topology.sh ---- http://enthusiasm.cozy.org tel:+1-781-791-3054 I forecast sunny weather! "The paradox of information retrieval is that a person must describe the information that he or she does not have." -- Christine L. Borgman
On Fri, Aug 29, 2014 at 10:07 AM, Ben Hyde <bhyde@pobox.com> wrote:
get’n off topic … scripting isn’t asdf ... should find a room.
No, YOU go first!
not long for this world. cleaning shaved 4 seconds of the run time soon I can remove that damn link farm.
new revision[1] [1] https://github.com/bhyde/yed-lisp/blob/master/topology.sh I recommend option -sp yed-lisp so you don't need all these
Oh, how did you shave it? Was it a yak? Did you "just" make the link farm your regularly configured source-registry? Do you have a clever patch to asdf? package qualifications. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org To fight a violent enemy, violence is necessary; but to fight violence itself, violence is vain.
I have implemented support for .cl-source-registry.cache in the faster-registry branch, and added a script ./bin/update-cl-source-registry.lisp to create or update cache files, as in: ~/common-lisp/asdf/bin/update-cl-source-registry.lisp ~/common-lisp/ It does significantly speed up the scanning of the source-registry indeed, and thus slash the startup time of cl-launch from "noticeably slow" to "acceptable". Problem: it's only useful if you use the install-asdf-as-module script to override your implementation's ASDF with that from the branch, or else, the old asdf will still spend time scanning the source-registry at startup... except the script seems to be broken in master, unlike in the minimakefile branch. Sigh. So I merged minimakefile into faster-registry. (Note to Robert: that's one more thing broken in master that's working in minimakefile.) There are still more things to speed up in starting cl-launch, though, like the loading (and possibly upgrading) of ASDF, and (on SBCL) priming the CLOS caches for all the ASDF classes and functions. But it looks like they are out of my reach, and they account for most of the .6s of startup left on SBCL. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Hi! I'm a signature virus. Copy me into your sig file and help me spread! On Thu, Aug 28, 2014 at 11:17 AM, Faré <fare@tunes.org> wrote:
Any insta-theories for where other half second comes from?
If you mean the second half of cl-launch's startup time, I fear it might be a combination of shell and CL compilation overhead, but I admit I haven't tried timing where the time is going. You're welcome to investigate. At some point, I was thinking of translating cl-launch to CL and having it output a bootstrap script to make itself. Then it would have a "fast" mode that loads code onto the cl-launch image, rather than start a new process (with maybe a different CL implementation) to implement the requested software. At that point, maybe Xach's buildapp is a better starting point. Or not.
The other half of cl-launch's startup time is spent in CL, and is definitely a performance bug: % time env SBCL=/bin/true cl -ip 1 env SBCL=/bin/true cl -ip 1 0.03s user 0.02s system 102% cpu 0.045 total % time env CL_SOURCE_REGISTRY="/does/not/exist" cl -ip 1 env CL_SOURCE_REGISTRY="/does/not/exist" cl -ip 1 0.43s user 0.06s system 98% cpu 0.494 total
Inserting a few (time) forms and enabling verbose loading, I find that the cl:load'ing of cl-launch itself is taking most of the time, and the most time in that is loading then upgrading asdf, i.e. loading the fasl's for it. I'm not sure how to speed THAT up; ASDF is big, and not likely to shrink significantly while preserving backward compatibility. SBCL also seems to be spending some notable fraction of the time compiling the cl-launch code itself, though most of the time is the fasl loading. Conclusion: FASt Loading is only so "fast".
PS: your topology.sh file has a ":" ; # ... comment as if it were a bilingual script, but it isn't bilingual. Also, the path /usr/local/bin/cl is not standard and neither is the -S argument.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "The Army is a place where you get up early in the morning to be yelled at by people with short haircuts and tiny brains." — Dave Barry
Ben Hyde wrote:
Would changing (:tree <path>) so it accepts an optional argument be less adhoc?
Possibly (:tree <path> &key (depth nil) (don-not-recure-after-asdf-encountered nil))
That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those.
I think the problem with this is that, as Fare points out, it's the author of the system who has laid out the directory tree, and who knows whether or not there are system definitions floating around in it. Similarly, if the library author makes the modification, that's work for one person. If the callers do, it's work for everyone. cheers, r
On Aug 26, 2014, at 3:22 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
Ben Hyde wrote:
Would changing (:tree <path>) so it accepts an optional argument be less adhoc?
Possibly (:tree <path> &key (depth nil) (don-not-recure-after-asdf-encountered nil))
That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those.
I think the problem with this is that, as Fare points out, it's the author of the system who has laid out the directory tree
your both quite wise, i agree
participants (4)
-
Ben Hyde -
Faré -
Faré -
Robert P. Goldman