[Asdf-devel] request for test: experimental-submodules branch
I have tried to pull all the dependencies for ASDF building into the repo. I have done this primarily using git submodules, but also had to pull in cl-ppcre as source, since it doesn't seem to be available as a git repo. If you can, please pull an update, checkout the experimental-submodules branch, and see if it configures properly. All the dependencies should pop up in the ext/ subdirectory (this may not be the best place; I'm open to suggestions for improvement). Please report any problems and Thanks, Robert
On Sun, Aug 24, 2014 at 7:45 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
I have tried to pull all the dependencies for ASDF building into the repo. I have done this primarily using git submodules, but also had to pull in cl-ppcre as source, since it doesn't seem to be available as a git repo.
All ediware has been migrated to github by hhuebner, and cl-ppcre is here: https://github.com/edicl/cl-ppcre Please don't push the cl-ppcre source onto the master branch of asdf.
If you can, please pull an update, checkout the experimental-submodules branch, and see if it configures properly. All the dependencies should pop up in the ext/ subdirectory (this may not be the best place; I'm open to suggestions for improvement).
For the record, in xcvb, I called that directory dependencies/ PS: xcvb has seen no new feature since Summer 2012, a few months before the work on ASDF 3 began. It has been slightly updated to account for ASDF 3, with a lot of library code moved out of XCVB: most xcvb-driver was moved to UIOP; UIOP also ate the previous asdf-utils and made xcvb-utils obsolete. lisp-invocation was spun off. I'm not sure it's worth investing in XCVB anymore at this point; most of its ideas have influenced ASDF 3, and what remains to be done (e.g cross-compilation) can probaby be done more easily on top of ASDF 3 than on top of XCVB. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org There is a difference between knowing the path and walking the path. — Morpheus, in "The Matrix"
Faré wrote:
On Sun, Aug 24, 2014 at 7:45 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
I have tried to pull all the dependencies for ASDF building into the repo. I have done this primarily using git submodules, but also had to pull in cl-ppcre as source, since it doesn't seem to be available as a git repo.
All ediware has been migrated to github by hhuebner, and cl-ppcre is here: https://github.com/edicl/cl-ppcre Please don't push the cl-ppcre source onto the master branch of asdf.
Thanks. Please pull an update, and have a look. I replaced the local copy with a git submodule.
If you can, please pull an update, checkout the experimental-submodules branch, and see if it configures properly. All the dependencies should pop up in the ext/ subdirectory (this may not be the best place; I'm open to suggestions for improvement).
For the record, in xcvb, I called that directory dependencies/
It's ext/ everywhere at SIFT. I'm inclined to keep the shorter name, unless there's some generally accepted practice to replace it with. I changed by ASDF_DEVEL_SOURCE_REGISTRY to point to the local submodules, and this seems to work adequately, but I don't have great tests. I'll see if this gets me through a next bug fix, which will test my use of the build scriptage. If so, I will merge the branch, and use it as a basis for merging minimakefile. Cheers, r
On Mon, Aug 25, 2014 at 9:27 AM, Robert P. Goldman <rpgoldman@sift.info> wrote:
Thanks. Please pull an update, and have a look. I replaced the local copy with a git submodule.
Looks like it works for me. You should update the README with instructions on how to use this feature, i.e. do git submodule update and export ASDF_DEVEL_SOURCE_REGISTRY, or whatever the trick is.
If you can, please pull an update, checkout the experimental-submodules branch, and see if it configures properly. All the dependencies should pop up in the ext/ subdirectory (this may not be the best place; I'm open to suggestions for improvement).
For the record, in xcvb, I called that directory dependencies/
It's ext/ everywhere at SIFT. I'm inclined to keep the shorter name, unless there's some generally accepted practice to replace it with.
I don't have strong feelings about it. I like short names. I don't think ext/ is very obvious — at the very least, the README should mention it.
I changed by ASDF_DEVEL_SOURCE_REGISTRY to point to the local submodules, and this seems to work adequately, but I don't have great tests.
Actually, the asdf initialization routine could unconditionally add the asdf tree to the path: either it's present or not, but it doesn't hurt to try, and so no need to export ASDF_DEVEL_SOURCE_REGISTRY, then. If you're going to merge in minimakefile, you only need to make this change after the merge, rather than twice (once before and once again after).
I'll see if this gets me through a next bug fix, which will test my use of the build scriptage. If so, I will merge the branch, and use it as a basis for merging minimakefile.
Wonderful! PS: meanwhile, Christoph Egger just blessed the asdf 3.1.3 and cl-launch 4.1 packages into debian unstable. Yay. In a few years, Common Lisp scripting will be universally available. How funny would it be that 10 years from now, Perl is a forgotten scripting language, but CL is doing strong? —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Yesterday I was clever, so I wanted to change the world. Today I am wise, so I am changing myself. — Rumi (1207–1273)
Faré wrote:
On Mon, Aug 25, 2014 at 9:27 AM, Robert P. Goldman <rpgoldman@sift.info> wrote: [...snip...]
I changed by ASDF_DEVEL_SOURCE_REGISTRY to point to the local submodules, and this seems to work adequately, but I don't have great tests.
Actually, the asdf initialization routine could unconditionally add the asdf tree to the path: either it's present or not, but it doesn't hurt to try, and so no need to export ASDF_DEVEL_SOURCE_REGISTRY, then.
I'm not entirely sure about this. Here's my question: if someone is developing code in one of those modules, and wants to see if it messes up ASDF development, might they want to override this to point to private copies of some or all of the dependencies? In that case, maybe I should leave it around.... This would also be a place to test your new quicker-loading option. Very much open to suggestions. Best, r
Actually, the asdf initialization routine could unconditionally add the asdf tree to the path: either it's present or not, but it doesn't hurt to try, and so no need to export ASDF_DEVEL_SOURCE_REGISTRY, then.
I'm not entirely sure about this. Here's my question: if someone is developing code in one of those modules, and wants to see if it messes up ASDF development, might they want to override this to point to private copies of some or all of the dependencies? In that case, maybe I should leave it around....
Another option: leave it around, but have it default to the ASDF source tree. Best of both worlds? OR, he could temporarily move away the ext/ directory while conducting his tests. As usual, I like being able to configure, but I like even more being able to NOT configure.
This would also be a place to test your new quicker-loading option.
Certainly.
Very much open to suggestions.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org When you are young you are afraid people will steal your ideas; when you are old you are afraid they won't. — David D. Friedman
Faré wrote:
Actually, the asdf initialization routine could unconditionally add the asdf tree to the path: either it's present or not, but it doesn't hurt to try, and so no need to export ASDF_DEVEL_SOURCE_REGISTRY, then. I'm not entirely sure about this. Here's my question: if someone is developing code in one of those modules, and wants to see if it messes up ASDF development, might they want to override this to point to private copies of some or all of the dependencies? In that case, maybe I should leave it around....
Another option: leave it around, but have it default to the ASDF source tree. Best of both worlds?
Yes, that's what I did. I just pushed a new version of experimental-submodules. If it looks good, I'll merge it into master and then we can see about merging it into minimakefile...
As usual, I like being able to configure, but I like even more being able to NOT configure.
Agreed. This should achieve that objective, I hope.
This would also be a place to test your new quicker-loading option.
Certainly.
OK, I'll look into that next.
Another option: leave it around, but have it default to the ASDF source tree. Best of both worlds?
Yes, that's what I did. I just pushed a new version of experimental-submodules. If it looks good, I'll merge it into master and then we can see about merging it into minimakefile...
Looks good to me. One command that helps (and that you can put in your Makefile for debugging): ./bin/asdf-builder re '(uiop:writeln (sort (alexandria:hash-table-alist asdf::*source-registry*) `string< :key `car))'
This would also be a place to test your new quicker-loading option. Certainly.
OK, I'll look into that next.
I pushed a branch faster-registry. The code change is a fix to uiop and a disabled-by-default special-variable-controlled option to not recurse once a .asd file is found. There is also a TODO entry based on my previous comments on asdf-devel: I believe the solution is to implement support for a [.]?cl-source-registry[.]conf file that asdf consults when searching a :tree + a script (part of asdf-tools?) to for developers, package managers and system administrators to automatically generate such a file when desired. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Honor among thieves is the ancestor of all honor. — John McCarthy
Faré wrote:
Another option: leave it around, but have it default to the ASDF source tree. Best of both worlds? Yes, that's what I did. I just pushed a new version of experimental-submodules. If it looks good, I'll merge it into master and then we can see about merging it into minimakefile...
Looks good to me. One command that helps (and that you can put in your Makefile for debugging):
./bin/asdf-builder re '(uiop:writeln (sort (alexandria:hash-table-alist asdf::*source-registry*) `string< :key `car))'
Thanks. BTW, isn't my ROOT_DIR definition a good replacement for your sourceDirectory? I believe the latter could fail if the Makefile is invoked from another directory.... Actually, your command shows a challenge with using the full ASDF source tree: it exposes all the test systems. I will fix this. Cheers, r
Thanks. BTW, isn't my ROOT_DIR definition a good replacement for your sourceDirectory? I believe the latter could fail if the Makefile is invoked from another directory....
sourceDirectory is actually gwking's. Did five years already go by? But to answer your question, make -C .../asdf/ will change the current directory, so taking the pwd is ok. Of course make -f .../asdf/Makefile from a different directory will do the wrong thing, but that's on purpose.
Actually, your command shows a challenge with using the full ASDF source tree: it exposes all the test systems.
That's a slight argument for stopping recursion at a directory containing a .asd and/or consulting a cl-source-registry.conf file: so you can have private .asd's that are only visible when you explicitly tweak your source-registry or central-registry, during tests or otherwise controlled builds.
I will fix this.
I'm not sure what you intend to do. Prefix all the system names with test-asdf or such? I admit that was one of the reasons for me to reduce the number of actual .asd files in our test directory: when possible, either use define-test-system in a .script, or use secondary systems of test-asdf.asd. But I didn't go through all existing files to fix them; I only made such changes as I was fixing other bugs. Maybe ongoing cleanups like this ought to be documented in a TODO entry. Regards, —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Judge and party — the ultimate nature of a monopoly government.
Faré wrote:
Thanks. BTW, isn't my ROOT_DIR definition a good replacement for your sourceDirectory? I believe the latter could fail if the Makefile is invoked from another directory....
sourceDirectory is actually gwking's. Did five years already go by?
But to answer your question, make -C .../asdf/ will change the current directory, so taking the pwd is ok. Of course make -f .../asdf/Makefile from a different directory will do the wrong thing, but that's on purpose.
On purpose? Why? Is there some reason it's *wrong* for me to make the file robust to being invoked from a different directory? I thought a reason to make this robust is in case someone grabs ASDF, puts it into a larger system (e.g., a lisp implementation), and does a make at a top level that invokes make in asdf.... But I'm not a make expert.
Actually, your command shows a challenge with using the full ASDF source tree: it exposes all the test systems.
That's a slight argument for stopping recursion at a directory containing a .asd and/or consulting a cl-source-registry.conf file: so you can have private .asd's that are only visible when you explicitly tweak your source-registry or central-registry, during tests or otherwise controlled builds.
I will fix this.
I'm not sure what you intend to do. Prefix all the system names with test-asdf or such? I admit that was one of the reasons for me to reduce the number of actual .asd files in our test directory: when possible, either use define-test-system in a .script, or use secondary systems of test-asdf.asd. But I didn't go through all existing files to fix them; I only made such changes as I was fixing other bugs.
Maybe ongoing cleanups like this ought to be documented in a TODO entry.
I did something much simpler: I enumerated the directories instead of just loading the whole tree. My current settting is: export CL_SOURCE_REGISTRY = ${sourceDirectory}/:${sourceDirectory}/uiop/:${sourceDirectory}/ext//: That gets only ASDF, UIOP, and the dependencies. Cheers, r
Thanks. BTW, isn't my ROOT_DIR definition a good replacement for your sourceDirectory? I believe the latter could fail if the Makefile is invoked from another directory....
sourceDirectory is actually gwking's. Did five years already go by?
But to answer your question, make -C .../asdf/ will change the current directory, so taking the pwd is ok. Of course make -f .../asdf/Makefile from a different directory will do the wrong thing, but that's on purpose.
On purpose? Why? Is there some reason it's *wrong* for me to make the file robust to being invoked from a different directory?
I thought a reason to make this robust is in case someone grabs ASDF, puts it into a larger system (e.g., a lisp implementation), and does a make at a top level that invokes make in asdf....
But I'm not a make expert.
Once again, the standard way of invoking a makefile is to change the current directory and call make, which is what GNU make does with make -C directory. Therefore $(shell pwd) will work (it's a GNU extension, BTW, since the online BSD make man page doesn't document it). Option -f is not meant for running a makefile in a different directory, but for selecting a makefile for the current directory, when e.g. you need to distinguish between Windows and Unix, between BSD make and GNU make, etc.
I did something much simpler: I enumerated the directories instead of just loading the whole tree. My current settting is:
export CL_SOURCE_REGISTRY = ${sourceDirectory}/:${sourceDirectory}/uiop/:${sourceDirectory}/ext//:
That gets only ASDF, UIOP, and the dependencies.
I saw that. This doesn't help when asdf is in the source-registry, though (which is the recommended way of having an asdf upgrade: "just" having its source in the source-registry, e.g. in ~/common-lisp/asdf/) —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I have many lucky numbers: 45, 357, 9, etc etc etc. but 911 is NOT one of them — I440r
Faré wrote:
I did something much simpler: I enumerated the directories instead of just loading the whole tree. My current settting is:
export CL_SOURCE_REGISTRY = ${sourceDirectory}/:${sourceDirectory}/uiop/:${sourceDirectory}/ext//:
That gets only ASDF, UIOP, and the dependencies.
I saw that. This doesn't help when asdf is in the source-registry, though (which is the recommended way of having an asdf upgrade: "just" having its source in the source-registry, e.g. in ~/common-lisp/asdf/)
Hm. And we have to have the asdf *tree* in the source-registry instead of only having the asdf *directory* in there because of the separation of uiop.asd and asdf.asd. Is there some reason we can't move uiop.asd up to a position next to asdf.asd? That seems like a much more pleasant solution than restructuring the repo to make the test and asdf source directories siblings instead of child/parent. R
I saw that. This doesn't help when asdf is in the source-registry, though (which is the recommended way of having an asdf upgrade: "just" having its source in the source-registry, e.g. in ~/common-lisp/asdf/)
Hm. And we have to have the asdf *tree* in the source-registry instead of only having the asdf *directory* in there because of the separation of uiop.asd and asdf.asd.
Is there some reason we can't move uiop.asd up to a position next to asdf.asd?
That seems like a much more pleasant solution than restructuring the repo to make the test and asdf source directories siblings instead of child/parent.
The reason is to make it easier to package uiop in a separate tarball, as used by e.g. quicklisp. If we didn't have to care for Windows, I'd have suggested a symlink in the top directory, or a link farm directory somewhere. But we do have to support Windows, don't we? What does git do with symlinks on Windows? Are .lnk files supported on all Windows implementations? Meh. Another trick might have been a trampoline uiop.asd that just calls (asdf::load-asd (subpathname *load-pathname* "uiop/uiop.asd")) or maybe just (load ...) since the context was already setup by whichever load-asd got us there (hopefully). Or we could play a game, with asdf-driver.asd being in the top directory and doing the load-asd trick, whereas uiop remains invisible by default. Sigh. That said, we could also move the defsystem files to a defsystem/ subdirectory, making the systems siblings, and being happy that way. Sometimes I think I should have done that before I released asdf3. Only inertia made me not do it (when asdf was a single file, it didn't matter; when I split it into many files, they were in the same system; when it appeared that a large fraction of the code was a separate portability library, I left it in the same directory at first, just a different system; when I separately packaged it and renamed it uiop as prompted by Xach, I moved just uiop to a different directory but left the .asd at the same place, and the rest of the defsystem with it.) If you're going to move files, merging branches will be "interesting". Best done by merging branches with pre-move master, then merging the move step, then merging what follows. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Dost thou love life? Then do not squander time, for that's the stuff life is made of. — Benjamin Franklin
Faré wrote:
I saw that. This doesn't help when asdf is in the source-registry, though (which is the recommended way of having an asdf upgrade: "just" having its source in the source-registry, e.g. in ~/common-lisp/asdf/) Hm. And we have to have the asdf *tree* in the source-registry instead of only having the asdf *directory* in there because of the separation of uiop.asd and asdf.asd.
Is there some reason we can't move uiop.asd up to a position next to asdf.asd?
That seems like a much more pleasant solution than restructuring the repo to make the test and asdf source directories siblings instead of child/parent.
The reason is to make it easier to package uiop in a separate tarball, as used by e.g. quicklisp.
If we didn't have to care for Windows, I'd have suggested a symlink in the top directory, or a link farm directory somewhere. But we do have to support Windows, don't we? What does git do with symlinks on Windows? Are .lnk files supported on all Windows implementations? Meh. Another trick might have been a trampoline uiop.asd that just calls (asdf::load-asd (subpathname *load-pathname* "uiop/uiop.asd")) or maybe just (load ...) since the context was already setup by whichever load-asd got us there (hopefully). Or we could play a game, with asdf-driver.asd being in the top directory and doing the load-asd trick, whereas uiop remains invisible by default. Sigh.
That said, we could also move the defsystem files to a defsystem/ subdirectory, making the systems siblings, and being happy that way.
So defsystem/ would contain asdf.asd, pointing to files in ../ and uiop.asd pointing to files in ../uiop/ ? Or would we move the source files, as well?
If you're going to move files, merging branches will be "interesting". Best done by merging branches with pre-move master, then merging the move step, then merging what follows.
I promise not to do any big reorganization without extensive checking! I think we can dodge this problem for a start, though. If one is doing the build process, and using the makefile, presumably one is sophisticated enough *not* to put the ASDF source tree into your default source registry. If you must do that, well, that's what the ASDF_DEVEL_SOURCE_REGISTRY is for: you must use it to specify for yourself a well-behaved source registry for development of ASDF. [I think part of the reason this doesn't bother me (and part of the reason I haven't liked the conf.d approach to setting things up) is that I have *many* different source registry configurations, for different projects that use different libraries and even different versions of the same library. I have never had a single, universal Lisp development environment, and don't expect that I ever will.] If we can dodge this for now, we can postpone dealing with it until we have closed out the current topic branches. Cheers, r
On Tue, Aug 26, 2014 at 7:06 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
That said, we could also move the defsystem files to a defsystem/ subdirectory, making the systems siblings, and being happy that way.
So defsystem/ would contain asdf.asd, pointing to files in ../ and uiop.asd pointing to files in ../uiop/ ?
Or would we move the source files, as well?
No pointing to files in ../ — indeed move the defsystem files in defsystem/ with asdf.asd, leave the uiop files in uiop/ with uiop.asd. Follow the asdf principle that you name files based on the system, and you locate the system thanks to the .asd file.
I think we can dodge this problem for a start, though. If one is doing the build process, and using the makefile, presumably one is sophisticated enough *not* to put the ASDF source tree into your default source registry. If you must do that, well, that's what the ASDF_DEVEL_SOURCE_REGISTRY is for: you must use it to specify for yourself a well-behaved source registry for development of ASDF.
I suppose the Makefile, even in the minimakefile branch, would have a target that does the git submodule update. For the record, I have always had asdf in my source registry; at times when testing cases where it shouldn't be, I renamed the asdf.asd away to asdf.asd.bak, or deleted it to get it back with git checkout asdf.asd
[I think part of the reason this doesn't bother me (and part of the reason I haven't liked the conf.d approach to setting things up) is that I have *many* different source registry configurations, for different projects that use different libraries and even different versions of the same library. I have never had a single, universal Lisp development environment, and don't expect that I ever will.]
I admit I usually have one "main" configuration for open source development, and when I was at work, the respective projects had their own streamlined build scripts that properly initialized the source registry, and from SLIME, I only compiled files at a time. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org To send men to the firing squad, judicial proof is unnecessary... These procedures are an archaic bourgeois detail. This is a revolution! And a revolutionary must become a cold killing machine motivated by pure hate. — Che Guevara
If we can dodge this for now, we can postpone dealing with it until we have closed out the current topic branches.
Cheers, r
Windows? Are .lnk files supported on all Windows implementations? Meh.
FTR, windoze supports proper symlinks on NTFS since Vista. https://en.wikipedia.org/wiki/NTFS_symbolic_link -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “The Army is a place where you get up early in the morning to be yelled at by people with short haircuts and tiny brains.” — Dave Barry
On Wed, Aug 27, 2014 at 3:54 AM, Attila Lendvai <attila.lendvai@gmail.com> wrote:
Windows? Are .lnk files supported on all Windows implementations? Meh.
FTR, windoze supports proper symlinks on NTFS since Vista.
And since that's been standard since 2006 (wow, time flies!), we can assume it's just there. http://en.wikipedia.org/wiki/List_of_default_file_systems —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "I'm going to live forever, or die trying!" — Spider Robinson
participants (3)
-
Attila Lendvai -
Faré -
Robert P. Goldman