Issues with new testing scripts
I'm having two issues with the new testing scripts: 1. The testing code is bleeding quicklisp into context: make test ./make.sh l='' L='' u='' U='' v='' s='' t='' test
Error: System "cl-scripting" not found While executing: (:INTERNAL QUICKLISP-CLIENT::RECURSE QUICKLISP-CLIENT::COMPUTE-LOAD-STRATEGY), in process listener(1). Type :GO to continue, :POP to abort, :R for a list of available restarts. If continued: Try again Type :? for other options. 1 >
I *definitely* did not ask to be using quicklisp when testing. This error arises inside (LOAD-SYS "asdf-tools"), which is redirected to use quicklisp: (defun load-sys (system) (if (featurep :quicklisp) (let ((*standard-output* (make-broadcast-stream))) ;; be quiet (symbol-call :ql :quickload system)) (load-system system :verbose nil))) This is done by tools/load-asdf.lisp AFAICT, this is done UNCONDITIONALLY BY LOAD-AND-CONFIGURE-ASDF This should not be. The user should get quicklisp only upon request: we added the dependencies to ext/ in order to make the ASDF distribution free-standing and avoid the quicklisp requirement. In my case, I use quicklisp only sometimes, and never yet with CCL, so having the ASDF test script go poking around for a QL installation and firing it up without asking me was problematic. Particularly problematic was that there's no (apparent) way to STOP the ASDF scripts from doing this. For now I am going to push a modification that comments out TRY-LOAD-QUICKLISP from LOAD-AND-CONFIGURE-ASDF. I have no objection to a patch that provides an *explicit option* to load from QL, but it must be an option, and it must not be the default. Frankly, even then I'm not excited about it. I don't want to have bugreports to ASDF that are caused by issues in Quicklisp. I'd rather people stick to using the dependencies in our submodules. 2. This was just a nuisance: three new submodules have been added, and "git submodule init" must be re-run when that happens. I have no idea why the git maintainers thought that a call to git submodule update should quietly fail to update un-initialized submodules (fail to update yes, QUIETLY fail to update, no). But going forward we need to trumpet any changes to the submodules because of this attribute of git.... Cheers, r
On Sat, Dec 19, 2015 at 5:52 PM, Robert Goldman <rpgoldman@sift.net> wrote:
I'm having two issues with the new testing scripts:
1. The testing code is bleeding quicklisp into context:
make test ./make.sh l='' L='' u='' U='' v='' s='' t='' test
Error: System "cl-scripting" not found While executing: (:INTERNAL QUICKLISP-CLIENT::RECURSE QUICKLISP-CLIENT::COMPUTE-LOAD-STRATEGY), in process listener(1). Type :GO to continue, :POP to abort, :R for a list of available restarts. If continued: Try again Type :? for other options. 1 >
I *definitely* did not ask to be using quicklisp when testing.
This error arises inside (LOAD-SYS "asdf-tools"), which is redirected to use quicklisp:
(defun load-sys (system) (if (featurep :quicklisp) (let ((*standard-output* (make-broadcast-stream))) ;; be quiet (symbol-call :ql :quickload system)) (load-system system :verbose nil)))
This is done by tools/load-asdf.lisp
AFAICT, this is done UNCONDITIONALLY BY LOAD-AND-CONFIGURE-ASDF
This should not be. The user should get quicklisp only upon request: we added the dependencies to ext/ in order to make the ASDF distribution free-standing and avoid the quicklisp requirement.
In my case, I use quicklisp only sometimes, and never yet with CCL, so having the ASDF test script go poking around for a QL installation and firing it up without asking me was problematic. Particularly problematic was that there's no (apparent) way to STOP the ASDF scripts from doing this.
For now I am going to push a modification that comments out TRY-LOAD-QUICKLISP from LOAD-AND-CONFIGURE-ASDF.
Yeah, asdf-tools was unconditionally loading quicklisp if found in ~/quicklisp/ or ~/.quicklisp/, with the understanding that the asdf configuration is having ext/ take precedence over quicklisp, so quicklisp auto-installation should only be invoked if you decided not to make ext. I see that you commented out the loading of quicklisp. That's a valid stance. Either way, the behavior and requirements should probably be better documented in the README.md.
I have no objection to a patch that provides an *explicit option* to load from QL, but it must be an option, and it must not be the default.
Frankly, even then I'm not excited about it. I don't want to have bugreports to ASDF that are caused by issues in Quicklisp. I'd rather people stick to using the dependencies in our submodules.
2. This was just a nuisance: three new submodules have been added, and "git submodule init" must be re-run when that happens.
I have no idea why the git maintainers thought that a call to git submodule update should quietly fail to update un-initialized submodules (fail to update yes, QUIETLY fail to update, no). But going forward we need to trumpet any changes to the submodules because of this attribute of git....
The git UI is deficient in many subtle ways. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The urge to save humanity is almost always a false front for the urge to rule. — H. L. Mencken
Hi, On 2015-12-19 23:52 CET, Robert Goldman <rpgoldman@sift.net> wrote:
I'm having two issues with the new testing scripts:
[...]
2. This was just a nuisance: three new submodules have been added, and "git submodule init" must be re-run when that happens.
I have no idea why the git maintainers thought that a call to git submodule update should quietly fail to update un-initialized submodules (fail to update yes, QUIETLY fail to update, no). But going forward we need to trumpet any changes to the submodules because of this attribute of git....
This is the reason why I prefer 'git subtree' to 'git submodule': with subtree the content of the ext/... directories become an integral part of the parent repo. A simple git pull/git fetch is enough to receive the dependencies at the correct version. At the same time, it is possible to 'split out' the sub-repo, e. g. to push a patch to upstream. Kambiz
This is the reason why I prefer 'git subtree' to 'git submodule': with
oh, git... why didn't i expect to have more than one way to implement the same thing? as opposed to e.g. having a flag that switches between the two modes of the same abstraction, both documented at the same place... maybe? thanks for the wakeup call! :) -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “Unthinking respect for authority is the greatest enemy of truth.” — Albert Einstein (1879–1955)
On 12/28/15 Dec 28 -11:57 AM, Attila Lendvai wrote:
This is the reason why I prefer 'git subtree' to 'git submodule': with
oh, git... why didn't i expect to have more than one way to implement the same thing? as opposed to e.g. having a flag that switches between the two modes of the same abstraction, both documented at the same place... maybe?
thanks for the wakeup call! :)
I just read the subtree tutorial, and I'm not convinced that this is any simpler (except that it avoids the problem of you mistakenly not ending up with your submodules empty instead of cloned. I quote from the tutorial: "Notice the commits near the highlighted area. See that the commits are repeated? This is git doing its magic on the pull with subtree strategy. It will bring all the commits from the other repo and will stay together with your own repo’s versions of it. "For this reason, it’s good practice to use the “squash” option when merging changes back." The problem of having clones not get auto-populated seems not worse than needing to do a magical incantation when you merge stuff back and forth at the cost of garbling your repo. The submodules cause annoying quiet failures, but even when the annoying quiet failures happen, you don't end up mangling your repo. A distributed, versioned object database is just hard. And the git UI isn't making it easier. Best, r
On 2015-12-29 01:58 CET, Robert Goldman <rpgoldman@sift.net> wrote:
I just read the subtree tutorial, and I'm not convinced that this is any simpler (except that it avoids the problem of you mistakenly not ending up with your submodules empty instead of cloned.
I quote from the tutorial:
"Notice the commits near the highlighted area. See that the commits are repeated? This is git doing its magic on the pull with subtree strategy. It will bring all the commits from the other repo and will stay together with your own repo’s versions of it.
"For this reason, it’s good practice to use the “squash” option when merging changes back."
The problem of having clones not get auto-populated seems not worse than needing to do a magical incantation when you merge stuff back and forth at the cost of garbling your repo.
The complete magical incantation reads: git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master and to update it to the lastest master, replace 'add' with 'pull'.
The submodules cause annoying quiet failures, but even when the annoying quiet failures happen, you don't end up mangling your repo.
The problem with submodules is that every single person who checks out the repo is concerned with it, whereas git subtree has to be performed only by one single person and for all others it just works without them even knowing that there is a 'subtree' being used somewhere.
A distributed, versioned object database is just hard. And the git UI isn't making it easier.
I use VCS since 1989 and have never had an easier way of tracking vendor branches than this. Cheers Kambiz
On 12/31/15 Dec 31 -5:46 AM, Kambiz Darabi wrote:
On 2015-12-29 01:58 CET, Robert Goldman <rpgoldman@sift.net> wrote:
I just read the subtree tutorial, and I'm not convinced that this is any simpler (except that it avoids the problem of you mistakenly not ending up with your submodules empty instead of cloned.
I quote from the tutorial:
"Notice the commits near the highlighted area. See that the commits are repeated? This is git doing its magic on the pull with subtree strategy. It will bring all the commits from the other repo and will stay together with your own repo’s versions of it.
"For this reason, it’s good practice to use the “squash” option when merging changes back."
The problem of having clones not get auto-populated seems not worse than needing to do a magical incantation when you merge stuff back and forth at the cost of garbling your repo.
The complete magical incantation reads:
git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master
and to update it to the lastest master, replace 'add' with 'pull'.
The submodules cause annoying quiet failures, but even when the annoying quiet failures happen, you don't end up mangling your repo.
The problem with submodules is that every single person who checks out the repo is concerned with it, whereas git subtree has to be performed only by one single person and for all others it just works without them even knowing that there is a 'subtree' being used somewhere.
Can you explain that stuff about needing to squash? I read the subtree tutorial and that discussion just seemed like gibberish to me. Very far from clear. Maybe it's easier than the discussion made it seem. As I read it, it seemed to be saying that if you didn't remember to do --squash at the right times you would end up with a repo clogged with duplicate commits. If that's the case...ugh. But perhaps it was just a bad explanation? Or does your subtree add expression above ensure that the squashing becomes the default? If we could figure this out maybe subtrees would be easier. But if you have to remember to squash every time, forget it....
A distributed, versioned object database is just hard. And the git UI isn't making it easier.
I use VCS since 1989 and have never had an easier way of tracking vendor branches than this.
FWIW, the subversion externals seem *much* easier to use. But subversion is centralized, which makes this problem much easier, so it's not a fair comparison. Thanks, r
Happy New Year, On 2015-12-31 16:12 CET, Robert Goldman <rpgoldman@sift.net> wrote:
On 12/31/15 Dec 31 -5:46 AM, Kambiz Darabi wrote:
On 2015-12-29 01:58 CET, Robert Goldman <rpgoldman@sift.net> wrote:
The problem of having clones not get auto-populated seems not worse than needing to do a magical incantation when you merge stuff back and forth at the cost of garbling your repo.
The complete magical incantation reads:
git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master
and to update it to the lastest master, replace 'add' with 'pull'.
The submodules cause annoying quiet failures, but even when the annoying quiet failures happen, you don't end up mangling your repo.
The problem with submodules is that every single person who checks out the repo is concerned with it, whereas git subtree has to be performed only by one single person and for all others it just works without them even knowing that there is a 'subtree' being used somewhere.
Can you explain that stuff about needing to squash? I read the subtree tutorial and that discussion just seemed like gibberish to me. Very far from clear. Maybe it's easier than the discussion made it seem. As I read it, it seemed to be saying that if you didn't remember to do --squash at the right times you would end up with a repo clogged with duplicate commits.
If that's the case...ugh. But perhaps it was just a bad explanation? Or does your subtree add expression above ensure that the squashing becomes the default?
My command above does contain the '--squash' option. The git-subtree author just chose the wrong default: usually, you are not interested in seeing each and every commit in the external repo. But again, if you do the wrong thing, it's just a matter of: git reset --hard HEAD~2 git subtree add/pull --squash ...
If we could figure this out maybe subtrees would be easier. But if you have to remember to squash every time, forget it....
IMO such commands should be automated with a makefile entry, so you wouldn't have to remember the --squash option.
A distributed, versioned object database is just hard. And the git UI isn't making it easier.
I use VCS since 1989 and have never had an easier way of tracking vendor branches than this.
FWIW, the subversion externals seem *much* easier to use. But subversion is centralized, which makes this problem much easier, so it's not a fair comparison.
I remember there was also some incantation with svn propset/propget, you had to commit separately if you changed something in the vendor branch. And we also had problems with tagging and branching which I'm happy not to remember any more. Cheers Kambiz
On 2016-01-01 09:59 CET, Kambiz Darabi <darabi@m-creations.com> wrote:
Happy New Year,
On 2015-12-31 16:12 CET, Robert Goldman <rpgoldman@sift.net> wrote:
On 12/31/15 Dec 31 -5:46 AM, Kambiz Darabi wrote:
On 2015-12-29 01:58 CET, Robert Goldman <rpgoldman@sift.net> wrote:
The problem of having clones not get auto-populated seems not worse than needing to do a magical incantation when you merge stuff back and forth at the cost of garbling your repo.
The complete magical incantation reads:
git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master
and to update it to the lastest master, replace 'add' with 'pull'.
The submodules cause annoying quiet failures, but even when the annoying quiet failures happen, you don't end up mangling your repo.
The problem with submodules is that every single person who checks out the repo is concerned with it, whereas git subtree has to be performed only by one single person and for all others it just works without them even knowing that there is a 'subtree' being used somewhere.
Can you explain that stuff about needing to squash? I read the subtree tutorial and that discussion just seemed like gibberish to me. Very far from clear. Maybe it's easier than the discussion made it seem. As I read it, it seemed to be saying that if you didn't remember to do --squash at the right times you would end up with a repo clogged with duplicate commits.
If that's the case...ugh. But perhaps it was just a bad explanation? Or does your subtree add expression above ensure that the squashing becomes the default?
My command above does contain the '--squash' option. The git-subtree author just chose the wrong default: usually, you are not interested in seeing each and every commit in the external repo.
But again, if you do the wrong thing, it's just a matter of:
git reset --hard HEAD~2 git subtree add/pull --squash ...
If we could figure this out maybe subtrees would be easier. But if you have to remember to squash every time, forget it....
IMO such commands should be automated with a makefile entry, so you wouldn't have to remember the --squash option.
I pushed a branch to my repo which contains the necessary changes: https://gitlab.common-lisp.net/darabi/asdf/tree/ext-subtree The specification of what to get where and which version is in ext/dependencies.lisp-expr and git checkout ext-subtree make ext does the job. Cheers Kambiz
On Fri, Jan 1, 2016 at 7:18 AM, Kambiz Darabi <darabi@m-creations.com> wrote:
I pushed a branch to my repo which contains the necessary changes:
https://gitlab.common-lisp.net/darabi/asdf/tree/ext-subtree
The specification of what to get where and which version is in ext/dependencies.lisp-expr
and
git checkout ext-subtree make ext
does the job.
Wait, if you still need to make ext, what is the advantage of subtree vs submodules? One thing I *like* about submodules, is that they are optional. So if I already have regular checkouts of libraries (and I do), I can use them instead of those of make ext. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Surprise exists in the map, not in the territory. — Eliezer Yudkowsky
On 2016-01-01 18:12 CET, Faré <fahree@gmail.com> wrote:
On Fri, Jan 1, 2016 at 7:18 AM, Kambiz Darabi <darabi@m-creations.com> wrote:
I pushed a branch to my repo which contains the necessary changes:
https://gitlab.common-lisp.net/darabi/asdf/tree/ext-subtree
The specification of what to get where and which version is in ext/dependencies.lisp-expr
and
git checkout ext-subtree make ext
does the job.
Wait, if you still need to make ext, what is the advantage of subtree vs submodules?
You don't need to 'make ext' as someone who just checks out the repo. A developer performs 'make ext' once in a while to update to current versions of the dependencies, run the tests and then push the new version of the dependencies. Or to reset a buggy HEAD of one of the dependencies to a known working version.
One thing I *like* about submodules, is that they are optional. So if I already have regular checkouts of libraries (and I do), I can use them instead of those of make ext.
IMO it would be better to have fixed versions of the dependencies which are used by each and every developer and packager. This is in my experience the basis of a solid and reliable build. Kambiz
One thing I *like* about submodules, is that they are optional. So if I already have regular checkouts of libraries (and I do), I can use them instead of those of make ext.
IMO it would be better to have fixed versions of the dependencies which are used by each and every developer and packager.
This is in my experience the basis of a solid and reliable build.
Yes, BUT, subtree makes it harder to distribute asdf independently from other libraries, which is very, very, very bad. People who use asdf for any serious purpose whatsoever already have their own library layout, packaging and versioning. The last thing they want is to deal with headaches because asdf has another set of copies of some of the libraries that non-deterministically may or may not appear first when recursively scanning a tree. OK, since 3.1.5, the .cl-source-registry.cache should make that not a problem anymore. Still a potential source of confusion and space waste. One solution would be to have a "raw" repository for asdf alone, and a separate repository asdf-dev that uses git subtree to provide asdf and all its dependencies together. submodules so far provided a compromise whereby the asdf git repository could be used either with or without submodules. Awkward either way, but works. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Eric Schmidt rule of management: always assume that the other person is as intelligent as you are.
On 2016-01-02 09:05 CET, Faré <fahree@gmail.com> wrote:
One thing I *like* about submodules, is that they are optional. So if I already have regular checkouts of libraries (and I do), I can use them instead of those of make ext.
IMO it would be better to have fixed versions of the dependencies which are used by each and every developer and packager.
This is in my experience the basis of a solid and reliable build.
Yes, BUT, subtree makes it harder to distribute asdf independently from other libraries, which is very, very, very bad. People who use asdf for any serious purpose whatsoever already have their own library layout, packaging and versioning.
I agree fully. Only developers/packagers should need the dependencies in ext/ and only for building and testing asdf and for nothing else.
The last thing they want is to deal with headaches because asdf has another set of copies of some of the libraries that non-deterministically may or may not appear first when recursively scanning a tree. OK, since 3.1.5, the .cl-source-registry.cache should make that not a problem anymore. Still a potential source of confusion and space waste.
One solution would be to have a "raw" repository for asdf alone, and a separate repository asdf-dev that uses git subtree to provide asdf and all its dependencies together.
Two repositories are maybe too much hassle. We could also have a master branch which doesn't contain ext/* and is only updated with a new release. Users who just git clone https://gitlab.../asdf.git would end up with the latest stable version which doesn't contain the ext/ dependencies and therefore don't risk to use them when asdf is checked out in a tree which is scanned recursively. Developers and implementation vendors should be warned explicitly about this change and instructed to check out the correct branch before building/testing. And asdf-tools can check for existence of the dependencies in ext and warn the developers/packagers accordingly. Kambiz
On 1/2/16 Jan 2 -11:31 AM, Kambiz Darabi wrote:
On 2016-01-02 09:05 CET, Faré <fahree@gmail.com> wrote:
One thing I *like* about submodules, is that they are optional. So if I already have regular checkouts of libraries (and I do), I can use them instead of those of make ext.
IMO it would be better to have fixed versions of the dependencies which are used by each and every developer and packager.
This is in my experience the basis of a solid and reliable build.
Yes, BUT, subtree makes it harder to distribute asdf independently from other libraries, which is very, very, very bad. People who use asdf for any serious purpose whatsoever already have their own library layout, packaging and versioning.
I agree fully. Only developers/packagers should need the dependencies in ext/ and only for building and testing asdf and for nothing else.
The last thing they want is to deal with headaches because asdf has another set of copies of some of the libraries that non-deterministically may or may not appear first when recursively scanning a tree. OK, since 3.1.5, the .cl-source-registry.cache should make that not a problem anymore. Still a potential source of confusion and space waste.
One solution would be to have a "raw" repository for asdf alone, and a separate repository asdf-dev that uses git subtree to provide asdf and all its dependencies together.
Two repositories are maybe too much hassle. We could also have a master branch which doesn't contain ext/* and is only updated with a new release.
No, I'm sorry, proliferation of branches is a non-starter. I am unwilling to spend time harmonizing a "developer" branch with a stripped down distribution branch before releasing. If people don't want the developer stuff they can either pull and not pull the ext/ or they can just get the tarball. Similarly, proliferation of repositories is unacceptable. If switching to git-subtree imposes too much cost on non-developers, we can simply stick to git submodules. They have their issues, but any change at this point requires not just an argument from tidiness, but a strong argument for being wildly better. Not a little better like better but for this annoying squash thing, but WILDLY, magically better. Right now, I find just getting the new CL-based tools to work everywhere is a slog. I've been spending all of my available ASDF time lately wrestling with added complexities that are "meta" to ASDF, leaving me with no time to work on ASDF itself. I'm not willing to add more cognitive burden or overhead. Sorry -- I appreciate your work to improve the process, Kambiz, but ASDF has just turned, against my will, into an laboratory for developing a CL-based scripting system. I am absolutely unwilling to see it turn into a testing ground for the bleeding edge of distributed version control, as well.
On Sat, Jan 2, 2016 at 1:42 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Right now, I find just getting the new CL-based tools to work everywhere is a slog. I've been spending all of my available ASDF time lately wrestling with added complexities that are "meta" to ASDF, leaving me with no time to work on ASDF itself. I'm not willing to add more cognitive burden or overhead.
Sorry -- I appreciate your work to improve the process, Kambiz, but ASDF has just turned, against my will, into an laboratory for developing a CL-based scripting system. I am absolutely unwilling to see it turn into a testing ground for the bleeding edge of distributed version control, as well.
I apologize for the trouble. It's notably my fault for not testing on Mac and Windows. But note that the need to download plenty of up-to-date libraries for testing purposes existed before the minimakefile merge. For instance, test-program always depended on lisp-invocation, test-encodings depended on asdf-encodings, etc. — though the policy was to let the test "succeed" with a warning if the dependency was not found. Similarly, the release script asdf-builder depended on inferior-shell and all its transitive dependencies, even before the current refactoring. The new asdf-tools only makes apparent early dependencies that were previously latent in the asdf test and release toolchain — some of them since ASDF 2 days, most of them since the initial ASDF 3 prerelease (2.27). I updated the README to explain that you may need to make ext again after you update ASDF. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I too am into martial arts: pistol shooting, rifle shooting, cannon shooting, flame throwing, grenade launching, missile firing, atom bombing...
On 2016-01-02 19:42 CET, Robert Goldman <rpgoldman@sift.net> wrote:
Sorry -- I appreciate your work to improve the process, Kambiz, but ASDF has just turned, against my will, into an laboratory for developing a CL-based scripting system. I am absolutely unwilling to see it turn into a testing ground for the bleeding edge of distributed version control, as well.
No problem at all. I was just trying to show another possibility to tackle the problem with the current tool set, which you expressed. But from a purely technical point of view, which is seldom the only basis for a decision. Kambiz
On 1/1/16 Jan 1 -6:18 AM, Kambiz Darabi wrote:
On 2016-01-01 09:59 CET, Kambiz Darabi <darabi@m-creations.com> wrote:
Happy New Year,
On 2015-12-31 16:12 CET, Robert Goldman <rpgoldman@sift.net> wrote:
On 12/31/15 Dec 31 -5:46 AM, Kambiz Darabi wrote:
On 2015-12-29 01:58 CET, Robert Goldman <rpgoldman@sift.net> wrote:
The problem of having clones not get auto-populated seems not worse than needing to do a magical incantation when you merge stuff back and forth at the cost of garbling your repo.
The complete magical incantation reads:
git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master
and to update it to the lastest master, replace 'add' with 'pull'.
The submodules cause annoying quiet failures, but even when the annoying quiet failures happen, you don't end up mangling your repo.
The problem with submodules is that every single person who checks out the repo is concerned with it, whereas git subtree has to be performed only by one single person and for all others it just works without them even knowing that there is a 'subtree' being used somewhere.
Can you explain that stuff about needing to squash? I read the subtree tutorial and that discussion just seemed like gibberish to me. Very far from clear. Maybe it's easier than the discussion made it seem. As I read it, it seemed to be saying that if you didn't remember to do --squash at the right times you would end up with a repo clogged with duplicate commits.
If that's the case...ugh. But perhaps it was just a bad explanation? Or does your subtree add expression above ensure that the squashing becomes the default?
My command above does contain the '--squash' option. The git-subtree author just chose the wrong default: usually, you are not interested in seeing each and every commit in the external repo.
But again, if you do the wrong thing, it's just a matter of:
git reset --hard HEAD~2 git subtree add/pull --squash ...
If we could figure this out maybe subtrees would be easier. But if you have to remember to squash every time, forget it....
IMO such commands should be automated with a makefile entry, so you wouldn't have to remember the --squash option.
Unfortunately, right now the makefile is in the worst condition of the whole project, because of the recent merge of minimakefile, which by its nature was largely untestable (since exercising its functionality would change the repository, and even the upstream repository). Indeed, just recently, I wasn't able to use the makefile to fix my problems with the addition of new ext/ submodules because..... the makefile didn't work until the submodules were added.
I pushed a branch to my repo which contains the necessary changes:
https://gitlab.common-lisp.net/darabi/asdf/tree/ext-subtree
The specification of what to get where and which version is in ext/dependencies.lisp-expr
and
git checkout ext-subtree make ext
does the job.
I'm happy to consider switching to using git-subtree. It may indeed be better. But not until it gets some better documentation. The discussion in the Pro-Git book, to be honest, just makes my head hurt. See https://git-scm.com/book/en/v1/Git-Tools-Subtree-Merging So, the remote repository is added as a remote for the *containing* repository? ASDF would have the cl-scripting as an upstream remote for ASDF, not for a contained thing? And this "Then, you can merge those changes back into your master branch. You can use git merge -s subtree and it will work fine; but Git will also merge the histories together, which you probably don’t want." ...may mean something to the person who wrote it, but it really doesn't mean anything clear to me. The whole section is like this. I am grateful to the author of this book, and I know documentation is hard, but this simply doesn't rise to the level of an explanation. I suspect that the problem is that giving a real explanation would require standing up repositories both with and without the use of git subtree and showing by worked example what is meant by merging histories together. Also, it seems like if you forget to use --squash merging and then push the results to the main (cl.net) repo, you end up messing up the ASDF repo's history in an unrecoverable way. That seems too risky to me. Yes, *if you notice* you can undo it with a git reset, but if you *don't* notice, and you push, what happens then? Best, r
On 2016-01-02 00:33 CET, Robert Goldman <rpgoldman@sift.net> wrote:
On 1/1/16 Jan 1 -6:18 AM, Kambiz Darabi wrote:
On 2016-01-01 09:59 CET, Kambiz Darabi <darabi@m-creations.com> wrote:
Happy New Year,
On 2015-12-31 16:12 CET, Robert Goldman <rpgoldman@sift.net> wrote:
On 12/31/15 Dec 31 -5:46 AM, Kambiz Darabi wrote:
On 2015-12-29 01:58 CET, Robert Goldman <rpgoldman@sift.net> wrote:
The problem of having clones not get auto-populated seems not worse than needing to do a magical incantation when you merge stuff back and forth at the cost of garbling your repo.
The complete magical incantation reads:
git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master
and to update it to the lastest master, replace 'add' with 'pull'.
The submodules cause annoying quiet failures, but even when the annoying quiet failures happen, you don't end up mangling your repo.
The problem with submodules is that every single person who checks out the repo is concerned with it, whereas git subtree has to be performed only by one single person and for all others it just works without them even knowing that there is a 'subtree' being used somewhere.
Can you explain that stuff about needing to squash? I read the subtree tutorial and that discussion just seemed like gibberish to me. Very far from clear. Maybe it's easier than the discussion made it seem. As I read it, it seemed to be saying that if you didn't remember to do --squash at the right times you would end up with a repo clogged with duplicate commits.
If that's the case...ugh. But perhaps it was just a bad explanation? Or does your subtree add expression above ensure that the squashing becomes the default?
My command above does contain the '--squash' option. The git-subtree author just chose the wrong default: usually, you are not interested in seeing each and every commit in the external repo.
But again, if you do the wrong thing, it's just a matter of:
git reset --hard HEAD~2 git subtree add/pull --squash ...
If we could figure this out maybe subtrees would be easier. But if you have to remember to squash every time, forget it....
IMO such commands should be automated with a makefile entry, so you wouldn't have to remember the --squash option.
Unfortunately, right now the makefile is in the worst condition of the whole project, because of the recent merge of minimakefile, which by its nature was largely untestable (since exercising its functionality would change the repository, and even the upstream repository).
AFAICS there are only two places where a remote repo ist changed: https://gitlab.common-lisp.net/asdf/asdf/blob/master/tools/git.lisp#L18 https://gitlab.common-lisp.net/asdf/asdf/blob/master/tools/git.lisp#L44 So testing makefile changes is only a matter of renaming the remotes 'origin', 'cl.net', and 'github'. One can even git clone locally, add the local 'remotes', name them appropriately and test the changes.
Indeed, just recently, I wasn't able to use the makefile to fix my problems with the addition of new ext/ submodules because..... the makefile didn't work until the submodules were added.
This is also a point in favor of subtree, as the dependencies are then part of the repo. These 9 lines were necessary to implement the new behaviour: https://gitlab.common-lisp.net/darabi/asdf/commit/fc24dae0a7ba8ccbe7633584da... I found it quite easy to find the correct place to modify the code although I'm not at all familiar with the code base.
I pushed a branch to my repo which contains the necessary changes:
https://gitlab.common-lisp.net/darabi/asdf/tree/ext-subtree
The specification of what to get where and which version is in ext/dependencies.lisp-expr
and
git checkout ext-subtree make ext
does the job.
I'm happy to consider switching to using git-subtree. It may indeed be better.
But not until it gets some better documentation. The discussion in the Pro-Git book, to be honest, just makes my head hurt. See https://git-scm.com/book/en/v1/Git-Tools-Subtree-Merging
This is the documentation for the concept of subtree merging, not the 'git subtree' command. You don't have to deal with the subtree merge yourself, the command 'git subtree' does it for you. I apologise for not having provided the pointer to the docs in the first place (and will give a short example below): man git subtree should give you this: http://manpages.ubuntu.com/manpages/trusty/man1/git-subtree.1.html
So, the remote repository is added as a remote for the *containing* repository? ASDF would have the cl-scripting as an upstream remote for ASDF, not for a contained thing?
No, the commits in cl-scripting (leading to HEAD) are imported into a local branch (but you don't have to deal with that branch in any way). Let's go through the process, after you pull my branch ext-subtree: after Faré's last commit 'tools: Simplify a ,@ into a ,', there is 'Remove git submodules in ext' which deletes .gitmodules and all ext subdirs: https://gitlab.common-lisp.net/darabi/asdf/commit/d028e6d2b9d729a47a7380b05e... then, I performed git subtree add --squash --prefix=ext/alexandria <alexandria-repo-url> master which creates two commits, look at the first one with 'git log --name-status 915257': Squashed 'ext/alexandria/' content from commit 5a17c07 git-subtree-dir: ext/alexandria git-subtree-split: 5a17c072970cf50213f7f896c40e6e640638391f A .boring A .gitignore A AUTHORS A LICENCE A README A alexandria-tests.asd A alexandria.asd ... As one can see, this is the content of the alexandria repo. Hidden behind the git-subtree-split property, is the branch which contains the history of alexandria which led to master: git log --oneline 5a17c07 5a17c07 fix hygiene and multiple evaluation in ENSURE-GETHASH b1c6ee0 Fix default ARRAY-INDEX and ARRAY-LENGTH. a019eda Extended DELETE-FROM-PLIST.1 test to test the removal of duplicate keys. 4c4361c Use the original DELETE-FROM-PLIST replacement. Now back to the second commit which was performed by 'git subtree add': commit 62116639895d8c2f537b0cb8f26314716edf3900 Merge: d028e6d 9152572 Author: Kambiz Darabi <darabi@m-creations.net> Date: Fri Jan 1 12:53:03 2016 +0100 Add ext/alexandria as git subtree git subtree add --prefix=ext/alexandria --squash https://gitlab.common-lisp.net/alexandria/alexandria.git master This commit is a merge and doesn't add any specific files, but if you look at ext/alexandria, you can see that is was populated with the content of "Squashed 'ext/alexandria/'". That's it. From now on, Alexandria's commit 5a17c07 will live in our repo at ext/alexandria. If one day you decide that it's time to update to the latest version of alexandria, you perform: git subtree pull --squash --prefix=ext/alexandria <alexandria-repo-url> master and the same thing happens again. Now to the makefile automation part: The file ext/dependencies.lisp-expr https://gitlab.common-lisp.net/darabi/asdf/commit/fc24dae0a7ba8ccbe7633584da... currently contains: (setf asdf-tools::*ext-dependencies* '(("alexandria" "master" "https://gitlab.common-lisp.net/alexandria/alexandria.git") ... ("optima" "master" "https://github.com/m2ym/optima.git"))) performing 'make ext' leads to a 'git subtree pull --squash' as described above. So, as a developer, you can decide at which point in time you would like to upgrade to newer versions of the dependencies. If I was the maintainer, I would edit dependencies.lisp-expr and replace 'master' with the latest release tag of the corresponding repos or a specific commit if the repo doesn't have release tags. This would lead to clearly defined dependencies which would make build and testing reproducible. But this is of course your decision. HTH Kambiz
participants (4)
-
Attila Lendvai -
Faré -
Kambiz Darabi -
Robert Goldman