[Asdf-devel] Moving MKCL's bundled version to ASDF 3.1.2
On Wed, Jun 18, 2014 at 1:55 AM, Faré <fahree@gmail.com> wrote:
Can you upgrade your ASDF from 3.0.3.0.2 to 3.1.2? Does 3.1.2 require a patch? If it does, then let's get it pushed upstream into e.g. 3.1.2.5.
I dropped in ASDF 3.1.2 from the release tarball in the MKCL 1.1.9 source tree as a direct replacement of the currently bundled ASDF. It compiled fine. The whole MKCL build completes and installs as usual. The thus resulting ASDF 3.1.2 reports all tests are good when 'make test l=mkcl" is run, as well as with 'make test-upgrade l=mkcl'. But when I come to try an upgrade from 3.1.2 to 3.1.2.4 syntax-control branch I then get a crash. Here is attached the transcript of what is produced at the repl. The error message basically says that 'build/asdf.lisp' cannot be found. And in fact it is missing (it used to be produced properly by 3.0.3.0.X). If I drop to a shell prompt and do make build/asdf.lisp and try the same upgrade again then it completes properly. So is there something broken now with the ASDF bundle 'concatenate' ops?
NB: please send a diff; I can review it, but Robert should commit and push it — I'm out of this business (except for the branches I wrote and would like to see merged — Robert, any news about them?)
PS: minor issue, but when I (quit), I get this annoying message:
(quit)
;; MKCL shutdown: Thread refused to be interrupted: Sleeping thread: #<thread "Imported thread pool filler daemon" active detached (27380) 0x7f1aa19d0700 0000000003286000>, in file threads.d at line 1076. ;; MKCL shutdown: Killing thread #<thread "Imported thread pool filler daemon" active detached (27380) 0x7f1aa19d0700 0000000003286000>.
Thanks for the report. I bet you get this only once in a blue moon. It is just a harmless annoyance. It also just reminds me that there is still some
Would do but I don't quite understand how to fix what I just reported here above. practical reality of thread programming I have come to terms with. It is the very low frequency combined with clear randomness that bothers me. I just have get back to it and silence this nagger for good... Thanks again.
Jean-Claude Beaudoin wrote:
On Wed, Jun 18, 2014 at 1:55 AM, Faré <fahree@gmail.com <mailto:fahree@gmail.com>> wrote:
Can you upgrade your ASDF from 3.0.3.0.2 to 3.1.2? Does 3.1.2 require a patch? If it does, then let's get it pushed upstream into e.g. 3.1.2.5.
I dropped in ASDF 3.1.2 from the release tarball in the MKCL 1.1.9 source tree as a direct replacement of the currently bundled ASDF. It compiled fine. The whole MKCL build completes and installs as usual. The thus resulting ASDF 3.1.2 reports all tests are good when 'make test l=mkcl" is run, as well as with 'make test-upgrade l=mkcl'.
Is this present in git? I just updated, and what I get from git @ common-lisp.net still has ASDF 3.0.3.0.2 So I haven't been able to test this. Do you have a branch on the git repo that I could test? If not will you please set one up? Thanks, r
On Fri, Jun 20, 2014 at 9:37 AM, Robert P. Goldman <rpgoldman@sift.info> wrote:
Jean-Claude Beaudoin wrote:
On Wed, Jun 18, 2014 at 1:55 AM, Faré <fahree@gmail.com <mailto:fahree@gmail.com>> wrote:
Can you upgrade your ASDF from 3.0.3.0.2 to 3.1.2? Does 3.1.2 require a patch? If it does, then let's get it pushed upstream into e.g. 3.1.2.5.
I dropped in ASDF 3.1.2 from the release tarball in the MKCL 1.1.9 source tree as a direct replacement of the currently bundled ASDF. It compiled fine. The whole MKCL build completes and installs as usual. The thus resulting ASDF 3.1.2 reports all tests are good when 'make test l=mkcl" is run, as well as with 'make test-upgrade l=mkcl'.
Is this present in git? I just updated, and what I get from git @ common-lisp.net still has ASDF 3.0.3.0.2
It is not in git repo yet, sorry.
So I haven't been able to test this.
Do you have a branch on the git repo that I could test? If not will you please set one up?
It was still just a napkin experiment when I wrote my email. I guess I will come around to it (branch) if I do not figure this out in the coming hours. Cheers.
Dear Jean-Claude, that's a pretty embarrassing bootstrap bug in ASDF indeed, that I introduced in 3.1.0.4. Interesting that the test suite didn't find it. Oops. Bisecting things a bit, it seems to be a bug introduced in 3.1.0.104. To test things: v=3.1.0.103; ASDF_UPGRADE_TEST_TAGS=$v ./test/run-tests.sh -H ; \rm build/asdf.lisp ; rlwrap sbcl --load build/asdf-$v.lisp --eval '(progn (format t "~A~%~S~%~S~%~S~%" (asdf:asdf-version) (asdf:traverse `asdf:load-op `asdf) (asdf:input-files `asdf:monolithic-concatenate-source-op "asdf/defsystem") (asdf:component-depends-on `asdf:prepare-op :asdf)) (quit))' When it works, such as with v=3.1.0.103, it prints ASDF/CONCATENATE-SOURCE:MONOLITHIC-CONCATENATE-SOURCE-OP as the first operation, because indeed prepare-op asdf has the correct dependency. When it doesn't work, such as with v=3.1.0.104, the dependency is missing. Oops. A further git bisect between 3.1.0.103 and 3.1.0.104 using make ; /bin/mv -f build/asdf.lisp build/asdf-x.lisp ; rlwrap sbcl --load build/asdf-x.lisp --eval '(progn (format t "~A~%~S~%~S~%~S~%" (asdf:asdf-version) (asdf:traverse `asdf:load-op `asdf) (asdf:input-files `asdf:monolithic-concatenate-source-op "asdf/defsystem") (asdf:component-depends-on `asdf:prepare-op :asdf)) (quit))' reveals: ae3b26b18ec33a570752bbfb584171781210f97c is the first bad commit Problem: the component-depends-on method being introduced is missing a ,@(call-next-method). Oops. Although — no, the call-next-method would defeat the purpose, since it's about overriding load-op (as obtained by sideway-operation superclass) by a *load-system-operation*. If we want this effect, we need something more like: (defmethod component-depends-on ((o prepare-op) (s system)) (loop :for (o . cs) :in (call-next-method) :collect (cons (if (eq o 'load-op) *load-system-operation* o) cs))) Indeed this change makes the test command above succeed when it otherwise fails. The other component-depends-on method in this file is also suspiciously missing a call-next-method. Should be a simple ,@(call-next-method). Jean-Claude: thanks *A LOT* for your bug report! Robert, I took the liberty of committing this fix as 3.1.2.5. You might want a different fix, or to scratch this feature, etc. Issue: That means that ASDF 3.1.2 is unfit to bootstrap further variants of ASDF, and they must be bootstrapped with make, or their asdf.asd modified in a yet-to-be-determined way to compensate for that bug. The bug is probably worth releasing a 3.1.3 at some point and encouraging implementations to ship 3.1.3 instead of 3.1.2. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Suppose 2/3 Congress were incinerated, what would we lose to offset our gain of their salaries and the salaries of their parasites? — H. L. Mencken
On Fri, Jun 20, 2014 at 7:53 PM, Faré <fahree@gmail.com> wrote:
Robert, I took the liberty of committing this fix as 3.1.2.5. You might want a different fix, or to scratch this feature, etc.
I can confirm that, as of now, ASDF git master head is completing a "live upgrade" like ASDF 3.0.3.0.X did.
On Fri, Jun 20, 2014 at 6:11 AM, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
I dropped in ASDF 3.1.2 from the release tarball in the MKCL 1.1.9 source tree as a direct replacement of the currently bundled ASDF. It compiled fine. The whole MKCL build completes and installs as usual. The thus resulting ASDF 3.1.2 reports all tests are good when 'make test l=mkcl" is run, as well as with 'make test-upgrade l=mkcl'.
But when I come to try an upgrade from 3.1.2 to 3.1.2.4 syntax-control branch I then get a crash. Here is attached the transcript of what is produced at the repl.
The error message basically says that 'build/asdf.lisp' cannot be found. And in fact it is missing (it used to be produced properly by 3.0.3.0.X). If I drop to a shell prompt and do make build/asdf.lisp and try the same upgrade again then it completes properly. So is there something broken now with the ASDF bundle 'concatenate' ops?
I suspect you are not bootstrapping ASDF properly; that would explain why your patch didn't fix things at first, but worked when I tried to reproduce: only the second time around would ASDF have been properly built. Most implementations "just" check in asdf.lisp. If you checkout the whole tree, you need to create asdf.lisp by running "make".
Would do but I don't quite understand how to fix what I just reported here above.
Please make sure you build asdf before you use asdf. asdf is not magical: it needs either make or asdf to build asdf. If you don't have asdf yet and try to build asdf, you're in trouble — unless you use make to escape the bootstrap loop.
PS: minor issue, but when I (quit), I get this annoying message:
(quit)
;; MKCL shutdown: Thread refused to be interrupted: Sleeping thread: #<thread "Imported thread pool filler daemon" active detached (27380) 0x7f1aa19d0700 0000000003286000>, in file threads.d at line 1076. ;; MKCL shutdown: Killing thread #<thread "Imported thread pool filler daemon" active detached (27380) 0x7f1aa19d0700 0000000003286000>.
Thanks for the report. I bet you get this only once in a blue moon. It is just a harmless annoyance. It also just reminds me that there is still some practical reality of thread programming I have come to terms with. It is the very low frequency combined with clear randomness that bothers me. I just have get back to it and silence this nagger for good...
It happens more than "once in a blue moon". Order of magnitude 5% of invocations of mkcl. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Life is like an onion: you peel off layer after layer, then you find there is nothing in it.
On Fri, Jun 20, 2014 at 9:40 AM, Faré <fahree@gmail.com> wrote:
On Fri, Jun 20, 2014 at 6:11 AM, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
I dropped in ASDF 3.1.2 from the release tarball in the MKCL 1.1.9 source tree as a direct replacement of the currently bundled ASDF. It compiled fine. The whole MKCL build completes and installs as usual. The thus resulting ASDF 3.1.2 reports all tests are good when 'make test l=mkcl" is run, as well as with 'make test-upgrade l=mkcl'.
But when I come to try an upgrade from 3.1.2 to 3.1.2.4 syntax-control branch I then get a crash. Here is attached the transcript of what is produced at the repl.
The error message basically says that 'build/asdf.lisp' cannot be found. And in fact it is missing (it used to be produced properly by 3.0.3.0.X). If I drop to a shell prompt and do make build/asdf.lisp and try the same upgrade again then it completes properly. So is there something broken now with the ASDF bundle 'concatenate' ops?
I suspect you are not bootstrapping ASDF properly; that would explain why your patch didn't fix things at first, but worked when I tried to reproduce: only the second time around would ASDF have been properly built. Most implementations "just" check in asdf.lisp. If you checkout the whole tree, you need to create asdf.lisp by running "make".
And I do as you say during MKCL's bootstrap. But that is not the situation of interest here. What we have here is MKCL fully and properly built and installed, with its bundled ASDF version already loaded, attempting a "live upgrade" of ASDF.
Would do but I don't quite understand how to fix what I just reported here above.
Please make sure you build asdf before you use asdf. asdf is not magical: it needs either make or asdf to build asdf. If you don't have asdf yet and try to build asdf, you're in trouble — unless you use make to escape the bootstrap loop.
You got me to doubt myself so I went back to my original setup with ASDF 3.0.3.0.2 and replayed the whole scenario from there, making sure to remove the 'build/asdf.lisp' file from the "live upgrade" directory first. I attached the transcript of the REPL this gives me. As you can see from it, the 'build/asdf.lisp' gets recreated without any explicit call to "make" on my part. This is not what happens when the bundled ASDF is 3.1.2!
participants (3)
-
Faré -
Jean-Claude Beaudoin -
Robert P. Goldman