On Mon, Mar 17, 2014 at 2:47 PM, Robert P. Goldman rpgoldman@sift.infowrote:
Jean-Claude Beaudoin wrote:
On Mon, Mar 17, 2014 at 12:48 PM, Robert P. Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:
Random minor question: In bundle.lisp why the replacement of 'defsystem with 'asdf/defsystem:defsystem? Did something go wrong for you with the former?The "defsystem" symbol was interned in the wrong package, being read at that point of the code into package "asdf/bundle". Package "asdf/bundle" does not use package "asdf/defsystem". All this resulting into a failure to properly expand the defsystem macro.
So, in other words, the bundle code was buggy before, right?
You are quite right yourself.
I wonder why we didn't see this.... What piece of your code exercised this? I'd like to know so we can test.
The bundle facility is a central part of the MKCL build process. A large part of what MKCL puts in its "contrib" sub-directory is built using it.
Other than that, I didn't see anything in the diff.I hope that you mean that you didn't see anything *wrong* in the diff.
You are quite right!
I'll merge to master and tag this afternoon.
Thanks, R
P.S.: I took the liberty to CC the asdf-devel list since your initial "defsystem" question had too, I left the CC out of my reply by simply not paying enough attention to which reply button I clicked.on.
Oops, yes the defsystem thing was a bug, and there was a missing dependency, too. I fixed that, as well as the minor nits I saw, since this was merged into master. I rebased my changes and pushed them all as 3.1.0.97.
I also implemented and tested image-op on other implementations — this probably broke MKCL. Can you re-test? I also refactored test-bundle, and I don't understand the compiler::*builder-default-libraries* hack — can't MKCL do that automatically?
Finally, please see to the fundamental issue I have with your code: I believe program-op should behave on MKCL just like on other platforms and * link in uiop * call restore-image as in ecl's uiop:create-image — handles not just the entry-point, but the important initialization hooks. * not necessitate any of :prefix-lisp-object-files, :extra-build-args or :epilogue *by default*, i.e. there should be no #+mkcl in hello-world-example.asd — though you're welcome to support these useful extra features, and add some additional test.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I once won an argument with a socialist. He was richer than I, so I reached for his wallet. He slapped me. QED. He didn't get it. His wife did.
0- OK, I managed to compile MKCL from git after updating its ASDF. Then, I massaged program-op to do what I wanted, which involves linking UIOP or ASDF — it's committed to branch build-op.
Problems I found: 1- you look for encodings in #p"SYS:ENCODINGS;" (upper case, as per the standard), which is mapped to .../lib/mkcl-1.1.8/encodings/ (lower case, as customary on Unix, thus as per the standard). However, the directory and its files have upper case names, and are therefore not found. This breaks asdf-encodings and thus the test-encodings.script test. Note: you need to download the latest version of asdf-encodings for this test to work.
2- even assuming SYS:ENCODINGS; were found, it is not obvious at all what is the right way to detect the presence of an encoding. i.e. given the keyword :latin-2, what is the recommended way to detect if an encoding is supported and/or to list supported encodings? e.g. check (si::make-encoding :latin-2), or see that mk-ext::latin-2 in bound to a keyword?
3- if I uncomment the lines: ;;(unless (or #+ecl (use-ecl-byte-compiler-p)) ;; (setf *load-system-operation* 'load-bundle-op)) I get an error in test-logical-pathname, with the .fasb apparently mapped to the wrong directory.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I'm only a stupid AI, but my creator is a real genius!
On Tue, Mar 25, 2014 at 3:12 AM, Faré fahree@gmail.com wrote:
...
Problems I found: 1- you look for encodings in #p"SYS:ENCODINGS;" (upper case, as per the standard), which is mapped to .../lib/mkcl-1.1.8/encodings/ (lower case, as customary on Unix, thus as per the standard). However, the directory and its files have upper case names, and are therefore not found. This breaks asdf-encodings and thus the test-encodings.script test. Note: you need to download the latest version of asdf-encodings for this test to work.
Thank you for the bug report. This will give me an opportunity to revisit that land of half-forgotten and so often much too brittle legacy.
2- even assuming SYS:ENCODINGS; were found, it is not obvious at all what is the right way to detect the presence of an encoding. i.e. given the keyword :latin-2, what is the recommended way to detect if an encoding is supported and/or to list supported encodings? e.g. check (si::make-encoding :latin-2), or see that mk-ext::latin-2 in bound to a keyword?
As I said above, half-forgotten legacy, so I'll need a little while to reply on this one...
3- if I uncomment the lines: ;;(unless (or #+ecl (use-ecl-byte-compiler-p)) ;; (setf *load-system-operation* 'load-bundle-op)) I get an error in test-logical-pathname, with the .fasb apparently mapped to the wrong directory.
???! What is the real need for this one? You see me a bit baffled here. Will investigate...
3- if I uncomment the lines: ;;(unless (or #+ecl (use-ecl-byte-compiler-p)) ;; (setf *load-system-operation* 'load-bundle-op)) I get an error in test-logical-pathname, with the .fasb apparently mapped to the wrong directory.
???! What is the real need for this one? You see me a bit baffled here. Will investigate...
If I understand correctly the case that Juanjo once made, the real need is that dlopen of a lot of small fasls seems to be more expensive than for a few big bundles — and also that if you depend on a lot of systems and have them loaded via load-bundle-op, then need to recompile one file, this happens through load-op which will cause all the previous systems to be re-loaded via load-op.
With (setf *load-system-operation* 'load-bundle-op), all the already-compiled dependencies are loaded as bundles, and only systems in need of recompilation are compiled with compile-op, load-op, and then again compile-bundle-op and load-bundle-op.
Unhappily, there are bugs, and at least on ECL, loading two different versions of the same library seems to lead to cases where the old one shadows the new one, or something. egrep '#.*(e|mk)cl' asdf/test/ can give a few hints, particularly in test-bundle.script and test-xach-update-bug.script.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org As far as we know, our computer has never had an undetected error. — Weisert
Take 2.
On Tue, Mar 25, 2014 at 3:12 AM, Faré fahree@gmail.com wrote:
...
Problems I found: 1- you look for encodings in #p"SYS:ENCODINGS;" (upper case, as per the standard), which is mapped to .../lib/mkcl-1.1.8/encodings/ (lower case, as customary on Unix, thus as per the standard). However, the directory and its files have upper case names, and are therefore not found. This breaks asdf-encodings and thus the test-encodings.script test. Note: you need to download the latest version of asdf-encodings for this test to work.
The fix should be in the MKCL git repo master head now.
2- even assuming SYS:ENCODINGS; were found, it is not obvious at all what is the right way to detect the presence of an encoding. i.e. given the keyword :latin-2, what is the recommended way to detect if an encoding is supported and/or to list supported encodings? e.g. check (si::make-encoding :latin-2), or see that mk-ext::latin-2 in bound to a keyword?
#'si::make-encoding returns NIL on failure plus a string as secondary value to explain the failure, it returns non-NIL on success. But that is surely awkward for your purpose. There is (si::all-encodings) that return a list of keywords each of which names a valid encoding. So I think that (member :latin-2 (si::all-encodings)) will do what you want.
BTW, the gimmick of binding symbols in package :mk-ext (not to a keyword but to a hashtable) is a part of that legacy code that is going away soon (MKCL 1.2.0), so please do not rely on it. Use #'si::all-encodings instead.
3- if I uncomment the lines: ;;(unless (or #+ecl (use-ecl-byte-compiler-p)) ;; (setf *load-system-operation* 'load-bundle-op)) I get an error in test-logical-pathname, with the .fasb apparently mapped to the wrong directory.
I am still scratching my head on this one. It feels so much like an optimization concern of questionable wisdom... Is it the fear of hitting some wall I am unaware of, like the 1024 file descriptor one, or something else I haven't seen yet? Is it some concern about memory consumption in a development environment where memory use is through the roof already anyway? I don't get it, and I am ready to wait until this becomes a real/confirmed problem before I address it (I bet it will take a very long time to see that happen). Can't we just drop this one?
Hi,
1- encodings...
The fix should be in the MKCL git repo master head now.
It works perfectly.
However, somehow ASDF upgrade is broken:
(require :asdf)
("ASDF" "asdf" "UIOP")
(asdf:load-system :asdf)
Debugger called in: #<thread "Initial" active (4648) 0x7f7c4f74b700 0000000001be2000>.
#<a INVALID-SLOT 89384352>: The index 35 (max: 33) is not a valid slot index in the object #<ASDF/PACKAGE-SYSTEM:PACKAGE-SYSTEM "asdf">.
It looks like you're failing to call upgrade-instance-for-redefined-class before you're calling a new method on an extended class.
3- if I uncomment the lines: ;;(unless (or #+ecl (use-ecl-byte-compiler-p)) ;; (setf *load-system-operation* 'load-bundle-op)) I get an error in test-logical-pathname, with the .fasb apparently mapped to the wrong directory.
I am still scratching my head on this one. It feels so much like an optimization concern of questionable wisdom... Is it the fear of hitting some wall I am unaware of, like the 1024 file descriptor one, or something else I haven't seen yet? Is it some concern about memory consumption in a development environment where memory use is through the roof already anyway? I don't get it, and I am ready to wait until this becomes a real/confirmed problem before I address it (I bet it will take a very long time to see that happen). Can't we just drop this one?
We can drop it, but the bugs it uncovers are real. It does appear to work with MKCL, though. I suspect it's related to some bad shadowing of initialization function in ECL, whereby the version from the old object file is called, instead of the version from the new one.
Did you change something in MKCL regarding initialization functions, e.g. using attributes to mark some functions as being in an initialization section that will cause them to be called at by the linker, when ECL for portability to old systems still makes its own function and calls it?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Lisp programmer available: Will write code that writes code that writes code for food. — Rob Warnock
On Thu, Mar 27, 2014 at 1:01 PM, Faré fahree@gmail.com wrote:
Hi,
1- encodings...
The fix should be in the MKCL git repo master head now.
It works perfectly.
Thank you for the confirmation.
However, somehow ASDF upgrade is broken:
(require :asdf)
("ASDF" "asdf" "UIOP")
(asdf:load-system :asdf)
Debugger called in: #<thread "Initial" active (4648) 0x7f7c4f74b700 0000000001be2000>.
#<a INVALID-SLOT 89384352>: The index 35 (max: 33) is not a valid slot index in the object #<ASDF/PACKAGE-SYSTEM:PACKAGE-SYSTEM "asdf">.
It looks like you're failing to call upgrade-instance-for-redefined-class before you're calling a new method on an extended class.
Thank you for the bug report. It will take a little while (2 or 3 days) for a fix since I am stuck in something else right now.
3- if I uncomment the lines: ;;(unless (or #+ecl (use-ecl-byte-compiler-p)) ;; (setf *load-system-operation* 'load-bundle-op)) I get an error in test-logical-pathname, with the .fasb apparently mapped to the wrong directory.
I am still scratching my head on this one. It feels so much like an optimization concern of questionable wisdom... Is it the fear of hitting some wall I am unaware of, like the 1024 file descriptor one, or
something
else I haven't seen yet? Is it some concern about memory consumption in
a
development environment where memory use is through the roof already
anyway?
I don't get it, and I am ready to wait until this becomes a
real/confirmed
problem before I address it (I bet it will take a very long time to see
that
happen). Can't we just drop this one?
We can drop it, but the bugs it uncovers are real. It does appear to work with MKCL, though. I suspect it's related to some bad shadowing of initialization function in ECL, whereby the version from the old object file is called, instead of the version from the new one.
Did you change something in MKCL regarding initialization functions, e.g. using attributes to mark some functions as being in an initialization section that will cause them to be called at by the linker, when ECL for portability to old systems still makes its own function and calls it?
Quite a few things have been changed at those low levels since the fork but nothing comes clearly enough to my mind to be an obvious explanation. I will need to do real archeology to figure this one out...
I suspect it's related to some bad shadowing of initialization function in ECL, whereby the version from the old object file is called, instead of the version from the new one.
Did you change something in MKCL regarding initialization functions, e.g. using attributes to mark some functions as being in an initialization section that will cause them to be called at by the linker, when ECL for portability to old systems still makes its own function and calls it?
Quite a few things have been changed at those low levels since the fork but nothing comes clearly enough to my mind to be an obvious explanation. I will need to do real archeology to figure this one out...
Well, I notice that with ECL, I have to pass a :init-function argument, so ECL will identify the init function from the name of the target file after renaming from its temporary name with -ASDFTMP suffix. In MKCL, you did away with this argument, which suggests a different way to identify the init function...
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Only presidents, editors, and people with tapeworms have the right to use the editorial "we." — Mark Twain
-
Faré -
Jean-Claude Beaudoin