Bug report: Loading a FASL into a package which doesn't import SETQ
The code below works in CCL, CLISP and SBCL, but not in ABCL. $ echo '(cl:format cl:t "~&I am in package ~s.~%" cl:*package*)' > foo.lisp $ java -jar abcl-1.7.1.jar Armed Bear Common Lisp 1.7.1 Java 1.8.0_265 Oracle Corporation OpenJDK 64-Bit Server VM Low-level initialization completed in 0.236 seconds. Startup completed in 0.931 seconds. Type ":help" for a list of available commands. CL-USER(1): (defpackage "BAR" (:use)) #<PACKAGE BAR> CL-USER(2): (let ((*package* (find-package "BAR"))) (load (compile-file "foo"))) ; Compiling /tmp/foo/abcl-bin-1.7.1/foo.lisp ... ; (COMMON-LISP:FORMAT COMMON-LISP:T ...) ; Wrote /tmp/foo/abcl-bin-1.7.1/foo.abcl (0.036 seconds) Error loading /tmp/foo/abcl-bin-1.7.1/foo.abcl at line 4 (offset 125) #<THREAD "interpreter" {1B31C40C}>: Debugger invoked on condition of type UNDEFINED-FUNCTION The function BAR::SETQ is undefined. Restarts: 0: CONTINUE Try again. 1: USE-VALUE Specify a function to call instead. 2: RETURN-VALUE Return one or more values from the call to SETQ. 3: TOP-LEVEL Return to top level. [1] BAR(3):
On 26 Oct 2020, at 00:57, Robert Munyer <2433647181@munyer.com> wrote:
The code below works in CCL, CLISP and SBCL, but not in ABCL.
$ echo '(cl:format cl:t "~&I am in package ~s.~%" cl:*package*)' > foo.lisp $ java -jar abcl-1.7.1.jar Armed Bear Common Lisp 1.7.1 Java 1.8.0_265 Oracle Corporation OpenJDK 64-Bit Server VM Low-level initialization completed in 0.236 seconds. Startup completed in 0.931 seconds. Type ":help" for a list of available commands. CL-USER(1): (defpackage "BAR" (:use)) #<PACKAGE BAR> CL-USER(2): (let ((*package* (find-package "BAR"))) (load (compile-file "foo"))) ; Compiling /tmp/foo/abcl-bin-1.7.1/foo.lisp ... ; (COMMON-LISP:FORMAT COMMON-LISP:T ...) ; Wrote /tmp/foo/abcl-bin-1.7.1/foo.abcl (0.036 seconds) Error loading /tmp/foo/abcl-bin-1.7.1/foo.abcl at line 4 (offset 125) #<THREAD "interpreter" {1B31C40C}>: Debugger invoked on condition of type UNDEFINED-FUNCTION The function BAR::SETQ is undefined. Restarts: 0: CONTINUE Try again. 1: USE-VALUE Specify a function to call instead. 2: RETURN-VALUE Return one or more values from the call to SETQ. 3: TOP-LEVEL Return to top level. [1] BAR(3):
Indeed, there’s a bug in abcl. $ clall -r '(defpackage "BAR" (:use))' ' (let ((*package* (find-package "BAR"))) (load (compile-file "foo.lisp")))' Armed Bear Common Lisp --> #<PACKAGE BAR> Armed Bear Common Lisp The function BAR::SETQ is undefined. Clozure Common Lisp --> #<Package "BAR"> Clozure Common Lisp --> #P"/private/tmp/foo.dx64fsl" CLISP --> #<PACKAGE BAR> CLISP --> #P"/private/tmp/foo.fas" ECL --> #<"BAR" package> ECL Note: Invoking external command: gcc -x c-header /usr/local/include/ecl/ecl-cmp.h -I/usr/local/include/ -g -O2 -fPIC -fno-common -D_THREAD_SAFE -Ddarwin -O2 -o /var/folders/pq/82920zm125n09frk81rrtp200000gn/T/ecl-include092A/ecl-cmp.h.gch SBCL --> #<PACKAGE "BAR"> SBCL --> T $ abcl --help|head -n 3 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Xmx6g Armed Bear Common Lisp 1.7.1 Java 11.0.8 AdoptOpenJDK OpenJDK 64-Bit Server VM -- __Pascal J. Bourguignon__
On Oct 26, 2020, at 00:57, Robert Munyer <2433647181@munyer.com> wrote:
The code below works in CCL, CLISP and SBCL, but not in ABCL.
Trac’in as <https://abcl.org/trac/ticket/475> I will try to address for abcl-1.8.0 as part of reworking pathnames for the fasl loader itself. Thanks for the report! -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
I will try to address for abcl-1.8.0 as part of reworking pathnames for the fasl loader itself.
Thanks for the report!
You're welcome! I've recently encountered a few other bits of weirdness that I think might be caused by ABCL pathname bugs; I'll try to get them isolated and reported quickly.
Trac’in as <https://abcl.org/trac/ticket/475>
I have found that I can "protect" a .abcl file from this bug, by using these commands: unzip foo.abcl __loader__._ sed -i 's/(setq/(CL:SETQ/ig' __loader__._ zip -f foo.abcl __loader__._ This is not a proper fix, because it would corrupt any other place where the five-character sequence "(setq" happens to appear. A proper fix would probably involve binding *PACKAGE* to a package in which none of the symbols that are in the content that is about to be written to __loader__._, other than keywords, are present or inherited.
On Nov 8, 2020, at 22:48, Robert Munyer <2433647181@munyer.com> wrote:
[…]
A proper fix would probably involve binding *PACKAGE* to a package in which none of the symbols that are in the content that is about to be written to __loader__._, other than keywords, are present or inherited.
Unfortunately, it doesn’t seem to be as simple as that. Or maybe I am still confused, because apparently I have a bug in the zip-cache used by our falls so I end up calling SYS:CLEAR-ZIP-CACHE. Anyways, I have made [some progress][1] on this, but even with this patch we still fail when loading a fasl the current package doesn't (use :cl) if the source file doesn’t have an explicit :IN-PACKAGE. The failing test works in SBCL/CCL so there is definitely something wonky in our loader implementation. [1]: <https://github.com/armedbear/abcl/pull/352> More when I know it, Mark -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
Hi, On Tue, Nov 10, 2020, 11:50 Mark Evenson <evenson@panix.com> wrote:
On Nov 8, 2020, at 22:48, Robert Munyer <2433647181@munyer.com> wrote:
[…]
A proper fix would probably involve binding *PACKAGE* to a package in which none of the symbols that are in the content that is about to be written to __loader__._, other than keywords, are present or inherited.
Unfortunately, it doesn’t seem to be as simple as that. Or maybe I am still confused, because apparently I have a bug in the zip-cache used by our falls so I end up calling SYS:CLEAR-ZIP-CACHE.
Anyways, I have made [some progress][1] on this, but even with this patch we still fail when loading a fasl the current package doesn't (use :cl) if the source file doesn’t have an explicit :IN-PACKAGE. The failing test works in SBCL/CCL so there is definitely something wonky in our loader implementation.
To be honest, I think the problem is with the FASL generator: it should be prefixing (all) symbols with their packages when that's required to load the FASL back into a similar environment. Regards, Erik.
[1]: <https://github.com/armedbear/abcl/pull/352>
More when I know it, Mark
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 10, 2020, at 15:06, Erik Huelsmann <ehuels@gmail.com> wrote:
[…]
To be honest, I think the problem is with the FASL generator: it should be prefixing (all) symbols with their packages when that's required to load the FASL back into a similar environment.
Regards,
Erik.
Its (almost) doing that, I think. I just have to take into account what package the CL:COMPILE-FILE finds itself in when doing the compilation, which I will tackle next (unless someone beats me to it). -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
To be honest, I think the problem is with the FASL generator: it should be prefixing (all) symbols with their packages when that's required to load the FASL back into a similar environment.
Yes. Here is some debug code: ---------------- ; put this before COMPILE-FILE's definition: (defvar cl-user::*setq-needs-prefix* nil) ; put this around COMPILE-FILE's body: (let ((cl-user::*setq-needs-prefix* (not (eq (find-symbol "SETQ") 'setq)))) ) ; put this before WRITE's definition: (defvar cl-user::*setq-needs-prefix* nil) ; put this before WRITE's body: (when (and cl-user::*setq-needs-prefix* (eq (find-symbol "SETQ") 'setq)) (break "SETQ needs package prefix")) ---------------- Backtracing within that BREAK points a finger at this line in COMPILE-FROM-STREAM: (let ((*package* (find-package '#:cl)) If I add one semicolon: (let (;(*package* (find-package '#:cl)) then all of my own tests pass, though I haven't tried the full suite. It's conceivably possible that _all_ of the manipulation of *PACKAGE* that happens during COMPILE-FILE could be removed, because the user is already required to have *PACKAGE* set correctly when invoking LOAD.
I wrote:
Backtracing within that BREAK points a finger at this line in COMPILE-FROM-STREAM:
(let ((*package* (find-package '#:cl))
If I add one semicolon:
(let (;(*package* (find-package '#:cl))
then all of my own tests pass, though I haven't tried the full suite.
I also get good results if I change that line to: (let ((*package* +keyword-package+)
On Nov 11, 2020, at 03:44, Robert Munyer <2433647181@munyer.com> wrote:
[…]
It's conceivably possible that _all_ of the manipulation of *PACKAGE* that happens during COMPILE-FILE could be removed, because the user is already required to have *PACKAGE* set correctly when invoking LOAD.
I don’t think this is true: COMPILE-FILE should actually capture the current package when COMPILE-FILE is invoked, arranging to have this package present when LOAD occurs. [An example of this][1] works on SBCL/CCL but fails on ABCL even with all of your suggestions. Please correct me if you believe this isn’t the case. [1]: <https://github.com/armedbear/abcl/pull/353/files#diff-886c987127185fcb79c86d6531feefaf00c1365126fc63afa248096f1da49e6eR12> By [explicitly putting an IN-PACKAGE form in our fasl prologue which includes the value of the package that COMPILE-FILE is invoked within, I am able to make all my current tests pass. [2]: https://github.com/armedbear/abcl/pull/353/files#diff-ccaa655ebf1432951ff70b... The new behavior is present in [pull][353]. I am going for a walk while I let the CI do its thing, pondering if I am really testing all the cases, but anticipate merging a consolidated version later today. [353]: https://github.com/armedbear/abcl/pull/353/ yours, Mark -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 11, 2020, at 10:40, Mark Evenson <evenson@panix.com> wrote:
On Nov 11, 2020, at 03:44, Robert Munyer <2433647181@munyer.com> wrote:
[…]
It's conceivably possible that _all_ of the manipulation of *PACKAGE* that happens during COMPILE-FILE could be removed, because the user is already required to have *PACKAGE* set correctly when invoking LOAD.
I don’t think this is true: COMPILE-FILE should actually capture the current package when COMPILE-FILE is invoked, arranging to have this package present when LOAD occurs. [An example of this][1] works on SBCL/CCL but fails on ABCL even with all of your suggestions. Please correct me if you believe this isn’t the case.
Strike “arranging to have this package present when LOAD occurs” as that is certainly not true in general. It’s more that the compiler should take the current package in account when it determines what the symbols it encounters actually refer to. More precise formulations of this are welcome… -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 11, 2020, at 12:00, Mark Evenson <evenson@panix.com> wrote:
On Nov 11, 2020, at 10:40, Mark Evenson <evenson@panix.com> wrote:
On Nov 11, 2020, at 03:44, Robert Munyer <2433647181@munyer.com> wrote:
[…]
It's conceivably possible that _all_ of the manipulation of *PACKAGE* that happens during COMPILE-FILE could be removed, because the user is already required to have *PACKAGE* set correctly when invoking LOAD.
I don’t think this is true: COMPILE-FILE should actually capture the current package when COMPILE-FILE is invoked, arranging to have this package present when LOAD occurs. [An example of this][1] works on SBCL/CCL but fails on ABCL even with all of your suggestions. Please correct me if you believe this isn’t the case.
Strike “arranging to have this package present when LOAD occurs” as that is certainly not true in general. It’s more that the compiler should take the current package in account when it determines what the symbols it encounters actually refer to. More precise formulations of this are welcome…
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
I’m still not quite sure how to describe what I did precisely, but [please see the latest version of my patch][1] for an implementation that seems to pass all of the simple cases of problems with the fasl so one doesn’t have to resort to use of sed. As soon as the [ci][] finishes, and if it succeeds, I intend to merge these changes to trunk to fix [475][]. [1]: <https://github.com/armedbear/abcl/pull/354> [ci]: <https://travis-ci.org/github/armedbear/abcl/builds/742960126> [475]: <https://abcl.org/trac/ticket/475> -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 11, 2020, at 16:15, Mark Evenson <evenson@panix.com> wrote: [475]: <https://abcl.org/trac/ticket/475>
[475][] closed with <https://gitlab.common-lisp.net/abcl/abcl/-/commit/94934a62629f85013d206325084b43ddc1efa401> Please let me know if there are still problems here for you. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
Thank you for working on these! I haven't been able to compile or test your changes, nor to write up the latest bug I found ("compile-file not doing minimal compilation inside defconstant/defparameter/defvar"), because I've been preparing for a not-quite-hurricane. But I do have time to answer your question...
It's conceivably possible that _all_ of the manipulation of *PACKAGE* that happens during COMPILE-FILE could be removed, because the user is already required to have *PACKAGE* set correctly when invoking LOAD.
I don't think this is true: COMPILE-FILE should actually capture the current package when COMPILE-FILE is invoked, arranging to have this package present when LOAD occurs. [An example of this][1] works on SBCL/CCL but fails on ABCL even with all of your suggestions. Please correct me if you believe this isn't the case.
This was discussed and voted on as issue COMPILE-FILE-SYMBOL-HANDLING: http://www.lispworks.com/documentation/HyperSpec/Issues/iss063_w.htm To see the outcome of that vote in the spec, see the subsection labeled "symbol:" in section 3.2.4.4: http://www.lispworks.com/documentation/HyperSpec/Body/03_bdd.htm I haven't looked at your test cases, but some of them may be trying to do things that 3.2.4.4 says that programs are not allowed to do, and compiler implementors are not expected to support.
On Nov 12, 2020, at 08:02, Robert Munyer <2433647181@munyer.com> wrote:
Thank you for working on these!
I haven't been able to compile or test your changes, nor to write up the latest bug I found ("compile-file not doing minimal compilation inside defconstant/defparameter/defvar"), because I've been preparing for a not-quite-hurricane.
But I do have time to answer your question...
It's conceivably possible that _all_ of the manipulation of *PACKAGE* that happens during COMPILE-FILE could be removed, because the user is already required to have *PACKAGE* set correctly when invoking LOAD.
I don't think this is true: COMPILE-FILE should actually capture the current package when COMPILE-FILE is invoked, arranging to have this package present when LOAD occurs. [An example of this][1] works on SBCL/CCL but fails on ABCL even with all of your suggestions. Please correct me if you believe this isn't the case.
This was discussed and voted on as issue COMPILE-FILE-SYMBOL-HANDLING: http://www.lispworks.com/documentation/HyperSpec/Issues/iss063_w.htm
To see the outcome of that vote in the spec, see the subsection labeled "symbol:" in section 3.2.4.4: http://www.lispworks.com/documentation/HyperSpec/Body/03_bdd.htm
I haven't looked at your test cases, but some of them may be trying to do things that 3.2.4.4 says that programs are not allowed to do, and compiler implementors are not expected to support.
Thanks for the reference which I will study. Good luck with the hurricane, and I’ll see ya on the other side. yours, Mark -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
I haven't looked at your test cases, but some of them may be trying to do things that 3.2.4.4 says that programs are not allowed to do, and compiler implementors are not expected to support.
Thanks for the reference which I will study.
I just tried commit 344b4f66, and found that I can remove all of my work-arounds for all of the bugs that I've reported. Thanks! There's one bug that I still need to report, the one about minimal compilation.
Good luck with the hurricane, and I’ll see ya on the other side.
Power failed ~2 min. after I sent that message. Eye of storm was aimed at me with amazing accuracy (exact center point missed me by ~4 mi.) but the leading eye wall and the trailing eye wall both "became disorganized" before they got here, so there was no damage here.
participants (4)
-
Erik Huelsmann -
Mark Evenson -
Pascal Bourguignon -
Robert Munyer