Re: [asdf-devel] [Sbcl-help] cl-fad, sb-grovel and failure to compile constants.lisp-temp
On 18 August 2010 14:49, James Fleming lisp@electronic-quill.net wrote:
The very short version of the question: what changed between 1.0.39 and 1.0.40, regarding the expected locations of source files, and when/how they're expected to be copied there during asdf::perform operations?
ASDF version supplied with SBCL changing is the probable culprit for at least some of this -- adding asdf-devel to CC.
(It seems the NEWS entry for ASDF2 was forgotten, oops...)
However, this bit right here
; compiling file "/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-grovel/def-to-lisp.lisp" (written 15
indicates that something else is wrong too: loading SB-GROVEL should not cause it to be compiled. If your ASDF-LOAD uses :FORCE, that's a the obvious reason, but if not I suspect something funny is going on with timestamps on those files since ASDF feels the need to recompile them.
Cheers,
-- Nikodemus
The context: I'm writing a wiki (yes, yet another one), and its back-end consists of plain-text files while I get to grips with writing a parser.
The text-file wrangling calls on cl-fad, which uses sb-posix, which in turn makes use of sb-grovel. Somewhere between 1.0.39 and 1.0.40, a change was made that broke sb-posix, at least on the two machines I use - both run 64-bit Linux, one on AMD and one on Intel.
The symptom: sb-grovel's def-to-lisp fails to compile at line 244, where it tries to evaluate (compile-file tmp-constants :output-file output-file) after defining 'tmp-constants in this manner: (let ((tmp-constants (merge-pathnames #p"constants.lisp-temp" real-output-file)))
It seems to come down to where source-files are expected to be, i.e. the ~/.cache tree, and how they're copied there. The problem can be reproduced on cue by invoking (asdf:oos 'asdf:load-op :sb-posix). This appears as (asdf-load :sb-posix) below because I'm a lazy git and defined that abbreviation in my .sbclrc.
I've been trying to follow along in the source-code, and the nearest relevant changes appear to be in contrib/asdf/asdf.lisp, but my thinking has been addled by a nasty head-cold. Could somebody please give me a pointer to what I need to look at next? What have I missed? Help!
Thanks in advance for any useful tips, James
P.S. For those who'd like to see exactly what happens, a bleeding great copy-and-paste follows...
- (asdf-load :sb-posix)
; compiling file "/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-grovel/def-to-lisp.lisp" (written 15 AUG 2010 07:51:57 PM): ; compiling (IN-PACKAGE #:SB-GROVEL) ; compiling (DEFVAR *DEFAULT-C-STREAM* ...) ; compiling (DEFUN ESCAPE-FOR-STRING ...) ; compiling (DEFUN SPLIT-CFLAGS ...) ; compiling (DEFUN C-ESCAPE ...) ; compiling (DEFUN AS-C ...) ; compiling (DEFUN PRINTF ...) ; compiling (DEFUN C-FOR-ENUM ...) ; compiling (DEFUN C-FOR-STRUCTURE ...) ; compiling (DEFUN PRINT-C-SOURCE ...) ; compiling (DEFUN C-CONSTANTS-EXTRACT ...) ; compiling (DEFCLASS GROVEL-CONSTANTS-FILE ...) ; compiling (DEFINE-CONDITION C-COMPILE-FAILED ...) ; compiling (DEFINE-CONDITION A-DOT-OUT-FAILED ...) ; compiling (DEFMETHOD PERFORM ...)
; /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-grovel/ASDF-TMP-def-to-lisp.fasl written ; compilation finished in 0:00:00.558 (/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.fasl /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.fasl /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/foo.c /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/a.out /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp)
debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread #<THREAD "initial thread" RUNNING {1002AC20A1}>: failed to find the TRUENAME of /home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp: No such file or directory
Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name): 0: [TRY-RECOMPILING] Try recompiling constants 1: [RETRY ] Retry performing #<ASDF:COMPILE-OP NIL {1003B23C31}> on #<SB-GROVEL:GROVEL-CONSTANTS-FILE "sb-posix" "constants">. 2: [ACCEPT ] Continue, treating #<ASDF:COMPILE-OP NIL {1003B23C31}> on #<SB-GROVEL:GROVEL-CONSTANTS-FILE "sb-posix" "constants"> as having been successful. 3: [ABORT ] Exit debugger, returning to top level.
(SB-IMPL::SIMPLE-FILE-PERROR "failed to find the TRUENAME of ~A" #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp" 2) 0] backtrace
0: (SB-IMPL::SIMPLE-FILE-PERROR "failed to find the TRUENAME of ~A" #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp" 2) 1: ((FLET SB-IMPL::FAIL) "failed to find the TRUENAME of ~A" #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp" 2) 2: (SB-IMPL::QUERY-FILE-SYSTEM #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp" :TRUENAME T) 3: (TRUENAME #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp") 4: ((FLET SB-C::TRY-WITH-TYPE) #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp" "lisp" T) 5: (SB-C::VERIFY-SOURCE-FILE #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp") 6: (SB-C::VERIFY-SOURCE-FILE #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp")[:EXTERNAL] 7: (COMPILE-FILE #P"/home/james/.cache/common-lisp/sbcl-1.0.41-linux-x86-64/opt/pkgs/sbcl-1.0.41/lib/sbcl/sb-posix/constants.lisp-temp")[:EXTERNAL] 8: ((SB-PCL::FAST-METHOD ASDF:PERFORM (ASDF:COMPILE-OP SB-GROVEL:GROVEL-CONSTANTS-FILE)) #<unavailable argument> #<unavailable argument> #<ASDF:COMPILE-OP NIL {1003B23C31}> #<SB-GROVEL:GROVEL-CONSTANTS-FILE "sb-posix" "constants">) 9: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. SB-PCL::.ARG1.)) #<unavailable argument> #<unavailable argument> #<ASDF:COMPILE-OP NIL {1003B23C31}> #<SB-GROVEL:GROVEL-CONSTANTS-FILE "sb-posix" "constants">) 10: ((SB-PCL::FAST-METHOD ASDF::PERFORM-WITH-RESTARTS (ASDF:COMPILE-OP ASDF:CL-SOURCE-FILE)) #<unavailable argument> #S(SB-PCL::FAST-METHOD-CALL :FUNCTION #<FUNCTION #> :PV NIL :NEXT-METHOD-CALL NIL :ARG-INFO (2)) #<ASDF:COMPILE-OP NIL {1003B23C31}> #<SB-GROVEL:GROVEL-CONSTANTS-FILE "sb-posix" "constants">) 11: ((LAMBDA ())) 12: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK)) 13: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-RECURSIVE-LOCK]324)) 14: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK #<CLOSURE (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK) {7FFFF79AB6C9}> #S(SB-THREAD:MUTEX :NAME "World Lock" :%OWNER #<SB-THREAD:THREAD "initial thread" RUNNING {1002AC20A1}> :STATE 1)) 15: ((FLET SB-C::WITH-IT)) 16: ((SB-PCL::FAST-METHOD ASDF:OPERATE (T T)) #<unavailable argument> #<unavailable argument> ASDF:LOAD-OP :SB-POSIX)[:EXTERNAL] 17: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ASDF-LOAD :SB-POSIX) #<NULL-LEXENV>) 18: (INTERACTIVE-EVAL (ASDF-LOAD :SB-POSIX))[:EXTERNAL] 19: (SB-IMPL::REPL-FUN NIL) 20: ((LAMBDA ())) 21: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #<CLOSURE (LAMBDA #) {1002EBD309}>) 22: (SB-IMPL::TOPLEVEL-REPL NIL) 23: (SB-IMPL::TOPLEVEL-INIT) 24: ((LABELS SB-IMPL::RESTART-LISP))
0]
This SF.net email is sponsored by
Make an app they can't live without Enter the BlackBerry Developer Challenge http://p.sf.net/sfu/RIM-dev2dev _______________________________________________ Sbcl-help mailing list Sbcl-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sbcl-help
Looking at the transcript, it seems something is going wrong with ASDF-OUTPUT-LOCATIONS.
It looks like ASDF is trying to build a SYSTEM-WIDE lisp library into YOUR PERSONAL cache directory. This suggests that ASDF-OUTPUT-LOCATIONS is misconfigured. That said, I don't know enough about A-O-T or SBCL installations to give more detailed advice. Suggest looking into the ASDF manual parts that concern ASDF-OUTPUT-LOCATIONS.
I suspect SBCL is shipping with an A-O-T configuration that is wrong.
Best, r
I can't reproduce your issue.
sbcl --eval '(require :asdf)' --eval '(trace compile-file load asdf::perform sb-ext:run-program)' --eval '(require :sb-posix)'
sbcl --eval '(require :asdf)' --eval '(trace compile-file load asdf::perform sb-ext:run-program)' --eval '(asdf:load-system :sb-posix)'
Both work fine for me.
What ASDF configuration do you use?
Could it be that you're running from a dumped image and that you're not exporting SBCL_HOME to a proper value? ASDF does something magic with SBCL_HOME for those output files.
But yeah, I shall submit a patch to SB-GROVEL so that things should work even in screwy cases like that.
Note that you don't need to define a ASDF-LOAD function now that ASDF has ASDF:LOAD-SYSTEM.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
(Adding sbcl-devel to a thread started on sbcl-help.)
Could it be that you're running from a dumped image and that you're not exporting SBCL_HOME to a proper value? ASDF does something magic with SBCL_HOME for those output files.
But yeah, I shall submit a patch to SB-GROVEL so that things should work even in screwy cases like that.
Easier said than done. the perform method in sb-grovel/def-to-lisp.lisp has a short-circuit do-not-grovel, set in sb-posix.asd and sb-bsd-sockets.asd. If I replace its value from :do-not-grovel #.(progn #-sb-building-contrib t) to :do-not-grovel #.(and #-sb-building-contrib t (sb-int:sbcl-homedir-pathname)) loading the stuff loses with: The CC environment variable has not been set in SB-GROVEL. Since this variable should always be set during the SBCL build process, this might indicate an SBCL with a broken contrib installation.
Note: I could reproduce the bug this way: rlwrap xcvb repl (progn (dolist (x (list :sb-bsd-sockets-system :sb-posix-system :sb-posix :sb-grovel)) (delete-package x)) (dolist (x (quote (:sb-posix :sb-grovel))) (asdf::clear-system x))(asdf:initialize-source-registry "/home/fare/src/sbcl//:")(asdf:initialize-output-translations)(asdf::load-system :sb-grovel)(trace sb-grovel::c-constants-extract) (asdf:load-system :sb-posix))
A more complete solution would involve (1) having a warning instead of an error there (2) remembering the build-time value of CC and CFLAGS, etc., and hoping that they are still valid at runtime, if not overridden by the user (?)
OR, it may involve (3) having a way to save a default sb-homedir-pathname in the dumped image, etc.
In any case, it's no trivial endeavor, it requires munging inside the SBCL build process.
WORKAROUND: be sure to include sb-posix and sb-bsd-sockets in your dumped image.
--#f Multiple instances of a same hacker with different context in his mental cache count as multiple hackers wrt documentation and testing needs.
-
Faré -
Nikodemus Siivola -
Robert Goldman