July 2015 - asdf-devel - mailman3.common-lisp.net

The case for merging the minimakefile branch
by Faré 01 Aug '15

01 Aug '15

What makes the minimakefile branch great: * It provides a nicer test experience: the output is quieter and more readable and makes it obvious which tests scripts succeeded or failed, with a nicer summary at the end. All the verbose output is still available in a clearly indicated log file. * It has a nicer failure behavior: more detailed instructions are printed when a test fails, and these instructions, being extracted from the test command, are guaranteed to be in synch with the command, whereas the shell script in the master branch often gets slightly out of synch, and probably is out of synch right now. * It has better-working release functions (for tarballing, making debian packages, etc.) than the master branch. Actually, what's in master currently fails, and I had to use minimakefile for the latest release. A lot of the release scripts could be backported, since even in master, the release is done by bin/asdf-builder which is already a Lisp script — but that would be work. * It is much easier to write and maintain than the shell scripts it replaces; iterating over lists, filtering them, building lists of files, ensuring that the list of files used to bootstrap asdf with make is identical to the list used to bootstrap asdf with asdf, etc. Higher-order functions with good scoping discipline beat shell functions any day. run-tests.sh is an unkempt horror, and so are all the little bits of Makefile magic. * It removes any dependency on perl, sed, dirname, etc. Actually, it removes dependency on any Unix utility, potentially even make, so the tests could run on Windows (with make and a Unix shell, a new ./make.bat file to start asdf-tools might be required, though; on Unix, a ./make.sh could replace make). * You can run and debug all the ASDF tests and all the ASDF release functions in subprocesses from the SLIME REPL instead of running them from the shell. Particularly useful when debugging them with tracing, stack inspection, etc. * It has no change to asdf itself, only to the build, test and release infrastructure. Previously valid objections that have been addressed: * Sure, Lisp scripting is a new thing, but even in master, that's what the release script bin/asdf-builder is, with the same dependency on inferior-shell. What the minimakefile does is generalize this mechanism for tests and for all the things that the Makefile does. * minimakefile uses many Lisp libraries as dependencies. But they have all been added as git submodule's already, and so should actually reduce the dependency on things outside source control, as opposed to all the using utilities used in master (including sed and perl -- ouch). * Sure, our Lisp dependencies themselves are moving targets and might not have completely stable APIs, but by using git submodule, we are protected from any future breakage — we can always keep versions that are known to work for us (as they currently are), or revert to them if we try to upgrade and fail. * Sure this branch is bigger: git diff --stat master minimakefile says there are 1455 insertions, 1199 deletions, which means the asdf-specific code is 21% bigger — but it is also vastly more documented that the Lisp and shell scripts it replaces, and does more things. Actually, if * Sure, writing it from scratch and dealing with the novelty of it would be hard, but it's been done already, the kinks have been ironed out, a few libraries have been spun off of it, and it's even running on Windows. Additionally, merging to master will save me the trouble of maintaining the branch. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org One advantage of living abroad is that there is no pretense whatsoever to represent me as I'm being taxed. TAXATION WITHOUT REPRESENTATION!

1 1

ASDF 3.1.5 is now released!
by Robert Goldman 21 Jul '15

21 Jul '15

1 0

Draft release email
by Robert Goldman 19 Jul '15

19 Jul '15

If you have any feedback about this before I send it out, please respond to the list or directly to me. Thanks!

2 2

restarts vs debugger command
by 73budden . 16 Jul '15

16 Jul '15

Hi! > You 'access' a debugger stack by using handler-bind and restart-case. > Then the user uses the debugger that can use the annotations. This > modular design is a great strength of CL. Yes, I was wrong. It is easy to capture condition with handler-bind and restart-case. But what about "normal" operation? E.g. I use "of-system" to declare and assert that the file is loaded through a correct system. Only a special variable works there as there is no condition at all. > You don't understand what I said. I was thinking of a handler-bind > that does something like > (handler-bind ((some-asdf-condition (e) ... (restart-case (error e) > (edit-file () :report ... (funcall *editor* (condition-component > e)))))) ...) What edit-file does? It can call external editor, wait while user finishes the editing and then retry operation. But in SLIME and Lispworks my code does just some other thing: asynchronous message is sent to IDE so that file is opened in the IDE. The editing is done in another thread and maybe in anoter process (as with SLIME/EMACS). Debugger with all restarts is still available, so user can choose any restart. When user decides what to do, he can invoke the restart he wants. After sending a message a control is returned to handler function. What is the reasonable action of that function? IMO the only reasonable action is to resignal the condition. This is unnecessary blinking, or unnecessary output: user just starting to edit, and resignaling does not supply user with any new useful information. This is the difference between restart and "debugger command". Restart either fixes the problem or resignals condition (maybe some new one). Debugger command (e.g. "show source" or "eval in frame") does another thing: it just evaluates some code in current context and then returns to the debugger without restarting. My extension can be kept separated, but it defines some around method and thus can interfere with other extensions which could also define the same method. This is not very good. Also maybe I can use some wrapper function to asdf::oos instead of patching asdf, but it will not work in the cases where load-system is called implicitly, e.g. from ql:quickload.

3 7

Almost there
by Faré 16 Jul '15

16 Jul '15

Dear lispers, we're ready to bless ASDF 3.1.4.25 as release 3.1.5, and now is a good time to test it before it's too late. Anton, do you have time and/or resource for a run of cl-test-grid? Nothing should have changed in the semantics of ASDF itself since the last run of cl-test-grid, so I don't expect any discrepancy. We did add better immutable-system support thanks to Dave Cooper, but that isn't cover by cl-test-grid, only by our regression tests and his usage. What we did change was to make UIOP more robust and portable. Robert Goldman and I, with the help of Dave Cooper, ran extensively tests. We have notably debugged a lot of pathname and run-program issues on Windows. These issues recently cropped up since we have added a lot of tests since last release, and since neither of us is a Windows user, they had previously only been run on Unix. We also made an important compatibility fix for SBCL, added CLASP support, enhanced support for new LispWorks 7, fixed chdir for ABCL and XCL. As far as I'm concerned, this means that portable scripting in Common Lisp is more a reality than ever. If you are careful, you can write portable CL code that runs as is on Unix (including Linux and MacOS) and Windows. As usual, the more portable you want to be, the more careful you have to be; but at least UIOP covers all the bases with a reasonable portable interface (please don't use anymore any of cl-fad, trivial-shell, run-shell-command, trivial-backtrace; use uiop). I recommend CCL for a portable scripting implementation, because SBCL on Windows isn't quite as good (and is not able to call out to CMD.EXE) and CLISP has too many quirks (and is looking for a new maintainer to release it anew after 5 years). For examples of scripting in CL, see https://github.com/fare/fare-scripts —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org In a country where the sole employer is the State, opposition means death by slow starvation. The old principle: who does not work does not eat, has been replaced by a new one: who does not obey shall not eat. — Leon Trotsky

8 27

Finding all the systems that subclass
by Javier Olaechea 09 Jul '15

09 Jul '15

Hi, I'm trying to find all the systems that use a particular subclass of system. The goal is that through that mechanism they signal/register themselves as modules of another system. In order to list all such modules I though map-systems would do the trick: (remove-if #'null (let (results) (asdf:map-systems (lambda (system) (when (stumpwm-module-p system) (setf results (cons system results))))) results)) However it appears map-systems only works on when the system *definition* has been loaded (is that the proper term?). Is there a way to load all the system definitions in asdf's path? (That is according to *source-registry* and *central-registry*) Cheers, Javier -- "I object to doing things that computers can do." — Olin Shivers

2 1

Editing component being compiled when compilation failed.
by 73budden . 08 Jul '15

08 Jul '15

Hi! I still don't know what is the official way to edit component being compiled. I asked at comp.lang.lisp but had no reply. But I have developed my own solution. For this I had to redefine several asdf functions. I bind special variables so that current component is known at the time of compilation. Also I define asdf::e and asdf::ep symbol macros that access special variables. When compilation fails, I have an EDIT-COMPONENT restart, which is not needed - it is just a help string. Then user should eval asdf::e in the debugger to edit component source. User should eval asdf::ep in the debugger to edit component's system source. Code is here, it shoud work in asdf-3.1.4 under lispworks 6 or under SBCL+SLIME. https://bitbucket.org/budden/budden-tools/src/866bcf15ad5c793868a0a31e60247… I'd like to receive any feedback.

3 6

Note to myself
by Robert Goldman 08 Jul '15

08 Jul '15

I have a note to myself to remove SOURCE-FILE-TYPE from the manual, and I see that it's there just for backwards compatibility. Am I correct in saying that this should be replaced by COMPONENT-TYPE? That's what the comment says, but then I see this: (defmethod source-file-type ((component file-component) (system parent-component)) (file-type component))) which suggests FILE-TYPE, not COMPONENT-TYPE is The Right Thing. Also, what the heck is the return type of this? I see the following: (defmethod source-file-type ((component parent-component) (system parent-component)) :directory) which suggests that it's something like (or string (eql :directory)). Ugh. Shouldn't asking for the file-type of a PARENT-COMPONENT simply be an error? Sorry to bother you about this: just trying to kill something that's languished on my todo list for more than a year. cheers, r

2 2

Testing running subsidiary lisps with multiple Allegros
by Robert Goldman 07 Jul '15

07 Jul '15

How is this to be managed? In the test suite, there are environment variables that signal which of the many Allegro flavors one is testing. But somehow this information needs to get pushed through the tests into the subsidiary lisp-invocations. How is this managed? Thanks, r

4 4

Tests ran overnight
by Robert Goldman 07 Jul '15

07 Jul '15

...not on Windows, I'm afraid: my automation isn't up to that yet. But all tests passed on Mac OS X. On linux, though, I got the following: Unexpected test failures on these implementations: build/results/allegro8-test.text build/results/allegromodern8-test.text build/results/allegromodern-test.text build/results/allegro-test.text build/results/cmucl-test.text For Allegro, I see test-program failing: TEST ABORTED: In (GETENV-PATHNAME "ALLEGRO"), invalid pathname #P"alisp8/": Expected an absolute pathname Evaluation stack: Warning: while extracting symbols from /lib/i386-linux-gnu/librt.so.1: couldn't find .symtab section Warning: while extracting symbols from /lib/i386-linux-gnu/libm.so.6: couldn't find .symtab section Warning: while extracting symbols from /lib/i386-linux-gnu/libdl.so.2: couldn't find .symtab section Warning: while extracting symbols from /lib/i386-linux-gnu/libc.so.6: couldn't find .symtab section Warning: while extracting symbols from /lib/ld-linux.so.2: couldn't find .symtab section Warning: while extracting symbols from /lib/i386-linux-gnu/libnss_files.so.2: couldn't find .symtab section Warning: while extracting symbols from /lib/i386-linux-gnu/libnss_compat.so.2: couldn't find .symtab section Warning: while extracting symbols from /lib/i386-linux-gnu/libnsl.so.1: couldn't find .symtab section Warning: while extracting symbols from /lib/i386-linux-gnu/libnss_nis.so.2: couldn't find .symtab section Above backtrace due to this condition: In (UIOP/FILESYSTEM:GETENV-PATHNAME "ALLEGRO"), invalid pathname #P"alisp/": Expected an absolute pathname The .symtab errors imply we are trying to build a program assuming a 32-bit execution model, instead of a 64-bit one. I have had this problem on Allegro before. I see that warning on allegro8-test.text, allegromodern-test.text, allegromodern8-test.text, but not on allegro-test.text. That's my fault, because I replaced my old 32-bit license with a 64-bit one, and it looks like there's some way to find the old license. This is a potential real issue, though: we have often found problems linking with C code on Allegro when we have only 32-bit Allegro on a 64-bit platform. It's getting less and less common to rig up one's 64-bit linux box with a full 32-bit build environment as well as the 64-bit one. CMUCL has a different error: stderr: This is make-hello-world, testing its stderr. 0: RUN-PROGRAM returned NIL NIL 0 - then use this executable image 0: (RUN-PROGRAM ("/home/rpg/common-lisp/asdf/build/fasls/cmu-20f___20f_unicode_-linux-x86/asdf/test/hello-world-example--all-systems.image" "-quiet" "-noinit" "-batch" "-eval" ...) :OUTPUT :LINES :ERROR-OUTPUT ...) 0: RUN-PROGRAM returned ("hello, world") NIL 0 - now, use this executable image with arguments 0: TEST ABORTED: Type-error in KERNEL::OBJECT-NOT-TYPE-ERROR-HANDLER: #<Function EQ {100138F1}> is not of type SEQUENCE 0: (DEBUG:BACKTRACE 69 #<Synonym Stream to SYSTEM:*STDOUT*>) 1: (UIOP/IMAGE:RAW-PRINT-BACKTRACE :STREAM #<Synonym Stream to SYSTEM:*STDOUT*> :COUNT 69 :CONDITION #<unused-arg>) 2: ("WITH-UPGRADABILITY NIL") 3: ("WITH-UPGRADABILITY NIL") 4: (LISP::%WITH-STANDARD-IO-SYNTAX #<Closure Over Function "WITH-UPGRADABILITY NIL" {585234D1}>) 5: (UIOP/IMAGE:PRINT-CONDITION-BACKTRACE #<TYPE-ERROR {5850089D}> :STREAM #<Synonym Stream to SYSTEM:*STDOUT*> :COUNT 69) 6: (NIL #<TYPE-ERROR {5850089D}>) 7: (NIL #<TYPE-ERROR {5850089D}>) 8: (SIGNAL #<TYPE-ERROR {5850089D}>) 9: (ERROR TYPE-ERROR :FUNCTION-NAME REVERSE :DATUM #<Function EQ {100138F1}> :EXPECTED-TYPE SEQUENCE) 10: (KERNEL::OBJECT-NOT-TYPE-ERROR-HANDLER REVERSE #<System-Area-Pointer: #x3FFFB864> #<Alien (* SYSTEM:SYSTEM-AREA-POINTER) at #x3FFFB40C> (144 16)) 11: (KERNEL::INTERNAL-ERROR #<System-Area-Pointer: #x3FFFB40C> #<unused-arg>) 12: ("call_into_lisp+#x74 [#x8059750] /usr/local/bin/cmucl") 13: ("funcall2+#x2C [#x805927C] /usr/local/bin/cmucl") 14: ("interrupt_internal_error+#x7A [#x805201A] /usr/local/bin/cmucl") 15: ("sigtrap_handler+#x183 [#x8059593] /usr/local/bin/cmucl") 16: (REVERSE 1 #<Function EQ {100138F1}>)[:EXTERNAL] 17: (DEBUG::PRINT-FRAME-CALL-1 #<Compiled-Frame UIOP/RUN-PROGRAM:RUN-PROGRAM>) 18: (DEBUG::PRINT-FRAME-CALL #<Compiled-Frame UIOP/RUN-PROGRAM:RUN-PROGRAM> :PRINT-LENGTH NIL :PRINT-LEVEL NIL :VERBOSITY 1 :NUMBER NIL) 19: ("DEFUN TRACE-START-BREAKPOINT-FUN" #<Compiled-Frame UIOP/RUN-PROGRAM:RUN-PROGRAM> #<unused-arg>) 20: (DEBUG-INTERNALS::INVOKE-BREAKPOINT-HOOKS (#<Breakpoint UIOP/RUN-PROGRAM:RUN-PROGRAM:FUNCTION-START> #<Breakpoint UIOP/RUN-PROGRAM:RUN-PROGRAM:FUNCTION-START>) #<Code Object "WITH-UPGRADABILITY NIL" {581163D7}> 617) 21: (DEBUG-INTERNALS::HANDLE-BREAKPOINT-AUX (#<Breakpoint UIOP/RUN-PROGRAM:RUN-PROGRAM:FUNCTION-START> #<Breakpoint UIOP/RUN-PROGRAM:RUN-PROGRAM:FUNCTION-START>) #<Breakpoint-Data UIOP/RUN-PROGRAM:RUN-PROGRAM at 617> 617 #<Code Object "WITH-UPGRADABILITY NIL" {581163D7}> #<System-Area-Pointer: #x3FFFBF4C>) 22: ("call_into_lisp+#x74 [#x8059750] /usr/local/bin/cmucl") 23: Above backtrace due to this condition: Type-error in KERNEL::OBJECT-NOT-TYPE-ERROR-HANDLER: #<Function EQ {100138F1}> is not of type SEQUENCE Script failed I'm not seeing an obvious explanation for this failure. Looks like it would be some kind of place where EQ appears in the wrong position, but the backtrace is unhelpful, and I don't know how to work through all the intricacies of LISP-INVOCATION. When I try to make this run at the command line, I fail as follows: > cmucl -core /home/rpg/common-lisp/asdf/build/fasls/cmu-20f___20f_unicode_-linux-x86/asdf/test/hello-world-example--all-systems.image Invalid magic number: 0x464c457f should have been 0x434f5245. ...but that seems like a different error, since the above transcript seems to suggest that the lisp invocation worked, but then we tried to test some result and encountered an error.

2 1