- asdf-devel - mailman3.common-lisp.net

RFC: Image dumping and program-op
by Elias Pipping 16 Sep '16

16 Sep '16

Dear list, from within a REPL, oftentimes, you can save the current status and create a file that’ll allow you to continue where you are now at a later point in time. This file could be a standalone executable or a core file that requires a lisp environment to run (e.g. via `lisp -core …`). These two cases can be subdivided into four overall use cases. (exe-repl) When the executable is run, you want it to print a banner and take you to a REPL. Maybe you’ve created this file after applying a patch to your lisp and you don’t want to do that over and over again. (exe-application) You’ve created an executable that should not print a banner and run an entirely new main() function. (core-repl) Analogous to the exe-repl case, but smaller at the cost of requiring a lisp to run (core-application) Analogous to the exe-application case (not sure if anyone does this). The function `uiop:dump-image` can be used to create images, where ‘image’ is an umbrella term for both standalone executables and cores. With :executable t, it creates the former; with :executable nil, the latter. The image is only told whether we want a REPL or application type image once it’s run and uiop:restore-image hands control over to what uiop:*image-entry-point*. That seems to work for some lisps, but for CMU CL it does not. CMU CL’s save-image provides options like :process-command-line t/nil :print-herald t/nil Whether the herald/banner is printed on startup is decided when save-image is called. It cannot be changed later. If you’re writing an application with a main method, and you don’t pass :print-herald nil, the banner will be printed before your application does anything else. If you want to read a list of options and arguments from the command line in your main method, this will not work if you passed :process-command-line nil to save-image, since then the command line arguments will be frozen (i.e., equal what they were when you called save-image). This suggests that for uiop:dump-image to cover relevant use cases on CMU CL, it will have to learn about the repl/application distinction, e.g. by growing another option. I’ve created a sample application that illustrates this and put it here(*): https://github.com/pipping/cl-echo echo.lisp does what its name suggests: It tells you what it is that you passed to it. If I want to create an exe-application type image from it, on sbcl e.g. that’ll work with uiop:dump-image and I can even use asdf’s program-op (please see build.lisp for how) For CMU CL, (uiop:dump-image :executable t) and asdf's program-op do something that I do not want: They leave the command line arguments frozen so that the resulting executable does not return what I want it to. I need something that’s different from what asdf’s program-op in build.lisp does (in particular, not pass :process-command-line nil), as is shown in build-cmucl.lisp. Were I to pass :print-herald nil with :executable t in uiop:dump-image on CMU CL, I’d make the exe-application case work as expected but not the exe-repl case. Hence my claim that we need a new option. If we add another option to dump-image, the question is how asdf's image-op and program-op should behave (here, the word image refers to core files only, as far as I can tell). Will we need new options here, too? If so, what should the defaults be? This write-up is the result of a conversation with Robert. Thanks again, Robert! I’d welcome feedback. Elias (*) The current version at the time of this writing is https://github.com/pipping/cl-echo/tree/faea91b6aa14ea81f2130edef3f6abccbd5… so if my notes don’t seem to make any sense, please give that revision a try.

2 2

Should marking a system preloaded drop its build information?
by Faré 15 Sep '16

15 Sep '16

In the patch he submitted, Daniel Kochmanski modified register-preloaded-system so that it eagerly replaces any previously loaded system with a placeholder object that has no build information. In the current ASDF, the preloaded system is only lazily registered if the system is requested but not previously loaded (or is cleared). In my current working branch, it is eagerly registered but only if no previous system was loaded or after clearing it. [Going from lazy to eager registration fixes a bug with already-loaded-systems and thus require-system.] The big design question is: should registering a system as preloaded drop its build information? Advantage of erasing the existing system information: the behavior of the image is uniform on all machines, and trying to resume development on a machine without the same source code at the same location won't cause a flurry of filesystem accesses at the wrong paths. [Note that immutable systems never try to access the filesystem, unlike regular preloaded systems, even with build information present.] Advantage of keeping the existing build information: you can resume the production image and resume incremental development on a machine that has the same source code at the same location. And if you to drop it, you can explicitly call clear-system after you call register-preloaded-system. I am strongly inclined to keeping the build information around: it's always easy to erase the information later, but hard to reconstitute dropped information. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org A Libertarian Constitutional Amendment: Congress shall make no law.

4 6

Windows status
by Robert Goldman 15 Sep '16

15 Sep '16

Merged Faré's patch and bumped to 3.1.17.20. Mostly working on Windows now, but still have one problem: run-program is failing on ecl_bytecodes. That's for tomorrow, though.

1 0

Immutable Systems
by Faré 15 Sep '16

15 Sep '16

Dear Dave and ASDF users, some of you like Dave may be using immutable-systems. Daniel Kochmanski noticed I made the mistake of advertising *immutable-systems*. I'm considering maybe deprecating this variable in a future release. If you're just accessing it to declare systems immutable, please instead use, e.g. (map () 'asdf:register-immutable-system (asdf:already-loaded-systems)) If you're accessing it to get a list, I'll export a new functional interface. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Associate yourself with people of good quality, for it is better to be alone than in bad company. ― Booker T. Washington (1856–1915)

2 2

Windows problems: moratorium continues
by Robert Goldman 14 Sep '16

14 Sep '16

Unfortunately, the last time I was able to run the Windows tests before 2 days ago was in mid-August. Since then, there have been a flurry of commits, many of which could have impacted the state of test-force.script. Indeed, test-force.script *itself* has changed, so that, although we call it a "regression test," we have no reason to believe that it *ever* worked on Windows in its present form. Unfortunately, it may take me a while to get enough bisecting done to figure out when we jumped the rails. I'll keep everyone posted, but in the meantime, we should consider the repo locked against more changes. Windows is badly broken, and we shouldn't be trying to move forward again until we have restored function on this platform. thanks, r

2 1

Re: Windows problems: moratorium continues
by Robert Goldman 14 Sep '16

14 Sep '16

On 9/14/16 Sep 14 -6:14 AM, Faré wrote: > I don't know whether that would help on Windows, but I have a fix for > the require-system function that hopefully makes test-force stable, in > !16 on gitlab. Mostly I'm working my way back through history, then forward. 3.1.7.16 was broken because of the typo. 3.1.7.17 has the typo fixed, but fails test-clean-load on ecl_bytecodes. I believe that this is spurious, though. It's crashing on the code that hooks into REQUIRE, and it crashes only when run as make test-clean-load l=ecl_bytecodes and NOT when run as run-tests.sh -c ecl_bytecodes Here's the error: An error occurred during initialization: In form (PROGN (PUSHNEW '("fasb" . SI:LOAD-BINARY) *LOAD-HOOKS* :TEST 'EQUAL :KEY 'CAR) (UNLESS (ASSOC "asd" *LOAD-HOOKS* :TEST 'EQUAL) (APPENDF *LOAD-HOOKS* '(("asd" . SI:LOAD-SOURCE)))) (DEFVAR *WRAPPED-MODULE-PROVIDER* (MAKE-HASH-TABLE)) (SETF (GETHASH 'MODULE-PROVIDE-ASDF *WRAPPED-MODULE-PROVIDER*) 'MODULE-PROVIDE-ASDF) (DEFUN WRAP-MODULE-PROVIDER (PROVIDER NAME) (LET ((RESULTS (MULTIPLE-VALUE-LIST (FUNCALL PROVIDER NAME)))) (WHEN (FIRST RESULTS) (REGISTER-PRELOADED-SYSTEM (COERCE-NAME NAME))) (VALUES-LIST RESULTS))) (SETF *MODULE-PROVIDER-FUNCTIONS* (LOOP :FOR PROVIDER :IN *MODULE-PROVIDER-FUNCTIONS* :COLLECT (ENSURE-GETHASH PROVIDER *WRAPPED-MODULE-PROVIDER* #'(LAMBDA (MODULE-NAME) (WRAP-MODULE-PROVIDER PROVIDER MODULE-NAME)))))) Wrong number of arguments passed to function NIL.. I have checked what I thought was the obvious hypothesis: a NIL in *MODULE-PROVIDER-FUNCTIONS* -- but that is not the cause. And why the intercession of "make" should matter, I cannot say. I'm just not sure if there's any version that passes the tests on Windows since 16 August 2016. I'll have another look, but may have to give up the bisect strategy and just try to fix the head. This is obviously deeply undesirable. But bisect assumes that I can detect the interesting failures in a hill-climbing way, whereas here we have multiple points at which breakage has been introduced. Unfortunately, unlike on Mac and Linux, I have absolutely no clue how to automate Windows testing, which leads to these messes, which are precisely what continuous integration is supposed to avoid. best, r > > > On Tue, Sep 13, 2016, 23:05 Robert Goldman <rpgoldman(a)sift.net > <mailto:rpgoldman@sift.net>> wrote: > > Unfortunately, the last time I was able to run the Windows tests before > 2 days ago was in mid-August. Since then, there have been a flurry of > commits, many of which could have impacted the state of > test-force.script. > > Indeed, test-force.script *itself* has changed, so that, although we > call it a "regression test," we have no reason to believe that it *ever* > worked on Windows in its present form. > > Unfortunately, it may take me a while to get enough bisecting done to > figure out when we jumped the rails. I'll keep everyone posted, but in > the meantime, we should consider the repo locked against more changes. > Windows is badly broken, and we shouldn't be trying to move forward > again until we have restored function on this platform. > > thanks, > r > >

3 7

ASDF master head with attached patches passes all tests on MKCL master head!
by Jean-Claude Beaudoin 14 Sep '16

14 Sep '16

Hello ASDF-devs, After pulling all latest commits from ASDF master and thus having a ASDF 3.1.7.19 that git describes as "debian/2%3.1.5-1-514-g7b7b9f0", and patching it with the two attached patches, and using an mkcl from its latest github master head, I then get the following result: -#--------------------------------------- Using /home/jean-claude/CL/MKCL/dev/releases/mkcl-1.1/git/mkcl.git/src/test/asdf/../../bin/mkcl Ran 61 tests: 61 passing and 0 failing all tests apparently successful -#--------------------------------------- ./test/run-tests.sh -c mkcl GOOD: Loading ASDF on mkcl produces no message ./test/run-tests.sh -l mkcl Loading all these systems: ;;; Loading "/home/jean-claude/CL/MKCL/dev/releases/mkcl-1.1/git/mkcl.git/src/test/asdf/build/asdf.lisp" Done! Script succeeded

3 2

Test results
by Robert Goldman 13 Sep '16

13 Sep '16

I ran the tests again on all three platforms. Linux: Unexpected test success on ecl-bytecodes (ECL 16.1.2) in this block: ;; forced, it should be later (DBG "Check that force reloading loads again") (setf test-package::*file3* :reset) (load-system 'test-asdf/force :force :all) (assert-compare (>= (get-file-stamp file1) file1-date)) (assert-equal test-package::*file3* t) I do *not* see this on ECL non-bytecodes. The tests here fail as before. Windows: Allegro, ccl, ECL-bytecodes, and SBCL all fail test-force. ECL-bytecodes also fails test-run-program. These seem to be failing here: (DBG "Check that require-system of touched .asd will reload the asdf.") (DBG "(That's what it does now, but if it could be fixed that'd be nice.)") (unset-asdf-cache-entry '(locate-system "test-asdf")) (unset-asdf-cache-entry '(find-system "test-asdf")) (unset-asdf-cache-entry '(find-system "test-asdf/force")) (touch-file "test-asdf.asd" :timestamp (+ 10000 (get-file-stamp file1))) (require-system 'test-asdf/force) (assert-equal (asymval :*times-loaded* :test-asdf-system) 2) Looks like *TIMES-LOADED* is evaluating to 1 instead of 2. Anyone know why this would have changed? Mac: Everything passes. Thx, r

3 12

[rfc][merge request]
by Daniel Kochmański 13 Sep '16

13 Sep '16

Dear all, me and Fare clearly can't reach a consensus about the changes I've recently proposed to ASDF: https://gitlab.common-lisp.net/asdf/asdf/merge_requests/13 I acknowledge a possibility, that I'm totally wrong here (and if that's true, my deepest apologies to Fare for using vast amount of his time), but my conviction is that this is a step forward and a good proposition. I would appreciate 3rd party opinions, because I'm clearly not objective here since I've wrote the code in question. The code isn't in final shape yet, but it proves no regressions wrt tests shipped with ASDF right now (tested on ABCL, CCL, ECL, SBCL 1.2.0, SBCL and MKCL). I'm planning to write a proper regressions and feature tests and document the behaviour in question in the manual, but only if there is sense in it (i.e it will be merged after fixing the potential issues). Thanks, Daniel Kochmański -- Daniel Kochmański ;; aka jackdaniel | Poznań, Poland TurtleWare - Daniel Kochmański | www.turtleware.eu "Be the change that you wish to see in the world." - Mahatma Gandhi

4 3

Moratorium on ASDF commits, please!
by Robert Goldman 12 Sep '16

12 Sep '16

I'm seeing a lot of failures on Windows. I'd like to take stock of this before anything more gets changed. thanks, r

1 0