On Thu, Mar 13, 2014 at 10:07 PM, Anton Vodonosov <avodonosov@yandex.ru> wrote:

The tests on 3.1.0.94 have completed.

No regressions detected, here is the report: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-37.html

The only failure is teepeedee2, which fails because of its own problem - it includes in its source code libraries like babel, incompatible with babel from quicklisp.

The lisps tested: abcl-1.2.0-fasl42-linux-x86 abcl-1.2.1-fasl42-linux-x86 ccl-1.9-f96-linux-x86 clisp-2.49-unix-x86 cmu-snapshot-2014-01__20e_unicode_-linux-x86 ecl-13.5.1-unknown-linux-i686-bytecode ecl-13.5.1-unknown-linux-i686-lisp-to-c sbcl-1.1.11-linux-x86

I am now preparing to test the patch. Fare, you say on the lates SBCL, therefore I have build sbcl-1.1.16. Will test on it.

Thanks a lot! The quality of ASDF releases just wouldn't be the same without you. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Demand Truly Free Health Care: where doctors need neither pay, food, sleep nor training, and hospitals grow on trees!

14.03.2014, 20:02, "Anton Vodonosov" <avodonosov@yandex.ru>:

Here is the report for the syntax patch: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-38.html

As systems depend on each other, a failing system breaks other systems. To get sense of the situation the following report may help: http://common-lisp.net/project/cl-test-grid/asdf/asdf-load-failures-3.1.0.94... It includes only systems which fail with the syntax pach, while load OK without the patch. The default sorting places more "important to fix" systems at top. The term "root-blocker" means a system failing by itself, i.e. none of its dependencies fail. For example atdoc has 1 root-blocker - clozure-common. If we fix clozure-common, atdoc will hopefully work (unless atdoc it has its own problem, which will be revealed in this case). The systems which have count of root-blockers = 0 fail by themselves. You may change sorting by clicking columns (holding Shift to sort by multiple columns). Let me know if you have questions about this report. Best regards, - Anton

On Fri, Mar 14, 2014 at 12:24 PM, Anton Vodonosov <avodonosov@yandex.ru> wrote:

14.03.2014, 20:20, "Anton Vodonosov" <avodonosov@yandex.ru>:

...

As systems depend on each other, a failing system breaks other systems.

To get sense of the situation the following report may help: http://common-lisp.net/project/cl-test-grid/asdf/asdf-load-failures-3.1.0.94... It includes only systems which fail with the syntax pach, while load OK without the patch.

The default sorting places more "important to fix" systems at top.

The term "root-blocker" means a system failing by itself, i.e. none of its dependencies fail.

For example atdoc has 1 root-blocker - clozure-common. If we fix clozure-common, atdoc will hopefully work (unless atdoc it has its own problem, which will be revealed in this case).

The systems which have count of root-blockers = 0 fail by themselves.

You may change sorting by clicking columns (holding Shift to sort by multiple columns).

Let me know if you have questions about this report.

Also "systems-blocked-exclusively" are systems which have no other failing dependencies except for the current system.

For example, ironclad has 120 systems blocked exclusively. This means if we fix ironclad the 120 systems may be fixed (Unless they have their own problems, which will be revealed in this case; but all the dependencies for these systems will be fixed).

Thanks a lot for your testing, Anton. This suggests that we can't transition to a cleaner syntax by "just" fixing one or two systems. We need to provide some migration path and/or some configuration magic so that old systems known to fail keep using dirty syntax, but new systems use a clean syntax. Completing the migration is clearly beyond the 3.1.1 release. Maybe providing the cleaner alternative could be done before, but even that is doubtful. An out-of-band system configuration scheme needs much thought, and no one seems to have ideas how to do it. Another useful test would be to determine how many of these failures happen when just the asd has a forced clean syntax. Probably a few still. Another solution might be one readtable per system and/or per set of systems using the same .asd file, i.e. defsystem remembers the readtable it was defined in, and load-asd binds the *readtable* and *print-pprint-syntax* to a new table. On the one hand, this might entrench bad syntax hygiene and make the loading of files noticeably different when done from ASDF than when done from the REPL — unless you use slime-asdf in a way that ensures files are compiled using ASDF. On the other hand, we're already heading that way with all the around compile hooks and manual around methods, etc. Why this matters: http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html#%28part._.Safety_before... —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Malthus was right. It's hard to see how the solar system could support much more than 10^28 people or the universe more than 10^50. — John McCarthy

On Fri, Mar 14, 2014 at 4:01 PM, Anton Vodonosov <avodonosov@yandex.ru> wrote:

...

Why this matters: http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html#%28part._.Safety_before...

You say "Creating a fresh copy of the standard readtable around each action is too expensive."

But why expensive? On CCL it takes 22 microseconds per readtable:

OK, on my machine (Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz running at 1200 MHz), it takes 4.3 µs on CCL and 3.2 µs on SBCL. for l in sbcl ccl ; do cl -l $l "(uiop::implementation-identifier)" "(time (dotimes (i 1000000) (copy-readtable)))" done sbcl-1.1.14.43.master.1-9f3f7eb-linux-x64 Evaluation took: 3.385 seconds of real time 3.372000 seconds of total run time (3.360000 user, 0.012000 system) [ Run times consist of 0.196 seconds GC time, and 3.176 seconds non-GC time. ] 99.62% CPU 8,781,837,804 processor cycles 5,903,919,248 bytes consed NIL ccl-1.10-f96-linux-x64 (DOTIMES (I 1000000) (COPY-READTABLE)) took 4,202,553 microseconds (4.202553 seconds) to run. 394,066 microseconds (0.394066 seconds, 9.38%) of which was spent in GC. During that period, and with 4 available CPU cores, 4,164,000 microseconds (4.164000 seconds) were spent in user mode 40,000 microseconds (0.040000 seconds) were spent in system mode 1,696,000,000 bytes of memory allocated. 924 minor page faults, 0 major page faults, 0 swaps. NIL Robert, what do you think? Should we "just" copy-readtable and copy-pprint-dispatch around each action? Or should we try to share the tables for all files in a system? Should we try to have defsystem save the values of *readtable* and *print-pprint-dispatch*, while load-asd does the creation of fresh tables? Should systems and/or individual components have a more general purpose variables and variable-values slot, which should be passed to progv, and be initialized from an alist? Also, as for the implementation of such bindings, one issue I see with the progv thing is that the more specific :around methods will be the outer ones, so that our least specific default :around method will be the innermost, and if it binds special variables, they will shadow bindings from the user's :around method, e.g. in ironclad. I'm not sure how to fix that in a backward-compatible way, short of reintroducing the good/bad old asdf:around method combination — we could of course split our a new method, just like we did with perform-with-restarts. But that's quite ugly. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Microphones, liberating singers from having to be loud to be heard, gave back to each language a music that matches its own prosody.

Quick follow-up: what about blowing this discussion into launchpad under the rubric of "Manage CL syntax" or something like that. I would prefer not to lose the discussion. I believe this is a place where a tool like POIU or XCVB really veers away from vanilla CL usage. Vanilla CL usage encourages* actions during building that side-effect the global state of the image. These actions may be bad if one wishes to build individual components in multiple different CL images. But they are natural to people developing code incrementally and interactively using a REPL. I don't want ASDF to drift into being a tool that prescriptively moves people away from that practice. I think that's too presumptive for a piece of infrastructure. I don't think it's my job to wreck code, just because it's bad style. And I don't think proper use of the readtable or of pretty-print dispatch is obvious enough that people will recover well if we mess with it. As you have said, I am inclined to be much more conservative about changes like this. Best, r * I am not endorsing or deprecating this encouragement.

Faré wrote:

On Fri, Mar 14, 2014 at 6:10 PM, Robert Goldman <rpgoldman@sift.net> wrote:

...

Quick follow-up: what about blowing this discussion into launchpad under the rubric of "Manage CL syntax" or something like that.

I would prefer not to lose the discussion.

I believe this is a place where a tool like POIU or XCVB really veers away from vanilla CL usage.

Vanilla CL usage encourages* actions during building that side-effect the global state of the image. These actions may be bad if one wishes to build individual components in multiple different CL images. But they are natural to people developing code incrementally and interactively using a REPL.

I don't want ASDF to drift into being a tool that prescriptively moves people away from that practice. I think that's too presumptive for a piece of infrastructure. I don't think it's my job to wreck code, just because it's bad style. And I don't think proper use of the readtable or of pretty-print dispatch is obvious enough that people will recover well if we mess with it.

As you have said, I am inclined to be much more conservative about changes like this.

Here is my theory: safe code needs to respect hygiene, because you never know which system are loaded before or after: it depends on the overall plan of the toplevel target system. Therefore is it never correct to rely on other systems having set a particular readtable, and it is never correct to side-effect the current readtable into something that modifies standard characters — or any character that any other program might want to use, for the matter. If you side-effect the readtable, and the readtable escapes, you are going to hurt someone, somewhere.

In the current setup, it is therefore a bug if you side-effect the readtable, and those programs that do are buggy. Using named-readtables is fine, but only if you reset the readtable to the standard one at the end of every file, or have a method that binds *readtable* around your compilation.

I see two ways forward: (1) recognize that it is a bug, and help programmers by being strict, modulo some compatibility mode for older systems until they are fixed, or (2) slightly change the semantics so that it's not a bug anymore, by making the readtable either per-file or per-system; then you can modify the readtable to your heart's content, because it's private.

My initial reaction was (1), and that's more or less the patch I sent to anton, who tells us it breaks a lot of things. At Anton's suggestion, I considered (2) and it looks promising — but we need to modify the protocol and/or use a new method-combination to achieve the effect in a backward compatible way.

No, we cannot do either (1) or (2). CL does not say it is a bug to rely on an upstream-loaded file to set the readtable to a value that you expect. So breaking code that makes that assumption by forcing file-scoped readtables and pprint dispatch tables on the programmer is not an OK thing for us to do. It's not ASDF's job to break code that we consider to be bad code. The CL spec went out of its way to *permit* the code that you are arguing to be buggy. They *could* have demanded a readtable argument, but they did not. They could have imposed a scope on readtables, but they did not. Furthermore, we are not omniscient: the programmer could knowingly count on a non-file-scoped READTABLE in a way that s/he knows is safe, because s/he can prove to him/herself that *READTABLE* will be properly bound at all locations in the code. That information could be perfectly correct, but invisible to ASDF. We should not break such code. Making constructs file-scoped willy-nilly is not something that ASDF should do. In my opinion, it's overreach. To analogize: that's a job for lint, not a job for make. Best, r

Fare, I wonder, why ironclad fails with your patch. I have checked the source, ironclad.asd creates new *ironclad-readtable* and binds cl:*readable* to it in the :around method for compile-op (like a home-made around hook) So, it looks like ironclad does not modify any global behavior. What your patch does? PS. My opinion. I would never write a library which modifies global *readtable* during load, it's of course a bad style (when ASDF around hooks provide an easy instrument for my code to use custom readtable without affecting global readtable) Should ASDF fail the libraries which modify global readtable - I doubt it's ASDF role. Should ASDF provide libraries with clean readtable when they are build - seems like a good service, so doesn't matter what I did in REPL, (ql:quickload :any-library) always works the same way. But I am not sure 100%, this is a complex issue, more intensive consideration is needed. It's rational to collect the information we already have into a ticket, and consider it more detailed when other currently active issues are closed, as Robert suggests. Best regards, - Anton

Faré <fahree@gmail.com> writes:

...

Should ASDF fail the libraries which modify global readtable - I doubt it's ASDF role.

Yes it is, see my paper: http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html#%28part._.Safety_before...

So you want to make a change rapidly to suit the deadlines of a conference, and the justification for the change is the paper you wish to submit? You can always adapt the paper to explain how the current asdf maintainers are too lazy to adapt to your insights, as you imply is the case for the current Quicklisp maintainer. Zach

Here is an example of a kind of system that will be broken by this proposed change: * We have a distributed system that is lisp-based. * There is a small set of ASDF systems that constitutes the common base of the modules of this distributed system. * For each module, there is an ASDF system, which incorporates the common base, the module proper, and possibly some additional systems. * The common base has a readtable (and it may have a PPRINT-DISPATCH-TABLE). * The distributed modules systems assume, safely, that the readtable will be set by the common base. * Code in the common base uses readtable-setting functions with the default values for the readtable. As I understand it, the proposed change will break this kind of system. Indeed, I think it's likely to break the parent common base system as well as breaking the system composition (the use of the readtable is not confined to a single file in the common base, and I believe it straddles multiple subsystems, which are represented as individual ASDF systems). I'm not going to defend the structure of this code, but I strongly feel that it is not the job of ASDF to impose a personal opinion that this coding pattern is bad, and should be *forced to break* when previously it worked perfectly adequately. To the best of my knowledge, the readtable manipulation functions in CL don't provide any easy replacement for the pattern of coding applied here. Yes, one *can* set a global to the special readtable, and then one can write a special PERFORM method (one? or many?) that will bind *READTABLE* to the value of this global. This is already a monumental PITA, since the PERFORM method will have to refer to a variable named in a package that doesn't exist at the time the ASDF system is defined. It also forces our poor programmer to know more than s/he ever wanted to know about ASDF, and IMO inappropriately bleeds details of the code of the system into the system's build file. I believe that the proposed hygiene practice *also* leaves the programmer in an uncomfortable position if s/he incrementally compiles functions inside an editor buffer, since the readtable will not be properly set. I.e., AFAICT, although careful binding of the readtable, rather than treating it as a global variable, feels like the Right Thing, it has undesirable consequences because we don't have good ways to bind the readtable around the editor. If a -*- READTABLE: foo; -*- file header worked, this would be less of a concern, but using a header like that requires named readtables. My conclusion is that enforcing some sort of compartmentalization of readtable is not only an inappropriate imposition of a particular style, but *also* means we are hanging the poor programmer out to dry. CL simply doesn't provide good tools for manipulating the readtable in the way we are proposing to enforce. IMO, named readtables are the way to go, but they are not a common feature of the language, and the portable library is bit-rotted.* I will not be pushing this proposed modification into the next release, and it will take a lot of convincing for me ever to incorporate it into ASDF. Best, r *This is partly my fault, but the current disarray of cl.net, the library's position as part of an abortive EDITOR-HINTS master system, and the previous maintainer's putting it into a revision control system (darcs) that I don't use and don't understand are additional contributing factors.

The clean thing to do would be to use named-readtables and/or cl-syntax, and have each file evaluate (in-readtable :foo) or have a perform :around method or around-compile hook that does it for you.

[...]

Without ASDF, there's no way the libraries will be safe by default.

well, there is, but it's not trivial, so most will not take the extra effort required: FTR, from hu.dwim.common we ended up shadowing cl:in-package in all our projects and added an internal mapping from package to readtable-setup-function (using a custom package definer that knows about readtables). our hu.dwim.common:in-package also sets the readtable and expects that our hu.dwim.asdf systems rebinds a copy of *readtabe* around compiling and loading. in a hu.dwim.common+swank integration system (or in one of those...) Slime is also notified about the readtables, so C-c C-c works as expected. i'd be happy if all this web of complexity could move out of our repos and into the environment, where it simultaneously becomes much simpler also. in short, i agree with Fare on this.

But polluting arbitrary systems loaded after you is definitely a bad idea:

yes, projects sideffecting *readtable* was annoying me quite a lot, and i've sent around my set of patches to a few projects that did that, including none other than iterate itself.

based on a general-purpose "each system has a set of private variables" protocol.

i don't have a strong opinion on this, but after some consideration we went with another protocol that is based on functions that modify a readtable that is given to them, which is much more flexible. not sure/don't remember whether this extra flexibility was actually needed, though. maybe it was needed for some combination of multiple syntaxes set up at the same time... -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “The Atheist does not say "there is no god", but he says "I do not know what you mean by god; I am without the idea of god; the word god is to me a sound conveying no clear or distinct affirmation. I do not deny god, because I cannot deny that of which I have no conception and the conception of which by its affirmer is so imperfect that he is unable to define it to me.” — Charles Bradlaugh (1833–1891), 'National Review' (1883–11–25)

PROPOSED NEXT STEPS: 1. A clear proposal for this modification be made. Right now the details of the proposed modification are wrapped in a fairly opaque discussion. The discussion is framed in terms that are either too vague "Make the CL syntax predictable" or too specific -- very particular proposals about how variables will be bound. By now the discussion is also "protected by volume" against people who want to join the discussion, but weren't parts of it from the beginning. 2. The proposed change should be modified to operate only in a "strict mode", allowing existing legal CL code to continue to work. I propose that individual systems be allowed to proclaim that they should be operated on in strict mode, or cannot be used in strict mode. "strict mode" would be available as a local declaration: a system could specify that it should be compiled and loaded with readtable hygiene, and that would *always* happen when it is operated on. The library user would also be allowed to specify a global strict mode setting, and this setting would be applied in some way like the following (this is a tentative proposal): a. systems that do not specify strict or non-strict mode would be compiled in strict mode if the user so specifies. This would lead to errors in some cases, but they would be errors that the user wants to see. b. If the caller specifies strict mode, it will be an error to depend on systems that want to be compiled in non-strict mode. This would achieve the safety that you wish to see. 3. Getting this right will require much testing and experimentation. It will not appear in the next ASDF release. Best, r

Faré wrote:

Unhappily, strict mode is a global flag: the question is "which readtable is this system going to be read with?". The only reasonable answer is: the readtable it was meant to be read with, which the author knows, and should be the standard readtable by default, unless explicitly overridden by the author. The backward-compatible (if it's not backward, it's not compatible) is "whichever readtable was active at the time", with sometimes comical consequences, especially when the user was using a non-standard one at the REPL.

I don't see that. If you know you aren't going to want to bleed readtable entries out of your library, and you don't want stuff creeping in, it seems to me eminently possible to mark your system as strict-mode wrt the readtable. Why is that impossible?

...

Agreed, and until the dust settles, "strict mode" cannot be the default. I'd argue it should become the default eventually (i.e. next year).

I doubt that's even feasible, since the strict mode will have to cross systems. It will be easier to have systems say "I know I behave according to the strict policy, so I will LOCALLY ask to be built and loaded strictly."

The problem is that strictness is a GLOBAL thing. Unless every single library is strict, then any readtable pollution will put the whole build at risk, and we're back to the tight readtable restrictions (i.e. no changing a standard character, no two libraries changing the same character). If I want my system to go beyond these restrictions, I have to ensure that all other libraries are protected by default. It's like changing the default encoding from :default to :utf-8: if I change the implementation's current *external-format* to :latin1, and all those utf-8 libraries are not marked :encoding :utf-8 and the default is still :default, then there's going to be mojibake. Worse: if I need to change the current encoding to EBCDIC because that's what my program does, then all those libraries are going to not even compile correctly. The only sane thing is if files get compiled in the encoding they are intended to be compiled with, rather than "whatever the user happens to be using right now". So yes, we could *demand* that every single system should specify its readtable, but that'd be *more* disruptive than providing a sane default, and *more* error-prone if it goes unenforced.

I think one reason you think that this change is feasible and I don't is that you have a stronger sense that there's a kind of "one library/system == one ASDF file."

No. I have a sense that it's trivial for those who depend on the current behavior to fix their systems, just like it was trivial for people using non-utf-8 encodings to specify :encoding :latin1 or :encoding :shift-jis, etc. I understand that it's breaking compatibility, which is a big migration matter even for a small change; but that's for the sake of making the build much more robust and of enabling things that were not possible before.

In our bigger multi-component systems that are based on ASDF, I find that we typically end up with multiple ASDF systems, because we have subsystems that use different libraries, and want to specify their own dependencies.

AFAIK you can't do this without introducing new top level systems, since modules cannot depend on libraries.

A few simple solutions: 1- tight knit systems can go in the same .asd file using secondary/system syntax, and they'll all inherit the same *readtable* from the .asd file. 2- otherwise, use named-readtables or cl-syntax and have each system change the syntax in its first file. That's the sane thing to do, and it's a two line change per user system, and a cleanup in whichever system defines the new readtable. I guess we could even automatically detect the offenders: 1- check that the *readtable* is one of the initial readtable or the standard one, at the end of the build. If not, then the system has leaked readtable state. 2- (harder) check that the initial readtable hasn't changed. That's easier if we always use the standard read-only readtable initially, but that is already a form of strictness that might break a lot of clients — we shouldn't enable that by default until after we get a clean cl-test-grid and give users a lot of notice.

That's why, for example, we have systems that bleed readtable across their boundaries: the systems really aren't stand alone entities, but limitations on ASDF expressiveness, and the desire to get more modularity than the (necessarily in-line) :MODULE will permit, causes a proliferation of systems that are NOT libraries, and have meaning only in context.

I'm not going to move ASDF towards breaking such systems.

These systems can be trivially fixed with a two line change, and this will enable a world where every system, library or not, can freely pick their favorite readtable and not be afraid of breaking other things.

...

Unhappily, strict mode is a global flag: the question is "which readtable is this system going to be read with?". The only reasonable answer is: the readtable it was meant to be read with, which the author knows, and should be the standard readtable by default, unless explicitly overridden by the author. The backward-compatible (if it's not backward, it's not compatible) is "whichever readtable was active at the time", with sometimes comical consequences, especially when the user was using a non-standard one at the REPL.

I don't see that.

If you know you aren't going to want to bleed readtable entries out of your library, and you don't want stuff creeping in, it seems to me eminently possible to mark your system as strict-mode wrt the readtable.

Why is that impossible?

Nobody EVER wants "stuff creeping in", just like a program written in utf-8 NEVER, EVER wants to be compiled in EBCDIC mode, or even latin1 for the matter. Whatever readtable you designed your code to be read with, by definition, you don't want your code to be read with another one. At no point, does anyone ever want the current modified REPL readtable to leak into the code he is compiling from the REPL. If I use EBCDIC, I still want my libraries to compile using whatever encoding they were designed to be read with. And I do want to be able to use EBCDIC or a new readtable. There are 700 libraries in Quicklisp, and not a single of them "wants stuff creeping in"; yet if safety is an "opt in" feature, then every single one you use needs to opt in safety so you may safely change the readtable at the REPL and call anything that might load a system. The entire readtable feature is crippled if the coding conventions preclude any significant modification and make any concurrent use of the same character an error. I think the point you don't see is that the readtable *of the REPL* is going to affect every single system being compiled, and there is no way to opt out. OK, I have two (separable or combinable) proposals that might provide both enough enough hygiene to allow for radical readtable modification while allowing for traditional unhygienic style of development. My main constraint is as follows: 0- what readtable a system is compiled with must not depend on anything but its declared dependencies, and in particular MUST NOT EVER be affected by whichever readtable the user is currently using at the REPL. enforcing the constraint that the readtable used to compile a system only depends on its declared dependencies. A compromise situation might be as follows: 1- ASDF maintains an asdf:*global-readtable*, which is the *readtable* object at the time it was loaded. 2- This *global-readtable* is subject to the current restrictions: A- no modifying any standard character, B- no two dependencies assigning different meaning to the same non-standard character. C- libraries need to document any change to the readtable D- free software libraries will register these changes on the page on cliki. 3- Unhappily, there is no cheap way to enforce these restrictions, but that's no regression with respect to the current situation. 4- ASDF wraps any compile-op and load-source-op in this asdf:*global-readtable*, but probably not load-op, to preserve combine-fasl linking semantics. 5- Systems that want to do crazier things with the readtable that may violate (2) must arrange to use their own private readtable, but can otherwise do it safely. It is an error (unhappily not enforceable) to modify the current readtable in these ways. 6A- ASDF binds *read-table* to the *global-readtable* at the start of every system's compilation (and loading?), and around the entire asdf:operate, leaving the *readtable* unchanged at the end. This easily supports systems that "modify the current readtable data structure". However, that doesn't systems that "bind *readtable* to a new value", because the changes they make will shadow the changes that other systems following this style make and depend on. To allow such, an idiom, we must also do the following: 6B- ASDF binds *read-table* to a proper "entry readtable" at the start of every system's compilation, and record an "exit readtable" at the end of the system's loading. 7- maintain a partial order on these readtable objects, assuming that each system's exit readtable supersedes the entry readtable. The least readtable is the *global-readtable*. It's enough to store for each new exit readtable, identified by the name of system that created it, the set of its inferior readtables, as a list or eq-hash-table, or an equal hash-table, with each readtable being identified by the name of the system that created it. 8- before a system is compiled or loaded, compute the maximum readtable of all the exit readtables of its dependencies. If this maximum is unique, then it will be the entry readtable of the system. If there is not a unique readtable that is more than all the other ones, that's an error, and we refuse to load the system. 9- after a system is loaded, check its exit readtable, if it already exists, check that this doesn't create a cycle or issue an error; it it doesn't already exist, add it to the set of all known exit readtables. 10- ASDF either A- binds the *read-table* to the *global-readtable* around the entire asdf:operate, leaving the *readtable* unchanged at the end, or B- always side-effects the *read-table* to correspond to the exit readtable of the loaded system, or C- operate does the binding around thing, but load-system does the side-effect after it's done operate'ing. Does that strike you as complex? Because it is. That's the price of *safely* supporting this "systems can bind a new value to *readtable*" style. Unhappily some of the constraints are not enforceable (2A and 2B), but that's the very same as now. So my next question is: do you want to safely support these conventions? Do your systems modify the current *readtable* structure, or do they bind *readtable* to a new value? PS: thank you for making me come up with better solutions. I care about enough hygiene to use readtables safely, and I also care about supporting legacy systems if possible. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org An insult may sometimes adequately fit the person who is insulted. However, it can only ever possibly tarnish but the person who insults.

As a personal request, let's table this discussion for now. This issue is too complex, and the related proposals have too many ramifications to push it into the impending release. I appreciate your enthusiasm for the discussion, and I apologize for trying to shut you down. Unfortunately, right now I don't have enough cycles to devote to the issue, and to getting the next release made. I had an enormous work deadline late last night, and another tomorrow. Every minute I spend on this discussion comes out of a very small pool of minutes to address the remaining issues that *must* be resolved before the new release. So no matter what happens, we'll have to spin this conversation back up when I am able to attend to it again. Again, I apologize; I simply must impose some priorities or I won't provide the services that ASDF requires. Thanks, R

16.03.2014, 19:20, "Robert P. Goldman" <rpgoldman@sift.info> wrote about readtable hygiene:

2.  The proposed change should be modified to operate only in a "strict mode", allowing existing legal CL code to continue to work. I propose that individual systems be allowed to proclaim that they should be operated on in strict mode

The same approach may be used for deferred warnings check. And for other improvements. If ASDF develops better mode of operation, which is incompatible with old mode, no need to break old systems. I've been thinking about new form: (asdf3:defystem ...) or (asdf:defystem3 ...). Systems defined like that benefit from new features. Old asdf:defsystem remains working as before (and may even be deprecated). Another alternative is a slot in system definition, like (asdf:defystem :mode :safe ; <<- :version "0.1.0" :serial t ...) The disadvantages of a :mode :safe is that it's more typing, and people can overlook/forget about it. If new mode is an improved one, it should be easily accessible. 29.03.2014, 02:39, "Daniel Herring" <dherring@tentpost.com>:

Would it be possible to have a special header in asdf files indicate that the contained system(s) should be loaded using the new syntax mechanism?

[...]

;; asdf:config '(asdf:+clean-readtable+)

No, it's inconvenient, why use special parser if we can just directly specify what we want in a lisp expression. Details may wary. For example, Robert says:

.. individual systems be allowed to proclaim that they should be operated on in strict mode, or cannot be used in strict mode.

To me the proclamation "cannot be used in strict mode" looks redundant. The main important idea - new features may be provided separately, without breaking old code, spending efforts on fixing old libraries, chasing their authors and asking for their attention to commit the changes; and other disturbances in CL ecosystem. It's a simpler and cheaper way. Best regards, - Anton

30.03.2014, 05:29, "Daniel Herring" <dherring@tentpost.com>:

On Sat, 29 Mar 2014, Anton Vodonosov wrote:

...

 29.03.2014, 02:39, "Daniel Herring" <dherring@tentpost.com>:

...

 Would it be possible to have a special header in asdf files indicate that  the contained system(s) should be loaded using the new syntax mechanism?

 [...]

 ;; asdf:config '(asdf:+clean-readtable+)  No, it's inconvenient, why use special parser if we can just directly specify  what we want in a lisp expression.

I haven't checked recently, but in the past ASDF files have had significant code even before the system definition.  It was common to find conditionals based on *features*, comments for users to select from a few customization options, numbers being read, etc.

Once you have read the form, it is too late to modify the environment that is used to read it.

We discuss not the syntax for reading .asd file, but syntax for reading .lisp files constituting the system. .asd files are always read inside of with-standard-io-syntax. Best regards, - Anton

Dear Anton, can you do a cl-test-grid test with the code in the standard-syntax branch from git://common-lisp.net/projects/asdf/asdf.git (commit 2c1107) ? That branch has my "each system gets its own syntax" thing, based on a general-purpose "each system has a set of private variables" protocol. This protocol allows to designate variables using a string that will be parsed later, but doesn't somehow try to canonicalize names — if you use two different names for the same variable, you lose. By adding :after methods to compute-system-variables, more variables can be configured with configure-system-variable and configure-system-variables — this makes it a good basis for some future out-of-band configuration of ASDF, though such mechanism would need to be part of ASDF itself, or loaded early via an ASDF init file of some sort to be effective. Some DSL could then specify variables to include in a system's private variables, with optional initial value, depending on the name and version of the system and on various features, with usual rules (or per-system rules?) to inherit or ignore other configuration settings. Such a DSL is *not* included in this branch. This protocol itself uses a new call-with-action-context protocol that perform-with-restart calls around perform. This is mostly backward compatible, except that existing clients who call perform directly (rather than perform-with-restarts) might miss those around methods — to get that effect we'd need a new method combination. swank-asdf would notably need to call perform-with-restarts instead of perform, and if possible to enable the try-compile-file-with-asdf hook function by default. I tested it to correctly process ironclad, at least. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Gods don't kill people. People with Gods kill people. — David Viaene

On Tue, Mar 25, 2014 at 12:16 AM, Anton Vodonosov <avodonosov@yandex.ru> wrote:

Hi Fare, sorry for the long reply.

Will do it, but was a bit overloaded recently by work and personal tasks, + was running tests for quicklisp 2014-03-17 and ABCL 1.3.0

Is you request still actual - standard-syntax branch (commit 2c1107)?

Dear Anton, I've since started hacking again and am now in the middle of various hacks, sorry about that. I'll tell you when the branch is ready. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Free Will: decisions are determined by processes located in individual brains and unpredictable by either same or other individuals. God-given souls, pixie dust, (quantum) non-determinism, mystic animal spirits and other magic sources of "free will" need not apply.

Dear Anton, the new thing to test is the HEAD of branch syntax-control. Please test with SBCL first. This branch contains my experimental syntax control, binding a *shared-readtable* around compilation (itself the *readtable* at the time ASDF is loaded), so that promiscuous systems may keep modifying a shared readtable under the usual conventional constraints (now documented: all standard characters reserved, no conflict allowed or bust), while mutual protection is implemented for these systems and those that might wish to go beyond these constraints, particularly so at the REPL. If this has no regressions, then I'll try to convince Robert to merge that in master and make it the new release candidate — assuming Robert's big promiscuous project also compiles with it (if it doesn't, I explained my more elaborate plan in an earlier email). If this has regressions, and/or if Robert is not convinced, then what's in the master branch is our release candidate. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Luck occurs when preparedness meets opportunity.

Thanks a lot, Anton! The results are very encouraging: only 4 red things, that look fixable. cl-quakeinfo is still doing plenty of ugly old-style things, package promiscuity with cl-user, etc., and needs maintenance (come on, disable-output-translations in the .asd file itself? Ugh!) I was so pissed, I ended up fixing the code myself: https://github.com/kraison/cl-geocode/pull/1 https://github.com/e40/cl-quakeinfo/pull/1 The code works for me after that, on CCL, SBCL and Allegro. Let's hope these changes get propagated to Quicklisp, and ignore these failures for now. cl-protobufs-tests was doing something wrong in its .asd file — fixed upstream. But this indeed reveals how my syntax-control branch makes ASDF slightly stricter in how it handle symbols as class designators and tolerates less package craziness. That's a good thing, if this is the only casualty. femlisp raises an interesting issue: it has (setq *READ-DEFAULT-FLOAT-FORMAT* 'double-float) in setup.lisp, which is cancelled by the with-standard-io-syntax that I introduce in my syntax-control branch. I just pushed a change in said branch to make the 'double-float the default in with-asdf-syntax, which I believe is a more useful convention than what the standard provides. Which means I probably should be setting that default globally in asdf.lisp somehow. But more generally, must we propagate all the syntax variables from file to file in the same system? In different systems? This is once again a case where me must choose between hygiene and backward-compatible support for promiscuity. As for graph-matrix, it just depends on femlisp. Can you re-run a full test with what I just pushed in branch syntax-control? —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org You may call yourself athiest, but I'm athier than you are! On Wed, Mar 26, 2014 at 2:52 PM, Anton Vodonosov <avodonosov@yandex.ru> wrote:

Hi Fare,

I didn't really understand how exactly the changes work, but here is the result: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-39.html

27.03.2014, 18:44, "Robert P. Goldman" <rpgoldman@sift.info>:

While I am very grateful to cl-test-grid, we need to be conscious of its limitations.

If anything, I personally do not insist on any changes to be done in ASDF. But if developers want to test whether the libraries built OK on my system previously remain OK after some change, I don't mind to test. Robert asked to postpone the discussion, so I didn't speak. But Robert, how about we continue the discusstion, and you can ignore it? When you finish current tasks and are ready to consider this you can ask for summary and then decide, Maybe right solution will evolve by that time, so you won't need to spend your energy on analysis. I think maybe some useful services are possible in regard to isolation from readtable and other global state modification. But not sure what exactly. BTW, I haven't experienced problems like that, so it's not a critical feature for me. If I am a library author I write code from assumption that standard CL syntax is used when the library is built. If my user changes readtable and breaks the library - it's his problem, he created this problem and he is able to fix it. If my dependency breaks readtable for me, it is an inconvenient dependency, I will try to find a better one. If my dependency breaks readtable for my user - that's unfortunate. The user need somehow to restore standard environment after loading me. (Or consider me a bad dependency and try to find another one). The user may somehow instruct ASDF: "when the application is built, use standard CL syntax and isolate the application from any readtable, etc, modifications done in REPL or by the app dependencies." Another possible request: "Isolate the application, and every app dependency from each other" - this allows the application to use dependencies otherwise interfering via readtable. As a library author I would try to not alter global readtable from my library, but provide user with a function to enable my reader macros. If ASDF wants to prevent the undesirable interferences, maybe it's enough/most important to develop and recommend an easy way to keep your readtable modifications scoped to your system? What would be the recommended way to do so? ASDF around-compile hooks? In CL we don't want to totally isolate systems from each other's global modifications - each system creates some packages, defines functions used by other systems. So the systems interact via global modifications... Another question is when global modifications are done because there is no easy way to keep them local. It's a complex area. I think it will help if we step back and formulate clearly what problem we are solving. Best regards, - Anton

Zach Beane wrote:

Faré <fahree@gmail.com> writes:

...

femlisp raises an interesting issue: it has (setq *READ-DEFAULT-FLOAT-FORMAT* 'double-float) in setup.lisp, which is cancelled by the with-standard-io-syntax that I introduce in my syntax-control branch. I just pushed a change in said branch to make the 'double-float the default in with-asdf-syntax, which I believe is a more useful convention than what the standard provides.

I don't like the idea of ASDF setting a different *READ-DEFAULT-FLOAT-FORMAT* value during ASDF operations than it will have in the REPL, during plain loading, or compile+loading individual files outside of ASDF.

Zach

I second this dispreference. As an old-school lisper, used to working extensively on incremental development in the editor buffer, and even in the REPL, this seems bad to me. The errors it will lead to will be odd corner cases, bewildering to ordinary programmers not versed in the fussy details of ASDF implementation. Put yourself in such a programmer's shoes, when his/her code that worked fine in the repl breaks in ASDF, or vice versa. What's the debugging process? Maybe the default is undesirable, but it's not our job to patch all the undesirable features of Common Lisp. The more divergences between CL-in-ASDF and CL that are introduced in this branch, the more likely it is that it will have to live on as a permanent fork and not be merged. Best, r

Faré <fahree@gmail.com> writes:

On Thu, Mar 27, 2014 at 12:30 PM, Robert Goldman <rpgoldman@sift.net> wrote:

...

Zach Beane wrote:

...

Faré <fahree@gmail.com> writes:

...

femlisp raises an interesting issue: it has (setq *READ-DEFAULT-FLOAT-FORMAT* 'double-float) in setup.lisp, which is cancelled by the with-standard-io-syntax that I introduce in my syntax-control branch. I just pushed a change in said branch to make the 'double-float the default in with-asdf-syntax, which I believe is a more useful convention than what the standard provides.

I don't like the idea of ASDF setting a different *READ-DEFAULT-FLOAT-FORMAT* value during ASDF operations than it will have in the REPL, during plain loading, or compile+loading individual files outside of ASDF.

Zach

I second this dispreference. As an old-school lisper, used to working extensively on incremental development in the editor buffer, and even in the REPL, this seems bad to me. The errors it will lead to will be odd corner cases, bewildering to ordinary programmers not versed in the fussy details of ASDF implementation. Put yourself in such a programmer's shoes, when his/her code that worked fine in the repl breaks in ASDF, or vice versa. What's the debugging process?

Maybe the default is undesirable, but it's not our job to patch all the undesirable features of Common Lisp.

The more divergences between CL-in-ASDF and CL that are introduced in this branch, the more likely it is that it will have to live on as a permanent fork and not be merged.

I also have this preference, but I slightly disagree on how to resolve it.

femlisp side-effects *READ-DEFAULT-FLOAT-FORMAT*, which means that every system compiled after it will be treated differently than if it were compiled before it — and once again, whoever writes the system does not and cannot control the order in which things are compiled, which ultimately depends on what toplevel system the user operates on and what systems have or haven't already been compiled (and are up to date) before he operates on that toplevel system. This means that the semantics of your system are already dependent not just on the REPL but also other uncontrolled and uncontrollable state. This is not desirable.

I do have control: If femlisp or any other library makes a boneheaded decision that breaks my software, I can stop using it. That is less of an option with ASDF. Zach

I do have control: If femlisp or any other library makes a boneheaded decision that breaks my software, I can stop using it.

yes, resolving that is trivial -- once you have identified the problem. regarding the recent discussions i'm generally baffled why it is at all a question whether to make a build software deterministic or not. in my view if there's anything in the global state that has an effect on the building of a software, anything, then it's a bug. and if this state can be set in a way that it leaks out and influences builds later in time, then it's an ugly bug feasting on programmer nerves and time. the only question to me is what should be the defaults, and how can the transition be made smoother with deprecation warnings and whatnot. and if there are some unmaintained, bitrotten libraries and some software depending on them, then they have all the freedom in the world not to upgrade ASDF, or to put on the maintainer hat and patch the bitrotten libs. by now the time on spent discussing this would have easily been enough to fix all of them twice over, and to add a section to the top of the manual, with bold, that lists the global state that ASDF guarantees and isolates. -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “Those who love peace must learn to organize as effectively as those who love war.” ― Martin Luther King, Jr. (1929–1968, assassinated)

On 28 Mar 2014, at 19:00, Robert Goldman <rpgoldman@sift.net> wrote:

I would be much happier to see this conversation direct itself towards a design for providing strict mode, rather than arguing about whether or not "strict mode" should be universally imposed. I will say it again: that will not happen.

Robert, I really like your idea of a strict mode of ASDF that isolates a single system from global state changes. While reading the thread, a funny analogy found its way into my head: To some degree, the isolation feature is similar to a network firewall. Strict mode could work like a firewall config, where one would specify a rule for allowed state changes (both inbound and outbound) in the system definition. This is just a rough draft of what it could look like for the ASDF user: (defsystem "my-app-system" :isolation ( ;; this system modifies the readtable (:export :readtable) ;; import changes made by my-base-system (:import :all "my-base-system") ;; use readtable changes made by dependencies (:import :readtable :dependencies) ;; use global declarations (:import :declarations) ;; catch-all rule: isolate against any other changes (:deny-export :all) (:deny-import :all) ) :components ...) For backward compatibility, the default rules would allow any state to be exported or imported. In order to make the build deterministic, rule violations should throw a condition (failing the build). If the isolation option can be applied to any component, it would even be possible to isolate individual source files/directories in a system against each other (although it would be kind of ridiculous if anyone ever did this in practice). It might also be feasible to allow external overrides/extensions for the rules defined by the systems. This way, the person responsible for build issues at e.g. ITA could maintain total control of the side effect propagation without modifying the system definitions themselves. I believe that given some development effort, a lot of things can be made controllable by this rule model. Things that come to mind right now are printer variables, reader variables, declarations and maybe even package locks, but I'm sure there's even more to it. I agree with Faré about the fact that it's OK if this feature works only when using ASDF to compile systems. Anyone who wants to use this during interactive development with SLIME will need to use SLIME's ASDF integration. -f

I think ASDF should not change the semantics and break existing system. We can provide new, improved functionality without breaking anything. And keep old systems working forever with new ASDF. Lets fist consider and decide, how exactly the best support for syntax customizations should work. Withoug compromises (like allow shared readtable modifications, but resetting *read...* variables - that's strange compromise). Lets ignore backward compatibility for now. Best regards, - Anton

29.03.2014, 19:09, "Attila Lendvai" <attila.lendvai@gmail.com>:

...

 We can provide new, improved functionality without breaking anything.  And keep old systems working forever with new ASDF.

"forever" is a very strong word Anton. e.g. if CL doesn't fade into the category of history in 10 years, meaning that a significant number of people will still use it to develop new software, then i'll be both surprised and disappointed.

I know people who work with PL/I till now; and even more people who use C++. I don't worry about CL in 10 years. But that's all off-topic for asdf-devel

...

and if this state can be set in a way that it leaks out and influences builds later in time, then it's an ugly bug feasting on programmer nerves and time.

This would be true only if "a software" was equivalent to "an ASDF system." That is not the case. I have seen many applications that are made up of a large number of ASDF systems, because ASDF does not necessarily provide nice structures for large applications. In such

we also have libs with 10+ asdf systems.

application builds, you WANT state to bleed from one ASDF system to another, because these ASDF systems are not freestanding entities.

one can write a trivial loader file that calls ASDF with some extra keyword parameters to load such systems.

Complaining that state leaks in and out of these systems is like complaining that state leaks into a closure. It's intended to leak, and that is not a bug.

the vast majority of the asdf systems don't want leakage, and defaults should be covering the majority use-case (if/when it's possible without setting up landmines). reproducible builds are a valuable feature. if i wanted to cater for the use-case you described then i'd still make full isolation the default with features to let systems explicitly turn off specific isolations, or just simply let them on their own to subclass and customize ASDF the way we currently have to subclass and customize asdf to avoid the leakage from the hu.dwim.* universe (we use e.g. readtable customizations). and whoever is not ready to invest time in following the changes, should simply not upgrade, and then it's a non-issue. but this starts to feel like preaching and i don't really have that much stake in this anymore because we have already invested in covering our bases... so, these are just some 0.02... -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “The three most harmful addictions are heroin, carbohydrates, and a monthly salary.” — Nassim Taleb (1960–)

29.03.2014, 02:32, "Faré" <fahree@gmail.com>:

Additionally, my ASDF syntax-control branch has a mechanism that allows systems to specify variables they want to bind around their system, so you could specify: (defsystem foo :variables ((*read-default-float-format* . (constantly double-float))) ...) where the assoc'ed value is a specification as per ensure-function, which lets you specify a string "(lambda () (not-yet-created-package:foo 1 2 3))"

Looks like extension of the around-compile hooks features. Is the :variables specification only works for compile and load-source operations, the same way as around-compile works?

...

regarding the recent discussions i'm generally baffled why it is at all a question whether to make a build software deterministic or not. in my view if there's anything in the global state that has an effect on the building of a software, anything, then it's a bug.

I think one question is whether it's worth taking a path to this goal that breaks programs that currently work. I don't think all such programs are bitrotted junk that can be fixed up in a day.

Nobody is proposing to break anything. On the contrary, the proposal is to make everything build in a more reliable way. Once again: 1- there will be no hacking into other people's machine to upgrade half their software. 2- there will be no changing the ASDF default until patches have been sent to all maintainers of libraries currently in Quicklisp and ample heads up was given to everyone who is not in Quicklisp. For the change in default character encoding, this took one year. I imagine a change in syntax control settings will take the same: the code will be there but disabled by default for a year, then the default will change. 3- code that was previously guaranteed to work portably will still work portably. 4- most code, that was depending on unenforceable but standard environmental dependencies, will now work reliably even when compiled in an environment when other systems and/or the user have bound variables incompatibly. 5- some code, that was depending on unenforceable and non-standard environmental dependencies, will now fail reliably even when compiled in an environment when other systems and/or the user have bound variables compatibly -- and will need to be fixed. Then users can change their *READ-BASE* to 36, their *READ-DEFAULT-FLOAT-FORMAT* to 'SHORT-FLOAT, their *READ-EVAL* to NIL or their *READTABLE* to something that implements a Ruby compiler, and still be able to call asdf:load-system under the hood and have everything work fine. Actually, an ASDF server in the background might be farming compilation of each and every file from a clean image with only its declared dependencies based on a clean environment, each on a different worker in a distributed network - and the semantics will be exactly the same. Frankly, the notion that syntax bindings from the REPL might leak into an unrelated system's compilation and break it sounds completely braindead to me, and I'm surprised anyone tries to defend the concept. The question to me is not should the change happen, but how. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org War does not determine who is right — only who is left. — Bertrand Russell

Attila Lendvai <attila.lendvai@gmail.com> writes:

...

...

regarding the recent discussions i'm generally baffled why it is at

...

...

all a question whether to make a build software deterministic or not. in my view if there's anything in the global state that has an effect on the building of a software, anything, then it's a bug.

I think one question is whether it's worth taking a path to this goal that breaks programs that currently work. I don't think all such programs are bitrotted junk that can be fixed up in a day.

those programs that currently work will keep on working if ASDF is not upgraded.

no upgrade, no breakage.

to me it's a very strange argument that the right way to avoid breaking unmaintained old code is by hindering change that makes new things better in the future.

Indeed, it would be strange to discuss such things. I'm not talking about unmaintained old code, I'm talking about code that works by design and as designed within the features and behaviors of the current system that will no longer work. I am receptive to "Some things will break, but change is necessary and this approach minimizes breakage" but less receptive to "Nothing will break" and "Things that break deserve it" and "People who do not want any breakage are braindead losers" and "I will personally fix anything that breaks." Zach

: p-cos

...

no upgrade, no breakage.

If you can’t upgrade ASDF, you may also not be able to upgrade quicklisp. If you can’t upgrade quicklisp, you may also not be able to get updates to existing libraries. At least, it may be harder than necessary (like in the pre-quicklisp, pre-asdf-install days), and you can’t benefit from the efforts that were put into ASDF and quicklisp to the same extent as others anymore. [1]

Sorry, you don't get to demand change and rant against change in the same paragraph. You don't get to praise the efforts put into ASDF and rail against them in the same paragraph. The same legitimate reasons that justify desirable change in your libraries also justify desirable change in ASDF. The same things that make these changes desirable are the things that break systems that relied on non-portable kludges and quirks. You only get to discuss whether a particular change is desirable.

Don’t try to abuse the popularity of ASDF and quicklisp to enforce changes to Common Lisp, the Language that may not be accepted by the community.

Don't accuse other people of things they are expressly not trying to do. If I were trying to "enforce changes" and caring about what a notional "community" accepted, I would not be discussing anything, only committing changes to the repository.

If the proposed change to enforce double-float as a default *read-default-float-format* was unambiguously an improvement, there would be no problem. But it’s not an improvement, not by a long shot. [...]

There are two dependent changes being discussed: (1) guaranteeing a value of *read-default-float-format* and other syntax variables when compiling a library. (2) having this value be 'double-float. It looks like (2) will definitely not happen. I still argue that (1) is essential for build determinism, and enabling users to change syntax at the REPL.

*read-default-float-format* lets you choose between different representations for such generic libraries. The scoping rules for dynamic variables in Common Lisp give you two choices: Either you tell a particular library that it chooses a particular representation for its generically chosen representations - you do this by dynamic binding. Or you tell a set of libraries (all that you are loading from now on) that they should all choose the same representation for their generically chosen representations - you do this by side effect.

That's not how it works, unless you include a bit for *rdff* in the name of the fasl cache directory — and since the planning is done based on pathnames before the compilation happens, that should still be *rdff* at the beginning of compilation. Otherwise, the build is not deterministic, and two different toplevel programs will poison each other's builds.

I am very happy that Common Lisp gives a lot of choices in this (and many other [!]) regards. These options shouldn’t be restricted due to the lack of imagination of some developers who believe that limiting choices is always better.

Or by the incompetence of people who give lessons about build systems when they don't understand a thing on the topic.

There are already (far too) many languages out there that go down that route. If you want Racket, you know where you can get it. Common Lisp is a language for people who know what they are doing, not for people who need to be held by their hands.

I don't want Racket. I want a deterministic, modular build system for CL. I want safe to be the default.

You also have the option to fork a particular Common Lisp implementation So do you. That's a non-argument that assumes you're the dictator of CL.

The fact that the proposed changes are discussed so actively is a clear sign that people care about these things.

And the one way to know whether they'll be discussed is to open the discussion indeed.

[1] I already encountered libraries that don’t load with current ASDF versions anymore, although they are defined as ASDF libraries, which is a shame!

Sorry, you don't get to complain unless (1) you issue a bug report, and (2) the bug isn't fixed by whichever party actually has a bug, whether it's the library or ASDF. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Compulsory child labor was not abolished, it was changed from productive work to counter-productive brainwashing, and made universal: public schooling.

...

(1) guaranteeing a value of *read-default-float-format* and other syntax variables when compiling a library. I still argue that (1) is essential for build determinism, and enabling users to change syntax at the REPL.

Are you also considering the following use case?

- Assume I’m developing a library for a specific Common Lisp implementation. Let’s choose LispWorks as an example, although it could be any other. (I’m just choosing LispWorks as an example because I happen to use it myself most often.) LispWorks comes with its own system definition facility, and with its own default values for pre-defined global variables in the CL package that may or may not be the same as those of other CL implementations. Since I’m also maintaining portable libraries, I prefer to use ASDF even for LispWorks-only libraries, so I don’t have to change tools on a regular basis. However, I would also like to make sure that LispWorks users can use the LispWorks system definition facility for my library as an alternative. Will it be possible to use ASDF without it messing with the default values for pre-defined global variables, so that I have a guarantee that I get the same result no matter whether I use ASDF or some other system definition facility? (I’m fine with doing extra work in the system definitions, but I’m not sure I’d be fine with adding extra headers to the lisp files.)

I admit I have only limited knowledge of LispWorks's defsystem — but that it made me appreciate Dan Barlow's design better (though not his implementation). Writing LW defsystem rules to process dependencies correctly is *possible*, but by no means trivial. I don't know what guarantees the LW defsystem does or does not provide in terms of variable bindings — if you know it, it would be great of you to explain what they are (or just link to a page that does, if there is one). Indeed, if these assumptions differ from those of ASDF today or tomorrow, it would be nice to have a simple way to declare around wrappers that provide the alternative assumptions.

...

That's not how it works, unless you include a bit for *rdff* in the name of the fasl cache directory — and since the planning is done based on pathnames before the compilation happens, that should still be *rdff* at the beginning of compilation. Otherwise, the build is not deterministic, and two different toplevel programs will poison each other's builds.

…not even if you :force t?

...

...

You also have the option to fork a particular Common Lisp implementation So do you. That's a non-argument that assumes you're the dictator of CL. Huh?!? The common "love it or leave it" slogan is assuming the utterer owns

If you make :force t the default, you lose incrementality, and fast startup time for end-user scripts. If you say "things are unsafe by default", you lose modularity and you make it impossible to distribute scripts to end users. Either way, if you don't have a deterministic build *by default*, easy deployment of scripts to end-users is not possible anymore. the domain at stake. Otherwise, why should YOU not be the one to leave when we disagree what "love it" means? —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Think you can, or think you can't — either way, you'll be right. — Henry Ford

I prefer the defaults of the CL implementation of my choice not to be overridden by some tool whose primary purpose is something unrelated to those defaults. That’s all.

i fail to see how ASDF's primary purpose is unrelated to global state that affects the outcome of a build process. -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “In order to rally people, governments need enemies. They want us to be afraid, to hate, so we will rally behind them. And if they do not have a real enemy, they will invent one in order to mobilize us.” ― Thich Nhat Hanh (1926–)

On 29 Mar 2014, at 20:07, Stelian Ionescu <sionescu@cddr.org> wrote:

On Sat, 2014-03-29 at 19:59 +0100, Pascal Costanza wrote: [...]

...

...

...

...

That's not how it works, unless you include a bit for *rdff* in the name of the fasl cache directory — and since the planning is done based on pathnames before the compilation happens, that should still be *rdff* at the beginning of compilation. Otherwise, the build is not deterministic, and two different toplevel programs will poison each other's builds.

…not even if you :force t?

If you make :force t the default, you lose incrementality, and fast startup time for end-user scripts. If you say "things are unsafe by default", you lose modularity and you make it impossible to distribute scripts to end users. Either way, if you don't have a deterministic build *by default*, easy deployment of scripts to end-users is not possible anymore.

I understand your desire for deterministic builds. I don’t understand your desire for deterministic builds being the default.

Utterances like this is what makes "academic" an insult in certain circles.

You’re quoting me out of context. If deterministic by default had no cost associated with it, it would clearly be the desirable choice. But it has a cost associated with it, so this is less clear. If I failed to convince you that this is the case, and that other people may have other preferences, then I apologize. Pascal -- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

On Thu, Mar 27, 2014 at 1:20 PM, Zach Beane <xach@xach.com> wrote:

...

femlisp side-effects *READ-DEFAULT-FLOAT-FORMAT*, which means that every system compiled after it will be treated differently than if it were compiled before it — and once again, whoever writes the system does not and cannot control the order in which things are compiled, which ultimately depends on what toplevel system the user operates on and what systems have or haven't already been compiled (and are up to date) before he operates on that toplevel system. This means that the semantics of your system are already dependent not just on the REPL but also other uncontrolled and uncontrollable state. This is not desirable.

I do have control: If femlisp or any other library makes a boneheaded decision that breaks my software, I can stop using it.

That is less of an option with ASDF.

If the semantics of your software depends 1.0 being 1.0f0 rather than 1.0d0, by expecting *read-default-float-format* to be 'single-float, you are already setting yourself for failure if you don't also make sure *read-default-float-format* is properly bound. If you control the top-level application, you can certainly counter the femlisp binding with your own. The problem is if you provide a different library and also expect *read-default-float-format* to be bound to a certain value without binding it yourself — some user is bound to suffer. And if you bind it through a side-effect, you are bound to bring sorrow to users of other libraries. With the current ASDF contract, library authors cannot depend on *read-default-float-format* yet also should not modify it globally either. This is unfortunate. That said, one option would indeed be to declare that femlisp is broken, by desiring to write 1.0 and expect it to be 1.0d0 without binding *read-default-float-format* around every file. Consider other variables, like *READ-BASE*. Aren't library authors entitled to expect it to be 10? I'd say they are, and ASDF is only making things easier by binding it to 10 by default, which only makes it nicer on those who want to make it 8, 16 or 36 locally, and can now do it safely in their own files or at the REPL. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The only saving grace of government is that they're incompetent, because if they could do what they really want, it would be horrible for all of us

On Thu, Mar 27, 2014 at 2:00 PM, Anton Vodonosov <avodonosov@yandex.ru> wrote:

Fare, if femlisp configures *read-default-float-format* why does it fail with your ASDF change? Do you restore standard CL syntax around every file compilation? Yes I do, although I just committed a change to override *read-default-float-format* to 'double-float, which I believe is more useful.

Maybe isolating syntax changes at system boundary is more convenient? It's also harder and more subtle to explain, especially since there's no way to unify changes that happen at each of the system's dependencies: if A depends on B and C, and B sets a variable to something, C the same variable to something else, which binding should prevail? Especially since the actual load order is dictated by the top-level system, not by A itself. The two sane things would then be for the system to be initialized (a) to a constant value, or (b) to the value at the exit of its last declared dependency, which might not at all be the last actually loaded dependency, since that depends on the top-level system. I'd say that (b) is way too clever, and that (a) might as well be around individual files.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If freedom makes you think of antisocial behavior, harming other people and not helping anyone — maybe the problem is with you, not freedom.

Fare> Yes I do, although I just committed a change to override Fare> *read-default-float-format* to 'double-float, which I believe is more Fare> useful.

I disagree that that is more useful. While I almost always use doubles, there are lots of reasons to use singles. Just because I like doubles doesn't mean we're right and everyone else is wrong.

You can still use singles as in 1f0 or 1.0f0. And you can still override *read-default-float-format*. I modified femlisp to do the right thing, whichever way ASDF goes: https://github.com/fare/femlisp

If you must choose a value, it should be 'single-float, because that is the CL default.

It's hardly "the" CL default. There is no CL default. CL tells of an "initial value", but the fact that it won't be modified is guaranteed not to be guaranteed by ASDF, and indeed some systems, like femlisp, bind it to 'double-float. If you expect it to be single-float always, you already lost.

I also tend to agree with Robert. asdf is a make, not a lint+make.

It's not a matter of lint at all. It's a matter of invoking the compiler in a reasonably known state, with reasonable compilation flags, reasonable warning behavior, etc. Where reasonable is under control of he who knows under what expectations he wrote the code, i.e. the system author, without risk of interference from the REPL, and where the defaults are predictable and sensible. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org It may be better to be a live jackal than a dead lion, but it is better still to be a live lion. And usually easier. — Robert Heinlein, "Time Enough For Love"

On Fri, Mar 28, 2014 at 6:19 AM, Zach Beane <xach@xach.com> wrote:

Faré <fahree@gmail.com> writes:

...

It's not a matter of lint at all.

Very true. Lint is an *optional* program, and you can easily choose to ignore it.

The analogy with lint is totally bogus here. The case that was vaguely analogous to lint indeed was the debate last year about turning deferred-warnings on, which could have been since as analogous to calling lint, though really, it was more like having GCC enable -Wimplicit-function-declaration by default, except in a context where -Werror is on by default, and that's where things broke and why I backed that change in the end.. If you want a C analogy, the situation is not at all analogous to make calling lint for you. The situation is analogous to make calling GCC with flags decided by the author, vs calling gcc with flags that vary depending on what the user uses for completely different purposes — including all-important flags that decide the syntax, i.e. K&R C vs C90 vs C99 vs C11 C++ vs GNU extensions on or off vs C++98 vs C++03 vs C++11 vs Objective C vs Objective C++ vs Pascal vs FORTRAN77 vs FORTRAN90, etc. Who should specify say which flags should be used, including which language each file is written in? Should it be the system author, or the user? What does the author know about which dialect each file is written in, anyway? While we're at it, why not have the file extension decided by the user? Maybe we should have the type of Lisp file not be "lisp" or whatever the author believes it is, but whatever the compiler::*compiler-default-type* variable says. And too bad if the author, that idiot, thought it was "lisp", when the user, that genius, knows it's "lsp". The user is obviously better suited than the author at telling the names of the files actually present in his filesystem. An even better idea: let the user decide the filenames and the order in which he'll load them. That will make for a simpler build system! Actually, maybe we should also let the user decide how to transform the author's source files to executable code. Ain't nobody got time for Lisp compilers. In the end it's a matter of WHO specifies the settings for some compilation: the system author or the user. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org To make an apple pie from scratch, you must first create the universe. — Carl Sagan

On Fri, Mar 28, 2014 at 10:12 AM, Zach Beane <xach@xach.com> wrote:

Faré <fahree@gmail.com> writes:

...

In the end it's a matter of WHO specifies the settings for some compilation: the system author or the user.

Or, in this case, Faré.

Once it's accepted that ASDF will enforce the file type decided by the author, ASDF must have a default. And it's "lisp", not "lsp", not "cl", not "LIS", and certainly not the fleeting content of compiler::*compiler-default-type*. So yes, Dan Barlow made it "lisp". So sue him. Once it's accepted that ASDF will enforce the file encoding decided by the author, ASDF must have a default. And it's :utf-8, not :us-ascii, not :latin1, and not uncontrollable meaning of :default. So yes, I made it :utf-8. So sue me. Once it's accepted that ASDF will enforce the syntax variables decided by the author, ASDF must have a default for each of them. And so Robert Goldman will I hope pick a default for *read-default-float-format* that is not "whatever value that variable had at a time not fully controlled by either library author, application programmer or end-user". So sue him. I will argue that the choice of 'double-float (which I propose) vs 'single-float (which other people defend) as default float format is analogous to :utf-8 vs :us-ascii as the default encoding: it just allows for more values with less lossy behavior for the user. Sure, 'single-float is the old de facto semi-standard from 20 years ago, the way that :us-ascii used to be. But that was never a guaranteed binding, and if we are to guarantee a default binding today, we should base the decision on today's expectations, not on expectations from 20 years ago. Now, these days, people who do actually care about floating point computations seem to prefer double by default, and single as a sometimes optimization. Why don't we have a poll with heavy floating point users, and ask them which they prefer and expect as a default? They should be the target audience for this choice. So far, my data is that neuss and gendl prefer double, and that out of 213 bindings of *read-default-float-format* in Quicklisp, by people who care, 40 explicitly demand 'double-float, and 23 explicitly demand 'single-float. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Due to circumstances beyond your control, you are master of your fate and captain of your soul.

On Fri, Mar 28, 2014 at 11:00 AM, Zach Beane <xach@xach.com> wrote:

Faré <fahree@gmail.com> writes:

...

Once it's accepted that ASDF will enforce the syntax variables decided

This seems more like an "if" than a "once" to me.

Then please argue that. I for one fully agree that the big question is not about the specific defaults chosen by Dan Barlow and me in the past, and possibly Robert Goldman in the future, but about whether ASDF should provide a default that system author can rely upon. Will you argue against ASDF letting the system author fully control the file type of source files, rather than it depending on a global variable? Will you argue against ADSF letting the system author fully control the encoding of source files, rather than it depending on a global variable? Will you argue against ADSF letting the system author fully control the syntax of source files, rather than it depending on a global variable? And since you're fond of C analogies: should gcc determine the syntax it uses to compile a file based on an environment variable or on information fully determined by the software author in his Makefile? That's the high-order bit question. Let's argue it, with more than an ad hominem attack. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The least deviation from truth will be multiplied later. — Aristotle

On Fri, Mar 28, 2014 at 11:33 AM, Zach Beane <xach@xach.com> wrote:

Faré <fahree@gmail.com> writes:

...

On Fri, Mar 28, 2014 at 11:00 AM, Zach Beane <xach@xach.com> wrote:

...

Faré <fahree@gmail.com> writes:

...

Once it's accepted that ASDF will enforce the syntax variables decided

This seems more like an "if" than a "once" to me.

Then please argue that. I for one fully agree that the big question is not about the specific defaults chosen by Dan Barlow and me in the past, and possibly Robert Goldman in the future, but about whether ASDF should provide a default that system author can rely upon.

Will you argue against ASDF letting the system author fully control the file type of source files, rather than it depending on a global variable?

Will you argue against ADSF letting the system author fully control the encoding of source files, rather than it depending on a global variable?

Will you argue against ADSF letting the system author fully control the syntax of source files, rather than it depending on a global variable?

And since you're fond of C analogies: should gcc determine the syntax it uses to compile a file based on an environment variable or on information fully determined by the software author in his Makefile?

That's the high-order bit question. Let's argue it, with more than an ad hominem attack.

I do not want to see a system that relies on personal intercession to reduce the friction introduced by the things it breaks. Even if those things are not open source or free software projects, even if they use dirty, unhygienic techniques, even if they did not report promptly to you when a problem arose, even if they do not want your help, I do not think they deserve to lose.

If there is no clear way to get the effects you want without breaking code that works today, I would prefer nothing be done until such a solution is clearer.

I don't see how this argument applies specifically against this change. It's an argument against change in general: any change will tautologically break some things and is tautologically fixed by a person somewhere doing the work. Now, mind that no one is forced to change. As far as software that I have been maintaining goes, QRes is still using CCL 1.8, because no one took the time to fix the concurrency issues that appear after upgrading to CCL 1.9 or 1.10 (probably due to sloppy code on the QRes side rather than the CCL side, but who knows - in the past CCL changes have notably revealed Linux kernel issues). janderson is reportedly using ASDF and libraries from at least 5 years ago. If someone is perfectly happy with the code he has, no one wants to force them to upgrade, and I encourage everyone to use source control and backups to keep their builds 100% reproducible, just like we did at ITA. If someone randomly upgrades libraries without having working versions in source control, maybe the problem is with their lack of proper source control, not with change happening. I promise I'm not going to hack into anyone's machine and upgrade half the software in their back while leaving bugs in the other half. The question is what to do going forward regarding the change that WILL happen, and indeed how do we reduce overall future friction. Ten years from now, will there have been more or less friction if system authors control syntax, or if syntax depends on the values of variables that none of library authors, application programmers and end-users fully control? I argue that putting system authors in full control of their files' syntax will minimize overall friction, by A WHOLE LOT. Making the build totally unpredictable based on changes at the REPL and/or environment variables? Here's what causes a lot of friction. Once again, lisp file type and character encoding are cases in point. Yes, there are transition costs, and yes, an overly eager transition would increase these costs. To assess these costs is exactly why we're having these discussions to begin with, and why Anton is running all those tests. It's obvious now that this is not a change to happen today, nor until all libraries in Quicklisp are fixed. And therefore, I'm not advocating for a change before ASDF release 3.1.1, as I previously and over-optimistically hoped. But the change needs to happen sometime, and the sooner the better - and I'm hoping to make it sooner, by sending fixes to each and every library that depends on *r-d-f-f*. Unless you want to document that the user is responsible for always ensure himself that *r-d-f-f* is bound to 'single-float before he compiles anything, which is as far as friction goes is HIGH. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org He was a great patriot, a humanitarian, a loyal friend; provided, of course, he really is dead.

27.03.2014, 01:21, "Faré" <fahree@gmail.com>:

Can you re-run a full test with what I just pushed in branch syntax-control?

Fare, here are the results for the syntax-control, commit d32afa0c: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-40.html Looks like several libraries rely on single-float. Best regards, - Anton

...

...

...

...

"Fare" == Far <Far> writes:

Fare> Thanks a lot, Anton! Fare> Well, here's a valid argument in favor of 'single-float as the Fare> default: making 'double-float the default breaks 22 systems in Fare> Quicklisp (a bit fewer when you consider that some systems only Fare> depended on a broken one, but still), when making it 'single-float Fare> only broke 4 of them, all of which I fixed yesterday. If the default Fare> had to be decided right now, that would indeed make it a strong case Fare> for 'single-float. Stronger to me than any I've seen so far, and one Fare> capable of swaying my opinion indeed. I'm not really surprised since the initial value (if that's not default, then I don't know what is) for *read-default-float-format* is 'single-float. Ray