Here is the report for the syntax patch: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-38.html

14.03.2014, 20:20, "Anton Vodonosov" avodonosov@yandex.ru:

As systems depend on each other, a failing system breaks other systems.

To get sense of the situation the following report may help: http://common-lisp.net/project/cl-test-grid/asdf/asdf-load-failures-3.1.0.94... It includes only systems which fail with the syntax pach, while load OK without the patch.

The default sorting places more "important to fix" systems at top.

The term "root-blocker" means a system failing by itself, i.e. none of its dependencies fail.

For example atdoc has 1 root-blocker - clozure-common. If we fix clozure-common, atdoc will hopefully work (unless atdoc it has its own problem, which will be revealed in this case).

The systems which have count of root-blockers = 0 fail by themselves.

You may change sorting by clicking columns (holding Shift to sort by multiple columns).

Let me know if you have questions about this report.

Also "systems-blocked-exclusively" are systems which have no other failing dependencies except for the current system.

For example, ironclad has 120 systems blocked exclusively. This means if we fix ironclad the 120 systems may be fixed (Unless they have their own problems, which will be revealed in this case; but all the dependencies for these systems will be fixed).

14.03.2014, 23:47, "Faré" fahree@gmail.com:

Why this matters: http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html#%28part._.Safety_before...

You say "Creating a fresh copy of the standard readtable around each action is too expensive."

But why expensive? On CCL it takes 22 microseconds per readtable:

CL-USER> (time (dotimes (i 1000000) (copy-readtable))) (DOTIMES (I 1000000) (COPY-READTABLE)) took 22,616,000 microseconds (22.616000 seconds) to run. 10,355,591 microseconds (10.355590 seconds, 45.79%) of which was spent in GC. During that period, and with 8 available CPU cores, 19,905,727 microseconds (19.905727 seconds) were spent in user mode 2,293,215 microseconds ( 2.293215 seconds) were spent in system mode 1,952,000,061 bytes of memory allocated.

Faré wrote:

Robert, what do you think? Should we "just" copy-readtable and copy-pprint-dispatch around each action? Or should we try to share the tables for all files in a system? Should we try to have defsystem save the values of *readtable* and *print-pprint-dispatch*, while load-asd does the creation of fresh tables? Should systems and/or individual components have a more general purpose variables and variable-values slot, which should be passed to progv, and be initialized from an alist?

I do not think that we should address this issue in our current release iteration.

The CL spec does not, IMO, provide strong enough guidance about the use of *READTABLE* and *PRINT-PPRINT-DISPATCH* that we can safely enforce good behavior.

Furthermore, the language does not provide the constructs needed to observe good hygiene, or at least not easily. There's no readtable or pprint-dispatch equivalent to IN-PACKAGE. That's why, for example, Allegro provides *named* readtables (which tcr has tried to provide in a portable library).

Over the years, I have looked at large piles of code that modify the current readtable or pprint-dispatch table, using the available functions without the optional table designators. This is not ideal style, IMO, but I don't feel that it's the job of ASDF to fix it.

This is certainly not uncontroversial enough that we should start pushing out a solution today. We have enough controversy on our plate already! I am unable to keep up the dialog about BUILD-OP (under whatever name) and the default code location. We need to leave this sleeping dog to lie for now.

Best, r

On Fri, Mar 14, 2014 at 6:10 PM, Robert Goldman rpgoldman@sift.net wrote:

Quick follow-up: what about blowing this discussion into launchpad under the rubric of "Manage CL syntax" or something like that.

I would prefer not to lose the discussion.

I believe this is a place where a tool like POIU or XCVB really veers away from vanilla CL usage.

Vanilla CL usage encourages* actions during building that side-effect the global state of the image. These actions may be bad if one wishes to build individual components in multiple different CL images. But they are natural to people developing code incrementally and interactively using a REPL.

I don't want ASDF to drift into being a tool that prescriptively moves people away from that practice. I think that's too presumptive for a piece of infrastructure. I don't think it's my job to wreck code, just because it's bad style. And I don't think proper use of the readtable or of pretty-print dispatch is obvious enough that people will recover well if we mess with it.

As you have said, I am inclined to be much more conservative about changes like this.

Here is my theory: safe code needs to respect hygiene, because you never know which system are loaded before or after: it depends on the overall plan of the toplevel target system. Therefore is it never correct to rely on other systems having set a particular readtable, and it is never correct to side-effect the current readtable into something that modifies standard characters — or any character that any other program might want to use, for the matter. If you side-effect the readtable, and the readtable escapes, you are going to hurt someone, somewhere.

In the current setup, it is therefore a bug if you side-effect the readtable, and those programs that do are buggy. Using named-readtables is fine, but only if you reset the readtable to the standard one at the end of every file, or have a method that binds *readtable* around your compilation.

I see two ways forward: (1) recognize that it is a bug, and help programmers by being strict, modulo some compatibility mode for older systems until they are fixed, or (2) slightly change the semantics so that it's not a bug anymore, by making the readtable either per-file or per-system; then you can modify the readtable to your heart's content, because it's private.

My initial reaction was (1), and that's more or less the patch I sent to anton, who tells us it breaks a lot of things. At Anton's suggestion, I considered (2) and it looks promising — but we need to modify the protocol and/or use a new method-combination to achieve the effect in a backward compatible way.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The only way to have a friend is to be one. — Ralph Waldo Emerson

...

: Faré

: Robert

...

Here is my theory: safe code needs to respect hygiene, because you never know which system are loaded before or after: it depends on the overall plan of the toplevel target system. Therefore is it never correct to rely on other systems having set a particular readtable, and it is never correct to side-effect the current readtable into something that modifies standard characters — or any character that any other program might want to use, for the matter. If you side-effect the readtable, and the readtable escapes, you are going to hurt someone, somewhere.

In the current setup, it is therefore a bug if you side-effect the readtable, and those programs that do are buggy. Using named-readtables is fine, but only if you reset the readtable to the standard one at the end of every file, or have a method that binds *readtable* around your compilation.

I see two ways forward: (1) recognize that it is a bug, and help programmers by being strict, modulo some compatibility mode for older systems until they are fixed, or (2) slightly change the semantics so that it's not a bug anymore, by making the readtable either per-file or per-system; then you can modify the readtable to your heart's content, because it's private.

My initial reaction was (1), and that's more or less the patch I sent to anton, who tells us it breaks a lot of things. At Anton's suggestion, I considered (2) and it looks promising — but we need to modify the protocol and/or use a new method-combination to achieve the effect in a backward compatible way.

No, we cannot do either (1) or (2).

CL does not say it is a bug to rely on an upstream-loaded file to set the readtable to a value that you expect. So breaking code that makes that assumption by forcing file-scoped readtables and pprint dispatch tables on the programmer is not an OK thing for us to do.

It's not ASDF's job to break code that we consider to be bad code.

The CL spec went out of its way to *permit* the code that you are arguing to be buggy. They *could* have demanded a readtable argument, but they did not. They could have imposed a scope on readtables, but they did not.

Furthermore, we are not omniscient: the programmer could knowingly count on a non-file-scoped READTABLE in a way that s/he knows is safe, because s/he can prove to him/herself that *READTABLE* will be properly bound at all locations in the code. That information could be perfectly correct, but invisible to ASDF. We should not break such code.

Making constructs file-scoped willy-nilly is not something that ASDF should do. In my opinion, it's overreach.

To analogize: that's a job for lint, not a job for make.

I disagree. You cannot ever rely on a readtable from a different system, because you don't control which system was loaded immediately before you. That depends on the user's toplevel system and its dependencies. Therefore, it's always a bug to rely on it. You might want to depend on changes made by the file before you if it's in the same system, and ASDF could either bind *readtable* to a per-system object, or, if you want to try harder, update the *readtable* object after each file that is compiled or loaded, initializing it in *prepare-op* to a copy-readtable nil. Similarly, it's always a bug to pollute the next system. It really can't reasonably expect any useful changes, so every system should get its own table. If we try hard, there's a whole lot of syntax variables that would have to be preserved from file to file. And of course, even within a system, you better only rely on changes made by files you depend on, and not rely on changes not being made by files that don't depend on you — the latter being trickier.

Without ASDF enforcement, any syntax change is a landmine, *even at the REPL*. I remember using fare-quasiquote at the REPL, then doing a load-system after having changed asdf or such — now every file in asdf using ` was compiled into something that depended on fare-quasiquote at runtime. Big bug. Really, safe syntax change requires ASDF support. Once again, see http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html#%28part._.Safety_before... (which I just edited some more).

This is all probably too much work to be a 3.1.1 issue, but that's definitely something you'll need if you want to support syntax modification rather than let it be a big danger.

PS: you wanted to somehow have things happen around the loading of a system. We now have that, kind of: the prepare-op first, the load-op last. Except it might be only a compile-op or a load-fasl-op. Also, while this require special user magic, ASDF does not prevent a non-system to depend on a system, yet kind of makes the assumption that this will not happen. I suppose this requires documentation and/or painful runtime checking.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Pacifism is a shifty doctrine under which a man accepts the benefits of the social group without being willing to pay — and claims a halo for his dishonesty. — Robert Heinlein

Dear Anton,

Fare, I wonder, why ironclad fails with your patch.

I have checked the source, ironclad.asd creates new *ironclad-readtable* and binds cl:*readable* to it in the :around method for compile-op (like a home-made around hook)

So, it looks like ironclad does not modify any global behavior.

What your patch does?

My patch was enforcing read-only syntax tables around not just the .asd file, but each and every Lisp file.

Since that obviously is too strict, the next thing to try is one set of tables per system, and if that still fails, updating said tables after every action on the system, i.e. initialize the system's readtable, and around every action have (let ((*table* (saved-table))) (call-next-method) (setf (saved-table) *table*)). Problem is: what when you re-compile the system after some of it has changed? New readtable, or modify the same old? Also, to preserve the fasl-op linking model, there shouldn't be anything around load-op, or maybe something stricter, or, I don't know. This is all quite tricky.

I would never write a library which modifies global *readtable* during load, it's of course a bad style (when ASDF around hooks provide an easy instrument for my code to use custom readtable without affecting global readtable)

It's in practice a very bad idea indeed. We need to fix systems that do it.

Should ASDF fail the libraries which modify global readtable - I doubt it's ASDF role.

Yes it is, see my paper: http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html#%28part._.Safety_before...

Should ASDF provide libraries with clean readtable when they are build - seems like a good service, so doesn't matter what I did in REPL, (ql:quickload :any-library) always works the same way.

Yes, it is very important to make it safe to call ASDF while at the REPL. At the very least, ASDF should bind *readtable* around operate, if not around each action. Otherwise, bad thing will happen, and I've been bitten in the past.

But I am not sure 100%, this is a complex issue, more intensive consideration is needed.

Indeed. And testing.

But systems that modify the readtable should still be fixed. I shamefully admit I had some systems that used to do it up until last year or so.

It's rational to collect the information we already have into a ticket, and consider it more detailed when other currently active issues are closed, as Robert suggests.

Who's game for it? :-( At the very least, find the URL for the discussion on some web archive, and link to that in the bug report.

http://thread.gmane.org/gmane.lisp.asdf.devel/3883

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Two possibilities exist: Either we are alone in the Universe or we are not. Both are equally terrifying. — Arthur C. Clarke

On Sat, Mar 15, 2014 at 9:08 AM, Zach Beane xach@xach.com wrote:

Faré fahree@gmail.com writes:

...

...

Should ASDF fail the libraries which modify global readtable - I doubt it's ASDF role.

Yes it is, see my paper: http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html#%28part._.Safety_before...

So you want to make a change rapidly to suit the deadlines of a conference, and the justification for the change is the paper you wish to submit?

No.

1- I want the release to happen before the paper goes final, in April, whether this change makes it or not. I don't believe that a release every six months is too much. As far as I can tell, there are no blockers, and ASDF 3.1.0.94 is otherwise ready for release. It has passed all tests on Linux and with cl-test-grid and probably should pass them on Windows, Mac (and possibly Genera(!?)) before release.

2- I want the change to happen, at whatever pace it will come, before or after the release. And the best time to start is now. I don't decide what goes inside a release anymore, though if the change is conservative as I wish it to be and passes cl-test-grid without regressions, then I think it will be a great thing to include just before a #+asdf3.1 feature increase rather than just afterwards. I don't think it should block the release, though.

You can always adapt the paper to explain how the current asdf maintainers are too lazy to adapt to your insights, as you imply is the case for the current Quicklisp maintainer.

Not at all what I implied. If that's my current paragraph on Quicklisp left an impression of being disparaging to you, I apologize, and I'll update it ASAP.

I understand that time resources are limited. No laziness involved. You do what you want with your time, I do what I want. You're obviously not interested in upgrading to ASDF 3, I'm obviously not interested in waiting for you to improve ASDF further. We also obviously have different ideas as to what the mission of ASDF is — this paper expounds my case.

Regards,

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Factum, non dictum, amicus quaerit

On Sat, Mar 15, 2014 at 4:52 PM, Robert P. Goldman rpgoldman@sift.info wrote:

Here is an example of a kind of system that will be broken by this proposed change:

• We have a distributed system that is lisp-based.

• There is a small set of ASDF systems that constitutes the common base

of the modules of this distributed system.

• For each module, there is an ASDF system, which incorporates the

common base, the module proper, and possibly some additional systems.

• The common base has a readtable (and it may have a PPRINT-DISPATCH-TABLE).

• The distributed modules systems assume, safely, that the readtable

will be set by the common base.

• Code in the common base uses readtable-setting functions with the

default values for the readtable.

The clean thing to do would be to use named-readtables and/or cl-syntax, and have each file evaluate (in-readtable :foo) or have a perform :around method or around-compile hook that does it for you.

We could also support your doing (in-readtable :foo) only once for the entire system, though it's trickier.

(Question, either way: how does the readtable get re-initialized (or not) when you recompile a modified system? Should you start from a fresh readtable every time, or reuse the readtable of the end of compilation while at the start of compilation?)

But polluting arbitrary systems loaded after you is definitely a bad idea: if someone else write a system that depends on part of yours, that also depends on unrelated systems, these unrelated systems will end up being affected by your readtable changes, and, if they are just as "dirty" as yours, they might in turn affect your system.

Such idiom is guaranteed to break, badly, in *some* circumstances, and we shouldn't encourage or support it. The readtable interferences will only get worse at the REPL, when a modified system is re-compiled with readtable modifications from a system that it doesn't depend on, and the next time you try to load it from clean, you find it mysteriously depends on the other system.

As I understand it, the proposed change will break this kind of system. Indeed, I think it's likely to break the parent common base system as well as breaking the system composition (the use of the readtable is not confined to a single file in the common base, and I believe it straddles multiple subsystems, which are represented as individual ASDF systems).

Yes, it's a feature to break this kind of system, so as to enable instead systems where the programmer can safely rely on his system not being either victim or author of interference with other systems, and may both rely on a sane readtable and be able to modify his readtable safely.

But that there are systems like that suggests that the change should not be made without a lot of heads up, and so definitely not for 3.1.1.

I'm not going to defend the structure of this code, but I strongly feel that it is not the job of ASDF to impose a personal opinion that this coding pattern is bad, and should be *forced to break* when previously it worked perfectly adequately.

It did not work "perfectly". It happened to work because you control the build process to make sure you don't interfere with other systems, and ASDF de facto imposes on general purpose libraries that they can't safely do anything fancy with readtables, which sucks.

To allow libraries at large to safely enjoy localized syntax modification at the cost of requiring a little bit of discipline from a handful of dirty systems sounds like a pretty good deal to me, and totally the job of ASDF. Without ASDF, there's no way the libraries will be safe by default. Once again, the example of a random load-system at the REPL causing havoc because I had syntax modifications at the REPL is pretty dreadful.

To the best of my knowledge, the readtable manipulation functions in CL don't provide any easy replacement for the pattern of coding applied here.

named-readtables and cl-syntax are there to help. ASDF could and in my opinion should make things safe by default instead of unsafe by default.

Yes, one *can* set a global to the special readtable, and then one can write a special PERFORM method (one? or many?) that will bind *READTABLE* to the value of this global.

Yes, and many systems do that, including ironclad. Actually, inspecting the ironclad failure in cl-test-grid, I find that the problem was one with my code, whereby my :around method was run inside ironclad's :around method rather than outside, causing the breakage. I will implement a better protocol, and see how much breakage that causes.

This is already a monumental PITA, since the PERFORM method will have to refer to a variable named in a package that doesn't exist at the time the ASDF system is defined. It also forces our poor programmer to know more than s/he ever wanted to know about ASDF, and IMO inappropriately bleeds details of the code of the system into the system's build file.

The referring to the variable can be done through the now usual "read the string late" method as long used to name component classes and functions and recently operation classes.

I don't see how this "bleeds details about ASDF". Hacking readtables was always tricky. Up until now, it is unsafe to modify the global readtable, and for that reason, library authors are strongly recommended to not do it, while application authors do it at their own risk. With some help from ASDF, it can be made safe for everyone, and those application authors who used to do it the ugly way can still do it at the expense of one call to in-readtable or similar.

I believe that the proposed hygiene practice *also* leaves the programmer in an uncomfortable position if s/he incrementally compiles functions inside an editor buffer, since the readtable will not be properly set. I.e., AFAICT, although careful binding of the readtable, rather than treating it as a global variable, feels like the Right Thing, it has undesirable consequences because we don't have good ways to bind the readtable around the editor. If a -*- READTABLE: foo; -*- file header worked, this would be less of a concern, but using a header like that requires named readtables.

No, it's the CURRENT practice that makes it unsafe for everyone to compile files, because that will be done with the *current* readtable, which will be completely inappropriate for files in a different system. I've been bitten, hard, by that.

With my proposed changes (see what I pushed in branch standard-syntax), programmers can recompile a file and that will happen automatically in the correct readtable. (Though they will need a properly updated swank-asdf, if using SLIME.)

My conclusion is that enforcing some sort of compartmentalization of readtable is not only an inappropriate imposition of a particular style, but *also* means we are hanging the poor programmer out to dry. CL simply doesn't provide good tools for manipulating the readtable in the way we are proposing to enforce.

IMO, named readtables are the way to go, but they are not a common feature of the language, and the portable library is bit-rotted.*

I will not be pushing this proposed modification into the next release, and it will take a lot of convincing for me ever to incorporate it into ASDF.

Yes, CL does provide the tools, we were just not using them. and *that* was hanging the poor programmer out to dry. By implementing per-system syntax, we're providing them the hygiene they sorely need. People who were previously leaking readtable state will have it private to their system; if they were leaking readtable state from one system to the next, that won't work anymore, but that was already a hazard to systems they were not writing, so we're trading a "that's not working for another poor sod who can't know and can't do anything about it" to a "that's not working for someone doing something ugly who should know better and can do something about it". And that was the guiding principle of ASDF2: "he who knows is who specifies the information, he who doesn't know doesn't have to do anything".

*This is partly my fault, but the current disarray of cl.net, the library's position as part of an abortive EDITOR-HINTS master system, and the previous maintainer's putting it into a revision control system (darcs) that I don't use and don't understand are additional contributing factors.

If you're the new maintainer, I suggest converting the darcs repository to git and maybe putting the library on github.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The aim of public education is not to spread enlightenment at all; it is simply to reduce as many individuals as possible to the same safe level, to breed and train a standardized citizenry, to put down dissent and originality. ― H.L. Mencken

Faré wrote:

The clean thing to do would be to use named-readtables and/or cl-syntax, and have each file evaluate (in-readtable :foo) or have a perform :around method or around-compile hook that does it for you.

We could also support your doing (in-readtable :foo) only once for the entire system, though it's trickier.

(Question, either way: how does the readtable get re-initialized (or not) when you recompile a modified system? Should you start from a fresh readtable every time, or reuse the readtable of the end of compilation while at the start of compilation?)

But polluting arbitrary systems loaded after you is definitely a bad idea: if someone else write a system that depends on part of yours, that also depends on unrelated systems, these unrelated systems will end up being affected by your readtable changes, and, if they are just as "dirty" as yours, they might in turn affect your system.

The problem with this claim is that you are imposing a single model of ASDF usage on the community.

"Polluting arbitrary systems after you" is *not an issue* in the use case I presented.

That system is an in-house system, and it NEVER ENCOUNTERS arbitrary systems after it. It *knows* what systems are going to follow it, and it sets up for those systems entirely correctly.

I am going to resist this change because it assumes that the open source library world is the only use case for ASDF.

That is simply not the only use case. The example I have exhibited shows a case where the authors of the system are INTENTIONALLY, and CORRECTLY side-effecting the global state.

You may not like it, but it's not your place to break it.

I feel like you are assuming that the only use case for ASDF is to maintain *libraries* (as opposed to systems) that are to be composed with arbitrary other pieces of software.

That's not the only use for ASDF, so we shouldn't force that pattern on users.

You say:

"It did not work "perfectly". It happened to work because you control the build process to make sure you don't interfere with other systems, and ASDF de facto imposes on general purpose libraries that they can't safely do anything fancy with readtables, which sucks."

I don't agree. What I would say is "you are going to break my code in order to allow it to compose safely with other systems -- something that it will NEVER be called upon to do."

AFAICT, you are contravening your own principle here:

"People who were previously leaking readtable state will have it private to their system; if they were leaking readtable state from one system to the next, that won't work anymore, but that was already a hazard to systems they were not writing, so we're trading a "that's not working for another poor sod who can't know and can't do anything about it" to a "that's not working for someone doing something ugly who should know better and can do something about it". And that was the guiding principle of ASDF2: "he who knows is who specifies the information, he who doesn't know doesn't have to do anything"."

This is a case where the authors of the system are in a position to know that what they are doing is correct, and you propose to break their code in order to force them to support a non-existent use case. WE know that these systems -- which no one outside our firm will ever encounter -- work correctly. You are proposing that ASDF say "sorry, that's not good enough, since that wouldn't work if you put your code in Quicklisp." This code is no "hazard to any systems that [we] were not writing" but you propose to break it anyway.

The bottom line: I am not going to accept a patch that does this.

Yours sincerely, R

On Sun, Mar 16, 2014 at 11:18 AM, Robert P. Goldman rpgoldman@sift.info wrote:

PROPOSED NEXT STEPS:

1. A clear proposal for this modification be made. Right now the

details of the proposed modification are wrapped in a fairly opaque discussion. The discussion is framed in terms that are either too vague "Make the CL syntax predictable" or too specific -- very particular proposals about how variables will be bound. By now the discussion is also "protected by volume" against people who want to join the discussion, but weren't parts of it from the beginning.

I opened this bug with links to the current discussion on gmane: https://bugs.launchpad.net/asdf/+bug/1293325

2. The proposed change should be modified to operate only in a "strict

mode", allowing existing legal CL code to continue to work.

Agreed, and until the dust settles, "strict mode" cannot be the default. I'd argue it should become the default eventually (i.e. next year).

I propose that individual systems be allowed to proclaim that they should be operated on in strict mode, or cannot be used in strict mode.

"strict mode" would be available as a local declaration: a system could specify that it should be compiled and loaded with readtable hygiene, and that would *always* happen when it is operated on.

The library user would also be allowed to specify a global strict mode setting, and this setting would be applied in some way like the following (this is a tentative proposal):

a. systems that do not specify strict or non-strict mode would be compiled in strict mode if the user so specifies. This would lead to errors in some cases, but they would be errors that the user wants to see.

b. If the caller specifies strict mode, it will be an error to depend on systems that want to be compiled in non-strict mode. This would achieve the safety that you wish to see.

Unhappily, strict mode is a global flag: the question is "which readtable is this system going to be read with?". The only reasonable answer is: the readtable it was meant to be read with, which the author knows, and should be the standard readtable by default, unless explicitly overridden by the author. The backward-compatible (if it's not backward, it's not compatible) is "whichever readtable was active at the time", with sometimes comical consequences, especially when the user was using a non-standard one at the REPL.

3. Getting this right will require much testing and experimentation.

It will not appear in the next ASDF release.

Indeed, this is not 3.1.1 material at all. Hopefully, disabled by default, for 3.1.2, and enabled in a year in 3.2.1 — or whatever you call the next releases.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I hold that America, Champion of the World is a betrayal of its humble beginnings as One Nation Underdog.

Faré wrote:

...

...

2. The proposed change should be modified to operate only in a "strict

mode", allowing existing legal CL code to continue to work.

Agreed, and until the dust settles, "strict mode" cannot be the default. I'd argue it should become the default eventually (i.e. next year).

I doubt that's even feasible, since the strict mode will have to cross systems. It will be easier to have systems say "I know I behave according to the strict policy, so I will LOCALLY ask to be built and loaded strictly."

I think one reason you think that this change is feasible and I don't is that you have a stronger sense that there's a kind of "one library/system == one ASDF file."

In our bigger multi-component systems that are based on ASDF, I find that we typically end up with multiple ASDF systems, because we have subsystems that use different libraries, and want to specify their own dependencies.

AFAIK you can't do this without introducing new top level systems, since modules cannot depend on libraries.

That's why, for example, we have systems that bleed readtable across their boundaries: the systems really aren't stand alone entities, but limitations on ASDF expressiveness, and the desire to get more modularity than the (necessarily in-line) :MODULE will permit, causes a proliferation of systems that are NOT libraries, and have meaning only in context.

I'm not going to move ASDF towards breaking such systems.

That's why, for example, we have systems that bleed readtable across their boundaries: the systems really aren't stand alone entities, but limitations on ASDF expressiveness, and the desire to get more modularity than the (necessarily in-line) :MODULE will permit, causes a proliferation of systems that are NOT libraries, and have meaning only in context.

I'm not going to move ASDF towards breaking such systems.

Robert, this is how i see it reading this thread:

- you know at least about one big and complex codebase with many asdf systems, where isolating *readtable* would break it and cause a lot of work/headache that is hard to debug, especially if the codebase doesn't compile free of warnings to begin with. this headache would probably be yours in case of this system.

- to Fare (and to me, who have wasted precious hours due to this) readtable higiene clearly looks like a major improvement when looking at it objectively, so much so that somewhere down the road breaking systems that now break other systems (by modifying the global readtable) is acceptable. my impression is that you would also agree with this if the legacy was not pulling your back.

*if* this is the case, then why not accommodate for such legacy systems/use-cases simply by e.g.:

;; force upgrade (asdf:load-system :asdf)

;; optionally, for extra safety (asdf:load-all-dependencies-of :my-legacy-system)

(setf asdf:*readtable-higiene* nil)

(asdf:load-system :my-legacy-system)

and then we can have our cake and eat it too...

(alternatively a new system class could be introduced and all legacy systems could specify a :class "hu.dwim.asdf:hu.dwim.system", just like we need to do currently for self-defense purposes against e.g. readtable polluters and against we polluting the readtables.)

Dear Anton, dear Robert,

can you run tests with the fare-3.1 branch of asdf?

I prepared this branch fare-3.1 with the changes I'd like to see in 3.1 *plus* some attempt at a minimal syntax control feature, as described in one previous email:

It notably has (defvar *shared-readtable* (copy-readtable nil) "This shared readtable allows legacy applications to keep modifying a global shared readtable while maintaining some hygiene for those who want to use their own readtable. It is subject to the following restrictions, which always existed but were previously implicit: A- no modifying any standard character, B- no two dependencies assigning different meaning to the same non-standard character. Using any non-standard character while expecting the implementation to treat some way counts as such an assignment of meaning. C- libraries need to document these assignments of meaning to non-standard characters. D- free software libraries will register these changes on: http://www.cliki.net/Macro%20Characters ") (defvar *shared-pprint-dispatch* (copy-pprint-dispatch nil) "*print-pprint-dispatch* table shared by all ASDF systems. It should match the extensions of *shared-readtable* -- see the latter variable's documentation.")

And it binds the *readtable* to that around ASDF compilations.

I don't make attempts to track modifications to the *readtable* binding itself, using the more complex strategy described in a previous email.

The branch is probably misnamed.

NB: while merging everything together and doing preliminary tests, I found bugs or backward incompatibilities in my previous branches — Robert, don't just merge them in, but if you agree to the functionality currently in the build-op and rename-bundle-op branches, tell me and I'll do the merging including fixes.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The rule is, jam to-morrow and jam yesterday, but never jam today. — Lewis Carroll

On Sat, 29 Mar 2014, Anton Vodonosov wrote:

29.03.2014, 02:39, "Daniel Herring" dherring@tentpost.com:

...

Would it be possible to have a special header in asdf files indicate that the contained system(s) should be loaded using the new syntax mechanism?

[...]

;; asdf:config '(asdf:+clean-readtable+)

No, it's inconvenient, why use special parser if we can just directly specify what we want in a lisp expression.

I haven't checked recently, but in the past ASDF files have had significant code even before the system definition. It was common to find conditionals based on *features*, comments for users to select from a few customization options, numbers being read, etc.

Once you have read the form, it is too late to modify the environment that is used to read it. Thus I suggested that we adopt a convention to check for a "magic header" that changes the syntax used for the rest of the file. The suggested syntax was meant to make this header invisible to legacy systems, unambiguous in meaning, extensible, and fairly easy to parse for new sysetms.

- Daniel

I haven't checked recently, but in the past ASDF files have had significant code even before the system definition. It was common to find conditionals based on *features*, comments for users to select from a few customization options, numbers being read, etc.

Once you have read the form, it is too late to modify the environment that is used to read it. Thus I suggested that we adopt a convention to check for a "magic header" that changes the syntax used for the rest of the file. The suggested syntax was meant to make this header invisible to legacy systems, unambiguous in meaning, extensible, and fairly easy to parse for new sysetms.

Emacs already has two such conventions. If someone wants to implement such a thing, that's the way to go. I'm sure Robert accepts patches, though in this case, you might want to start with an ASDF extension, a la asdf-encodings, that does exactly that with respect to encodings (many thanks to Douglas Crosher).

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If money is your hope for independence you will never have it. The only real security that a man will have in this world is a reserve of knowledge, experience, and ability. — Henry Ford

Hi Fare, sorry for the long reply.

Will do it, but was a bit overloaded recently by work and personal tasks, + was running tests for quicklisp 2014-03-17 and ABCL 1.3.0

Is you request still actual - standard-syntax branch (commit 2c1107)?

16.03.2014, 09:17, "Faré" fahree@gmail.com:

Dear Anton,

can you do a cl-test-grid test with the code in the standard-syntax branch from git://common-lisp.net/projects/asdf/asdf.git (commit 2c1107) ?

That branch has my "each system gets its own syntax" thing, based on a general-purpose "each system has a set of private variables" protocol. This protocol allows to designate variables using a string that will be parsed later, but doesn't somehow try to canonicalize names — if you use two different names for the same variable, you lose. By adding :after methods to compute-system-variables, more variables can be configured with configure-system-variable and configure-system-variables — this makes it a good basis for some future out-of-band configuration of ASDF, though such mechanism would need to be part of ASDF itself, or loaded early via an ASDF init file of some sort to be effective. Some DSL could then specify variables to include in a system's private variables, with optional initial value, depending on the name and version of the system and on various features, with usual rules (or per-system rules?) to inherit or ignore other configuration settings. Such a DSL is *not* included in this branch.

This protocol itself uses a new call-with-action-context protocol that perform-with-restart calls around perform. This is mostly backward compatible, except that existing clients who call perform directly (rather than perform-with-restarts) might miss those around methods — to get that effect we'd need a new method combination. swank-asdf would notably need to call perform-with-restarts instead of perform, and if possible to enable the try-compile-file-with-asdf hook function by default.

I tested it to correctly process ironclad, at least.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Gods don't kill people. People with Gods kill people. — David Viaene

On Tue, Mar 25, 2014 at 12:16 AM, Anton Vodonosov avodonosov@yandex.ru wrote:

Hi Fare, sorry for the long reply.

Will do it, but was a bit overloaded recently by work and personal tasks,

• was running tests for quicklisp 2014-03-17 and ABCL 1.3.0

Is you request still actual - standard-syntax branch (commit 2c1107)?

Actually, it's now branch fare-3.1, commit d615e7fb.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Faré's Second Law of Dissent: I am Right, whence it follows that all who disagree with me are either (1) evil, (2) stupid or (3) crazy. (The alternatives are not mutually exclusive.) This universal law is valid for all values of "me", including "you".

Hi Fare,

I didn't really understand how exactly the changes work, but here is the result: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-39.html

Best regards, - Anton

25.03.2014, 21:54, "Faré" fahree@gmail.com:

Dear Anton,

the new thing to test is the HEAD of branch syntax-control. Please test with SBCL first. This branch contains my experimental syntax control, binding a *shared-readtable* around compilation (itself the *readtable* at the time ASDF is loaded), so that promiscuous systems may keep modifying a shared readtable under the usual conventional constraints (now documented: all standard characters reserved, no conflict allowed or bust), while mutual protection is implemented for these systems and those that might wish to go beyond these constraints, particularly so at the REPL.

If this has no regressions, then I'll try to convince Robert to merge that in master and make it the new release candidate — assuming Robert's big promiscuous project also compiles with it (if it doesn't, I explained my more elaborate plan in an earlier email).

If this has regressions, and/or if Robert is not convinced, then what's in the master branch is our release candidate.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Luck occurs when preparedness meets opportunity.

Faré fahree@gmail.com writes:

femlisp raises an interesting issue: it has (setq *READ-DEFAULT-FLOAT-FORMAT* 'double-float) in setup.lisp, which is cancelled by the with-standard-io-syntax that I introduce in my syntax-control branch. I just pushed a change in said branch to make the 'double-float the default in with-asdf-syntax, which I believe is a more useful convention than what the standard provides.

I don't like the idea of ASDF setting a different *READ-DEFAULT-FLOAT-FORMAT* value during ASDF operations than it will have in the REPL, during plain loading, or compile+loading individual files outside of ASDF.

Zach

On Thu, Mar 27, 2014 at 12:30 PM, Robert Goldman rpgoldman@sift.net wrote:

Zach Beane wrote:

...

Faré fahree@gmail.com writes:

...

femlisp raises an interesting issue: it has (setq *READ-DEFAULT-FLOAT-FORMAT* 'double-float) in setup.lisp, which is cancelled by the with-standard-io-syntax that I introduce in my syntax-control branch. I just pushed a change in said branch to make the 'double-float the default in with-asdf-syntax, which I believe is a more useful convention than what the standard provides.

I don't like the idea of ASDF setting a different *READ-DEFAULT-FLOAT-FORMAT* value during ASDF operations than it will have in the REPL, during plain loading, or compile+loading individual files outside of ASDF.

Zach

I second this dispreference. As an old-school lisper, used to working extensively on incremental development in the editor buffer, and even in the REPL, this seems bad to me. The errors it will lead to will be odd corner cases, bewildering to ordinary programmers not versed in the fussy details of ASDF implementation. Put yourself in such a programmer's shoes, when his/her code that worked fine in the repl breaks in ASDF, or vice versa. What's the debugging process?

Maybe the default is undesirable, but it's not our job to patch all the undesirable features of Common Lisp.

The more divergences between CL-in-ASDF and CL that are introduced in this branch, the more likely it is that it will have to live on as a permanent fork and not be merged.

I also have this preference, but I slightly disagree on how to resolve it.

femlisp side-effects *READ-DEFAULT-FLOAT-FORMAT*, which means that every system compiled after it will be treated differently than if it were compiled before it — and once again, whoever writes the system does not and cannot control the order in which things are compiled, which ultimately depends on what toplevel system the user operates on and what systems have or haven't already been compiled (and are up to date) before he operates on that toplevel system. This means that the semantics of your system are already dependent not just on the REPL but also other uncontrolled and uncontrollable state. This is not desirable.

In effect, when you write 1.0 instead of 1.0f0 or 1.0d0 in a file, you're currently saying "it's OK to compile with single-float and double-float alike". Thus, ASDF binding *READ-DEFAULT-FLOAT-FORMAT* to either value (1) is not breaking any previous contract, and (2) is making things more deterministic, hence better. And double-float is a generally more useful choice than single-float in this context, not just for femlisp, but for all newbies who don't expect precision loss, etc.

My solution would be to (setf *read-default-float-format* 'double-float) as part of loading ASDF, just like we (setf *print-readably* nil) for the sake of Allegro, just like we change the package to ASDF-USER before loading an .asd file, etc. Then, the REPL and ASDF still agree by default, and the user is responsible if he makes them disagree, and the library authors don't have to worry what the user uses. Note that :around-compile hooks and other :around methods already make compiling via ASDF different from just calling compile-file. Note that swank-asdf already has an option (unhappily, disabled by default) to not call compile-file but use asdf:compile-op on C-c C-k. It unhappily doesn't do anything at all for C-c C-c, and that would be a trickier thing to fix, especially if the around wrapper changes the readtable (you'd have to call that), or if it enforces a file-granular invariant that locally broken by individual forms (you'd have to not call that).

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The trouble with opportunity is that it always comes disguised as hard work. — Herbert V. Prochnow

The thing that concerns me about this proposed change is that the poor programmer may look up the CLHS and see that the default value is specified to be 'single-float.

If anything, I'd say that ASDF should force standards compliance, and use 'SINGLE-FLOAT.

But I'd rather have ASDF just leave this alone.

CL is a language with side-effects baked into it. It's not our job to fix all of them: if we don't like that, there's always Scheme, Clojure, or Haskell.

Note that we can still tell the femlisp maintainer that we don't like what the library does and ask for it to be fixed, without feeling that we have to go out and break applications that use the current femlisp behavior, or that assume that floats are read as single-floats by default.

To add to the concern about breakage, this pushes more complexity into ASDF....

Fare, if femlisp configures *read-default-float-format* why does it fail with your ASDF change? Do you restore standard CL syntax around every file compilation? Maybe isolating syntax changes at system boundary is more convenient?

...

...

...

...

"Fare" == Far <Far> writes:

Fare> On Thu, Mar 27, 2014 at 2:00 PM, Anton Vodonosov avodonosov@yandex.ru wrote: >> Fare, if femlisp configures *read-default-float-format* >> why does it fail with your ASDF change? Do you restore standard >> CL syntax around every file compilation? Fare> Yes I do, although I just committed a change to override Fare> *read-default-float-format* to 'double-float, which I believe is more Fare> useful.

I disagree that that is more useful. While I almost always use doubles, there are lots of reasons to use singles. Just because I like doubles doesn't mean we're right and everyone else is wrong.

If you must choose a value, it should be 'single-float, because that is the CL default.

I also tend to agree with Robert. asdf is a make, not a lint+make.

Ray

Faré fahree@gmail.com writes:

It's not a matter of lint at all.

Very true. Lint is an *optional* program, and you can easily choose to ignore it.

Zach

Faré fahree@gmail.com writes:

In the end it's a matter of WHO specifies the settings for some compilation: the system author or the user.

Or, in this case, Faré.

Zach

Faré fahree@gmail.com writes:

Once it's accepted that ASDF will enforce the syntax variables decided

This seems more like an "if" than a "once" to me.

Zach

Faré fahree@gmail.com writes:

On Fri, Mar 28, 2014 at 11:00 AM, Zach Beane xach@xach.com wrote:

...

Faré fahree@gmail.com writes:

...

Once it's accepted that ASDF will enforce the syntax variables decided

This seems more like an "if" than a "once" to me.

Then please argue that. I for one fully agree that the big question is not about the specific defaults chosen by Dan Barlow and me in the past, and possibly Robert Goldman in the future, but about whether ASDF should provide a default that system author can rely upon.

Will you argue against ASDF letting the system author fully control the file type of source files, rather than it depending on a global variable?

Will you argue against ADSF letting the system author fully control the encoding of source files, rather than it depending on a global variable?

Will you argue against ADSF letting the system author fully control the syntax of source files, rather than it depending on a global variable?

And since you're fond of C analogies: should gcc determine the syntax it uses to compile a file based on an environment variable or on information fully determined by the software author in his Makefile?

That's the high-order bit question. Let's argue it, with more than an ad hominem attack.

I do not want to see a system that relies on personal intercession to reduce the friction introduced by the things it breaks. Even if those things are not open source or free software projects, even if they use dirty, unhygienic techniques, even if they did not report promptly to you when a problem arose, even if they do not want your help, I do not think they deserve to lose.

If there is no clear way to get the effects you want without breaking code that works today, I would prefer nothing be done until such a solution is clearer.

Zach

I do have control: If femlisp or any other library makes a boneheaded decision that breaks my software, I can stop using it.

yes, resolving that is trivial -- once you have identified the problem.

regarding the recent discussions i'm generally baffled why it is at all a question whether to make a build software deterministic or not. in my view if there's anything in the global state that has an effect on the building of a software, anything, then it's a bug.

and if this state can be set in a way that it leaks out and influences builds later in time, then it's an ugly bug feasting on programmer nerves and time.

the only question to me is what should be the defaults, and how can the transition be made smoother with deprecation warnings and whatnot.

and if there are some unmaintained, bitrotten libraries and some software depending on them, then they have all the freedom in the world not to upgrade ASDF, or to put on the maintainer hat and patch the bitrotten libs.

by now the time on spent discussing this would have easily been enough to fix all of them twice over, and to add a section to the top of the manual, with bold, that lists the global state that ASDF guarantees and isolates.

On 28 Mar 2014, at 19:00, Robert Goldman rpgoldman@sift.net wrote:

I would be much happier to see this conversation direct itself towards a design for providing strict mode, rather than arguing about whether or not "strict mode" should be universally imposed. I will say it again: that will not happen.

Robert, I really like your idea of a strict mode of ASDF that isolates a single system from global state changes.

While reading the thread, a funny analogy found its way into my head:

To some degree, the isolation feature is similar to a network firewall. Strict mode could work like a firewall config, where one would specify a rule for allowed state changes (both inbound and outbound) in the system definition.

This is just a rough draft of what it could look like for the ASDF user:

(defsystem "my-app-system" :isolation ( ;; this system modifies the readtable (:export :readtable)

;; import changes made by my-base-system (:import :all "my-base-system")

;; use readtable changes made by dependencies (:import :readtable :dependencies)

;; use global declarations (:import :declarations)

;; catch-all rule: isolate against any other changes (:deny-export :all) (:deny-import :all) ) :components ...)

For backward compatibility, the default rules would allow any state to be exported or imported. In order to make the build deterministic, rule violations should throw a condition (failing the build).

If the isolation option can be applied to any component, it would even be possible to isolate individual source files/directories in a system against each other (although it would be kind of ridiculous if anyone ever did this in practice).

It might also be feasible to allow external overrides/extensions for the rules defined by the systems. This way, the person responsible for build issues at e.g. ITA could maintain total control of the side effect propagation without modifying the system definitions themselves.

I believe that given some development effort, a lot of things can be made controllable by this rule model. Things that come to mind right now are printer variables, reader variables, declarations and maybe even package locks, but I'm sure there's even more to it.

I agree with Faré about the fact that it's OK if this feature works only when using ASDF to compile systems. Anyone who wants to use this during interactive development with SLIME will need to use SLIME's ASDF integration.

-f

I think ASDF should not change the semantics and break existing system.

We can provide new, improved functionality without breaking anything. And keep old systems working forever with new ASDF.

Lets fist consider and decide, how exactly the best support for syntax customizations should work.

Withoug compromises (like allow shared readtable modifications, but resetting *read...* variables - that's strange compromise). Lets ignore backward compatibility for now.

Best regards, - Anton

29.03.2014, 19:09, "Attila Lendvai" attila.lendvai@gmail.com:

...

We can provide new, improved functionality without breaking anything.  And keep old systems working forever with new ASDF.

"forever" is a very strong word Anton. e.g. if CL doesn't fade into the category of history in 10 years, meaning that a significant number of people will still use it to develop new software, then i'll be both surprised and disappointed.

I know people who work with PL/I till now; and even more people who use C++. I don't worry about CL in 10 years.

But that's all off-topic for asdf-devel

...

and if this state can be set in a way that it leaks out and influences builds later in time, then it's an ugly bug feasting on programmer nerves and time.

This would be true only if "a software" was equivalent to "an ASDF system." That is not the case. I have seen many applications that are made up of a large number of ASDF systems, because ASDF does not necessarily provide nice structures for large applications. In such

we also have libs with 10+ asdf systems.

application builds, you WANT state to bleed from one ASDF system to another, because these ASDF systems are not freestanding entities.

one can write a trivial loader file that calls ASDF with some extra keyword parameters to load such systems.

Complaining that state leaks in and out of these systems is like complaining that state leaks into a closure. It's intended to leak, and that is not a bug.

the vast majority of the asdf systems don't want leakage, and defaults should be covering the majority use-case (if/when it's possible without setting up landmines).

reproducible builds are a valuable feature.

if i wanted to cater for the use-case you described then i'd still make full isolation the default with features to let systems explicitly turn off specific isolations, or just simply let them on their own to subclass and customize ASDF the way we currently have to subclass and customize asdf to avoid the leakage from the hu.dwim.* universe (we use e.g. readtable customizations).

and whoever is not ready to invest time in following the changes, should simply not upgrade, and then it's a non-issue. but this starts to feel like preaching and i don't really have that much stake in this anymore because we have already invested in covering our bases... so, these are just some 0.02...

29.03.2014, 02:32, "Faré" fahree@gmail.com:

Additionally, my ASDF syntax-control branch has a mechanism that allows systems to specify variables they want to bind around their system, so you could specify: (defsystem foo :variables ((*read-default-float-format* . (constantly double-float))) ...) where the assoc'ed value is a specification as per ensure-function, which lets you specify a string "(lambda () (not-yet-created-package:foo 1 2 3))"

Looks like extension of the around-compile hooks features. Is the :variables specification only works for compile and load-source operations, the same way as around-compile works?

...

regarding the recent discussions i'm generally baffled why it is at all a question whether to make a build software deterministic or not. in my view if there's anything in the global state that has an effect on the building of a software, anything, then it's a bug.

I think one question is whether it's worth taking a path to this goal that breaks programs that currently work. I don't think all such programs are bitrotted junk that can be fixed up in a day.

Nobody is proposing to break anything. On the contrary, the proposal is to make everything build in a more reliable way.

Once again: 1- there will be no hacking into other people's machine to upgrade half their software. 2- there will be no changing the ASDF default until patches have been sent to all maintainers of libraries currently in Quicklisp and ample heads up was given to everyone who is not in Quicklisp. For the change in default character encoding, this took one year. I imagine a change in syntax control settings will take the same: the code will be there but disabled by default for a year, then the default will change. 3- code that was previously guaranteed to work portably will still work portably. 4- most code, that was depending on unenforceable but standard environmental dependencies, will now work reliably even when compiled in an environment when other systems and/or the user have bound variables incompatibly. 5- some code, that was depending on unenforceable and non-standard environmental dependencies, will now fail reliably even when compiled in an environment when other systems and/or the user have bound variables compatibly -- and will need to be fixed.

Then users can change their *READ-BASE* to 36, their *READ-DEFAULT-FLOAT-FORMAT* to 'SHORT-FLOAT, their *READ-EVAL* to NIL or their *READTABLE* to something that implements a Ruby compiler, and still be able to call asdf:load-system under the hood and have everything work fine. Actually, an ASDF server in the background might be farming compilation of each and every file from a clean image with only its declared dependencies based on a clean environment, each on a different worker in a distributed network - and the semantics will be exactly the same.

Frankly, the notion that syntax bindings from the REPL might leak into an unrelated system's compilation and break it sounds completely braindead to me, and I'm surprised anyone tries to defend the concept.

The question to me is not should the change happen, but how.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org War does not determine who is right — only who is left. — Bertrand Russell

On 28 Mar 2014, at 20:37, Attila Lendvai attila.lendvai@gmail.com wrote:

...

...

regarding the recent discussions i'm generally baffled why it is at

...

...

all a question whether to make a build software deterministic or not. in my view if there's anything in the global state that has an effect on the building of a software, anything, then it's a bug.

I think one question is whether it's worth taking a path to this goal that breaks programs that currently work. I don't think all such programs are bitrotted junk that can be fixed up in a day.

those programs that currently work will keep on working if ASDF is not upgraded.

no upgrade, no breakage.

If you can’t upgrade ASDF, you may also not be able to upgrade quicklisp. If you can’t upgrade quicklisp, you may also not be able to get updates to existing libraries. At least, it may be harder than necessary (like in the pre-quicklisp, pre-asdf-install days), and you can’t benefit from the efforts that were put into ASDF and quicklisp to the same extent as others anymore. [1]

Don’t try to abuse the popularity of ASDF and quicklisp to enforce changes to Common Lisp, the Language that may not be accepted by the community.

to me it's a very strange argument that the right way to avoid breaking unmaintained old code is by hindering change that makes new things better in the future.

If the proposed change to enforce double-float as a default *read-default-float-format* was unambiguously an improvement, there would be no problem. But it’s not an improvement, not by a long shot.

Keep in mind that floats are always only approximations. Doubles are only better approximations, but they are approximations nonetheless. You can rarely use a particular float format without thinking about the impact it has on the results of your algorithms. Blindly assuming that whatever default defined by the language, or by some tool, is the best one for whatever purpose will not help you get good results in the general case.

Doubles are not generally better than singles. Doubles can be better than singles because they are more precise - and they are better only if this is what you care about. However, singles can be better because they use less space, and operations on singles can be executed more efficiently on CPUs. Vector units can fit more singles in their vector registers than doubles, so you will get even better performance from singles if/when Common Lisp implementations start to support vector operations better (especially AVX). 64-bit Common Lisp implementations can represent singles as immediate values, whereas doubles need to be boxed, which causes additional performance overheads. Depending on your goals, singles may therefore be a much better default than doubles.

If portability is a concern, than the answer may again not be so clearcut: For example, a language like Java enforces particular semantics for floats to the extent that, depending on the underlying CPU, you may not get the best performance out of it. Java allows you to specify strict or non-strict semantics, which allows you to choose between result portability and performance portability. Which of those is the best default is again open for debate. It’s not so clear that a focus on result portability is the best choice, considering that we’re dealing only with approximations anyway. Choosing doubles as the default *read-default-float-format* out of a concern for result portability may not help you because, if the CL implementations involved choose better performance over duplicating results exactly /across processors/, you didn’t win that much in the first place.

Common Lisp allows library authors to be very precise with regard to what they expect from floating point operations: The number syntax allows you to be very precise about the number representation (D, F, L, or S). It also allows you to specify that you want to be generic with regard to number representation (E). Choosing E is akin to stating that your library can work with any representation.

*read-default-float-format* lets you choose between different representations for such generic libraries. The scoping rules for dynamic variables in Common Lisp give you two choices: Either you tell a particular library that it chooses a particular representation for its generically chosen representations - you do this by dynamic binding. Or you tell a set of libraries (all that you are loading from now on) that they should all choose the same representation for their generically chosen representations - you do this by side effect. This is desirable when your goal is to ensure that the different libraries work well together. (This is for example how I understand Femlisp’s choice: It states that it works well only for doubles, and you better use doubles also with other libraries that you want to use in conjunction with Femlisp. I have no particular opinion about whether this is a good idea or not, since I’m not a domain expert and cannot judge this, but it’s likely that neither can you.)

I am very happy that Common Lisp gives a lot of choices in this (and many other [!]) regards. These options shouldn’t be restricted due to the lack of imagination of some developers who believe that limiting choices is always better. There are already (far too) many languages out there that go down that route. If you want Racket, you know where you can get it. Common Lisp is a language for people who know what they are doing, not for people who need to be held by their hands.

You also have the option to fork a particular Common Lisp implementation and change it in incompatible ways to demonstrate to a larger audience that such changes could be beneficial, if that is your goal. But again, don’t attempt to force /all/ Common Lisp implementations to behave a certain way. This will certainly fail, and that would be a pity, because this would invalidate many benefits that did come with ASDF and quicklisp. The fact that the proposed changes are discussed so actively is a clear sign that people care about these things.

For what it’s worth, I’m very happy that Robert is taking a conservative stance here, and that ASDF is slowing down with regard to pushing out features that potentially cause problems for existing working libraries. It already created too many unnecessary problems in the past.

Pascal

[1] I already encountered libraries that don’t load with current ASDF versions anymore, although they are defined as ASDF libraries, which is a shame! Personally, I am maintaining Closer to MOP, and because of that, I can afford not paying attention to newer ASDF/quicklisp versions only to a limited extent.

-- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

On 29 Mar 2014, at 18:07, Faré fahree@gmail.com wrote:

...

: p-cos

...

...

no upgrade, no breakage.

If you can’t upgrade ASDF, you may also not be able to upgrade quicklisp. If you can’t upgrade quicklisp, you may also not be able to get updates to existing libraries. At least, it may be harder than necessary (like in the pre-quicklisp, pre-asdf-install days), and you can’t benefit from the efforts that were put into ASDF and quicklisp to the same extent as others anymore. [1]

Sorry, you don't get to demand change and rant against change in the same paragraph. You don't get to praise the efforts put into ASDF and rail against them in the same paragraph. The same legitimate reasons that justify desirable change in your libraries also justify desirable change in ASDF. The same things that make these changes desirable are the things that break systems that relied on non-portable kludges and quirks.

You only get to discuss whether a particular change is desirable.

That is precisely what I’m discussing, nothing else. The paragraph above is an answer to the suggestion that you can choose not to upgrade ASDF if you don’t like its changes. It’s arguing that one may be in a position where you cannot afford not to upgrade. That’s why some controversial suggestions for change may affect some users more than others.

...

If the proposed change to enforce double-float as a default *read-default-float-format* was unambiguously an improvement, there would be no problem. But it’s not an improvement, not by a long shot. [...]

There are two dependent changes being discussed: (1) guaranteeing a value of *read-default-float-format* and other syntax variables when compiling a library. (2) having this value be 'double-float.

It looks like (2) will definitely not happen. I still argue that (1) is essential for build determinism, and enabling users to change syntax at the REPL.

Are you also considering the following use case?

- Assume I’m developing a library for a specific Common Lisp implementation. Let’s choose LispWorks as an example, although it could be any other. (I’m just choosing LispWorks as an example because I happen to use it myself most often.) LispWorks comes with its own system definition facility, and with its own default values for pre-defined global variables in the CL package that may or may not be the same as those of other CL implementations. Since I’m also maintaining portable libraries, I prefer to use ASDF even for LispWorks-only libraries, so I don’t have to change tools on a regular basis. However, I would also like to make sure that LispWorks users can use the LispWorks system definition facility for my library as an alternative. Will it be possible to use ASDF without it messing with the default values for pre-defined global variables, so that I have a guarantee that I get the same result no matter whether I use ASDF or some other system definition facility? (I’m fine with doing extra work in the system definitions, but I’m not sure I’d be fine with adding extra headers to the lisp files.)

...

*read-default-float-format* lets you choose between different representations for such generic libraries. The scoping rules for dynamic variables in Common Lisp give you two choices: Either you tell a particular library that it chooses a particular representation for its generically chosen representations - you do this by dynamic binding. Or you tell a set of libraries (all that you are loading from now on) that they should all choose the same representation for their generically chosen representations - you do this by side effect.

That's not how it works, unless you include a bit for *rdff* in the name of the fasl cache directory — and since the planning is done based on pathnames before the compilation happens, that should still be *rdff* at the beginning of compilation. Otherwise, the build is not deterministic, and two different toplevel programs will poison each other's builds.

…not even if you :force t?

...

You also have the option to fork a particular Common Lisp implementation

So do you. That's a non-argument that assumes you're the dictator of CL.

Huh?!?

...

The fact that the proposed changes are discussed so actively is a clear sign that people care about these things.

And the one way to know whether they'll be discussed is to open the discussion indeed.

I agree.

...

[1] I already encountered libraries that don’t load with current ASDF versions anymore, although they are defined as ASDF libraries, which is a shame!

Sorry, you don't get to complain unless (1) you issue a bug report, and (2) the bug isn't fixed by whichever party actually has a bug, whether it's the library or ASDF.

I’m not complaining, I’m just mentioning.

Pascal

-- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

On 29 Mar 2014, at 19:47, Faré fahree@gmail.com wrote:

...

...

(1) guaranteeing a value of *read-default-float-format* and other syntax variables when compiling a library. I still argue that (1) is essential for build determinism, and enabling users to change syntax at the REPL.

Are you also considering the following use case?

• Assume I’m developing a library for a specific Common Lisp implementation. Let’s choose LispWorks as an example, although it could be any other. (I’m just choosing LispWorks as an example because I happen to use it myself most often.) LispWorks comes with its own system definition facility, and with its own default values for pre-defined global variables in the CL package that may or may not be the same as those of other CL implementations. Since I’m also maintaining portable libraries, I prefer to use ASDF even for LispWorks-only libraries, so I don’t have to change tools on a regular basis. However, I would also like to make sure that LispWorks users can use the LispWorks system definition facility for my library as an alternative. Will it be possible to use ASDF without it messing with the default values for pre-defined global variables, so that I have a guarantee that I get the same result no matter whether I use ASDF or some other system definition facility? (I’m fine with doing extra work in the system definitions, but I’m not sure I’d be fine with adding extra headers to the lisp files.)

I admit I have only limited knowledge of LispWorks's defsystem — but that it made me appreciate Dan Barlow's design better (though not his implementation). Writing LW defsystem rules to process dependencies correctly is *possible*, but by no means trivial. I don't know what guarantees the LW defsystem does or does not provide in terms of variable bindings — if you know it, it would be great of you to explain what they are (or just link to a page that does, if there is one). Indeed, if these assumptions differ from those of ASDF today or tomorrow, it would be nice to have a simple way to declare around wrappers that provide the alternative assumptions.

…same for mk-defsystem, same for Allegro’s system definition facility, and so on. Or maybe, alternatively, such things don’t belong in a system definition facility, but should be covered by some other tool?

...

...

That's not how it works, unless you include a bit for *rdff* in the name of the fasl cache directory — and since the planning is done based on pathnames before the compilation happens, that should still be *rdff* at the beginning of compilation. Otherwise, the build is not deterministic, and two different toplevel programs will poison each other's builds.

…not even if you :force t?

If you make :force t the default, you lose incrementality, and fast startup time for end-user scripts. If you say "things are unsafe by default", you lose modularity and you make it impossible to distribute scripts to end users. Either way, if you don't have a deterministic build *by default*, easy deployment of scripts to end-users is not possible anymore.

I understand your desire for deterministic builds. I don’t understand your desire for deterministic builds being the default.

...

...

...

You also have the option to fork a particular Common Lisp implementation

So do you. That's a non-argument that assumes you're the dictator of CL.

Huh?!?

The common "love it or leave it" slogan is assuming the utterer owns the domain at stake. Otherwise, why should YOU not be the one to leave when we disagree what "love it" means?

I believe you are reading too much into what I wrote. I prefer the defaults of the CL implementation of my choice not to be overridden by some tool whose primary purpose is something unrelated to those defaults. That’s all.

Pascal

-- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

On Sat, 2014-03-29 at 19:59 +0100, Pascal Costanza wrote: [...]

...

...

...

That's not how it works, unless you include a bit for *rdff* in the name of the fasl cache directory — and since the planning is done based on pathnames before the compilation happens, that should still be *rdff* at the beginning of compilation. Otherwise, the build is not deterministic, and two different toplevel programs will poison each other's builds.

…not even if you :force t?

If you make :force t the default, you lose incrementality, and fast startup time for end-user scripts. If you say "things are unsafe by default", you lose modularity and you make it impossible to distribute scripts to end users. Either way, if you don't have a deterministic build *by default*, easy deployment of scripts to end-users is not possible anymore.

I understand your desire for deterministic builds. I don’t understand your desire for deterministic builds being the default.

Utterances like this is what makes "academic" an insult in certain circles.

You’re quoting me out of context. If deterministic by default had no cost associated with it, it would clearly be the desirable choice. But it has a cost associated with it, so this is less clear.

am i right to assume here that the costs we are talking about is merely adding markers to those few already existing defsystem's that depend on leakage, or depend on non-portable (read: random) defaults?

because that "it has a cost associated with it" above sounds like some serious pain...

If I failed to convince you that this is the case, and that other people may have other preferences, then I apologize.

reading from this side of the screen your preference seems to be to have change in what you like (new goodies coming from qicklisp), but not in what you don't like (the primary build infrastructure of the entire "community" maturing into something useful).

On Thu, Mar 27, 2014 at 1:06 PM, Faré fahree@gmail.com wrote:

On Thu, Mar 27, 2014 at 12:30 PM, Robert Goldman rpgoldman@sift.net wrote:

...

Zach Beane wrote:

...

Faré fahree@gmail.com writes:

...

femlisp raises an interesting issue: it has (setq *READ-DEFAULT-FLOAT-FORMAT* 'double-float) in setup.lisp

Gendl also sets *read-default-float-format* to 'double-float, and expects it to be set that way for compiling any Gendl/GDL application code (all the internal operations for points, vectors, transforms, etc are expected to deal in double-floats).

That is the only non-ANSI-compliant thing in Gendl, as far as I know.

If there is a reasonable way to back out of this to make Gendl 100% ANSI compliant, I'd be interested to look into it. But the current situation is that users expect double-float to be the default when working with Gendl.

27.03.2014, 18:44, "Robert P. Goldman" rpgoldman@sift.info:

While I am very grateful to cl-test-grid, we need to be conscious of its limitations.

If anything, I personally do not insist on any changes to be done in ASDF. But if developers want to test whether the libraries built OK on my system previously remain OK after some change, I don't mind to test.

Robert asked to postpone the discussion, so I didn't speak. But Robert, how about we continue the discusstion, and you can ignore it? When you finish current tasks and are ready to consider this you can ask for summary and then decide, Maybe right solution will evolve by that time, so you won't need to spend your energy on analysis.

I think maybe some useful services are possible in regard to isolation from readtable and other global state modification. But not sure what exactly.

BTW, I haven't experienced problems like that, so it's not a critical feature for me.

If I am a library author I write code from assumption that standard CL syntax is used when the library is built. If my user changes readtable and breaks the library - it's his problem, he created this problem and he is able to fix it.

If my dependency breaks readtable for me, it is an inconvenient dependency, I will try to find a better one.

If my dependency breaks readtable for my user - that's unfortunate. The user need somehow to restore standard environment after loading me. (Or consider me a bad dependency and try to find another one). The user may somehow instruct ASDF: "when the application is built, use standard CL syntax and isolate the application from any readtable, etc, modifications done in REPL or by the app dependencies." Another possible request: "Isolate the application, and every app dependency from each other" - this allows the application to use dependencies otherwise interfering via readtable.

As a library author I would try to not alter global readtable from my library, but provide user with a function to enable my reader macros.

If ASDF wants to prevent the undesirable interferences, maybe it's enough/most important to develop and recommend an easy way to keep your readtable modifications scoped to your system? What would be the recommended way to do so? ASDF around-compile hooks?

In CL we don't want to totally isolate systems from each other's global modifications - each system creates some packages, defines functions used by other systems. So the systems interact via global modifications... Another question is when global modifications are done because there is no easy way to keep them local.

It's a complex area. I think it will help if we step back and formulate clearly what problem we are solving.

Best regards, - Anton

Thanks a lot, Anton!

Well, here's a valid argument in favor of 'single-float as the default: making 'double-float the default breaks 22 systems in Quicklisp (a bit fewer when you consider that some systems only depended on a broken one, but still), when making it 'single-float only broke 4 of them, all of which I fixed yesterday. If the default had to be decided right now, that would indeed make it a strong case for 'single-float. Stronger to me than any I've seen so far, and one capable of swaying my opinion indeed.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Hopefully, they ban not just gay marriage, but straight marriage too, and at last we have separation of state and marriage.

On Fri, Mar 28, 2014 at 11:12 AM, Anton Vodonosov avodonosov@yandex.ru wrote:

27.03.2014, 01:21, "Faré" fahree@gmail.com:

...

Can you re-run a full test with what I just pushed in branch syntax-control?

Fare, here are the results for the syntax-control, commit d32afa0c: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-40.html

Looks like several libraries rely on single-float.

Best regards,

• Anton