I haven't fully grokked the code changes in the syntax-control branch yet. However, I have read over the design notes in TODO, and pulled them out into a separate file, which I am attaching here. I have annotated with a number of open questions, and added a bullet point to document rationale, which is currently mostly a place-holder for a statement.
WRT my concerns about the syntax-control modifications:
I have done some rooting around and have found a few libraries that are intended to set up a system for interactive use with a special readtable. These are things like expert system shells, etc.
So at some point near the end of their system load, they will set *READTABLE* prior to finishing the load and dropping out for the user to deal with them.
I can see that someone might not like this -- indeed, I'd rather there was a (START-FOO-SHELL) command instead. On the other hand, note that these are not really LIBRARIES; these are SYSTEMS. These are not intended to be slurped into something else: the programmer expects that his/her system will load, set up things for an interactive user, and then hand back control to the user through the REPL.
Isn't the syntax control approach fundamentally incompatible with this?
I don't think this is an especially tasty thing to do, but I'm also not convinced that it's ASDF's job to break arbitrary CL anti-patterns. Changing this behavior is likely to give very confusing bugs that won't be obvious to the poor user.
On the other hand, I see files where there's something that sets the READTABLE obviously where what's intended is to simply set the readtable for the compiling and loading of this one file, and leaking that change is wrong.
Note that a programmer who wants to be clean about this, in the current world, can set the readtable at the top of a file, and restore it at the bottom. If we take away the current behavior, the programmer who wants to set things up for the user is just completely out of luck. Or, maybe not: the very sophisticated ASDF user could add a PERFORM method on LOAD-OP on the interactive system. I think. I haven't grokked what's going on in syntax-control well enough.
I am in a quandary about this. IMO the problem is a hole in the CL spec, which doesn't provide a good way to distinguish between two different intended uses for readtables.
Before merging the syntax-control mods, I need to be convinced that it's appropriate for the ASDF community to force a solution to this problem on the CL community, instead of leaving the CL community to pick its way through the issue as it has to date.
Dear Robert,
On Sun, Jun 1, 2014 at 6:25 PM, Robert P. Goldman rpgoldman@sift.info wrote:
I haven't fully grokked the code changes in the syntax-control branch yet. However, I have read over the design notes in TODO, and pulled them out into a separate file, which I am attaching here. I have annotated with a number of open questions, and added a bullet point to document rationale, which is currently mostly a place-holder for a statement.
OK, I have updated said document with FRR: replies to your RPG: questions.
WRT my concerns about the syntax-control modifications:
I have done some rooting around and have found a few libraries that are intended to set up a system for interactive use with a special readtable. These are things like expert system shells, etc.
So at some point near the end of their system load, they will set *READTABLE* prior to finishing the load and dropping out for the user to deal with them.
I can see that someone might not like this -- indeed, I'd rather there was a (START-FOO-SHELL) command instead. On the other hand, note that these are not really LIBRARIES; these are SYSTEMS. These are not intended to be slurped into something else: the programmer expects that his/her system will load, set up things for an interactive user, and then hand back control to the user through the REPL.
Isn't the syntax control approach fundamentally incompatible with this?
My point is precisely that the *LACK OF* syntax control approach is fundamentally incompatible with this.
If in such an expert system, you directly or indirectly call ASDF or compile without ASDF (e.g. using SLIME), then your custom readtable with confuse the compilation and seriously screw up both your current image and your fasl cache — unless you use an ASDF with the syntax-control feature (and SLIME also uses it, possibly via ASDF).
I don't think this is an especially tasty thing to do, but I'm also not convinced that it's ASDF's job to break arbitrary CL anti-patterns. Changing this behavior is likely to give very confusing bugs that won't be obvious to the poor user.
In this case, we'd be enabling this behavior, which isn't an antipattern, but the entire point of readtables. Even without an expert system, I *want* to be able to safely use e.g. fare-quasiquote at the REPL, if only because I debug systems that use it internally and so want it enabled while I interactively explore them.
On the other hand, I see files where there's something that sets the READTABLE obviously where what's intended is to simply set the readtable for the compiling and loading of this one file, and leaking that change is wrong.
If it's setting the variable, the CLHS mandate on LOAD and COMPILE-FILE to bind *READTABLE* prevent any leak here though there may still be leak at fasl loading time if the effect is not in a (eval-when (:compile-toplevel :execute) ...) and you're either using a concatenated fasl, or your implementation non-conformantly fails to bind *READTABLE* around loading a fasl (vs a source file).
Note that a programmer who wants to be clean about this, in the current world, can set the readtable at the top of a file, and restore it at the bottom. If we take away the current behavior, the programmer who wants to set things up for the user is just completely out of luck. Or, maybe not: the very sophisticated ASDF user could add a PERFORM method on LOAD-OP on the interactive system. I think. I haven't grokked what's going on in syntax-control well enough.
In the syntax-control branch, such a crazy perform method would be ineffective. In the current master branch, it would work, but frankly be crazy.
I am in a quandary about this. IMO the problem is a hole in the CL spec, which doesn't provide a good way to distinguish between two different intended uses for readtables.
Well at least the spec mandates LOAD and COMPILE-FILE to bind *READTABLE* so it does intend a certain level of syntax-control. I suppose enough early lispers must have been bitten by that for the spec to mandate this.
Before merging the syntax-control mods, I need to be convinced that it's appropriate for the ASDF community to force a solution to this problem on the CL community, instead of leaving the CL community to pick its way through the issue as it has to date.
I believe it's the right thing to do. It shouldn't break any program, except maybe real crazy ones that probably don't exist in the wild. And it makes it safe to bind *READTABLE* at your REPL, which is something that is definitely intended by the CLHS and expected by the CL community.
It's one small corner case that wasn't important when ASDF used to be full of holes but that is necessary for it to be complete now that it has matured.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "Reality is that which, when you stop believing in it, doesn't go away". — Philip K. Dick
Dear Robert,
I saw you had another question in the syntax-control.txt document, about what breaks if you try to work around a readtable conflict by somehow forcing load order. I replied in the document.
You say that you had a system that was broken by my branch — did it indeed have a readtable conflict while somehow changing the readtable e.g. in a .asd file itself?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Personal dishonesty is not needed to produce a dishonest business plan or research proposal. Wishful thinking suffices. — John McCarthy
Faré wrote:
Dear Robert,
I saw you had another question in the syntax-control.txt document, about what breaks if you try to work around a readtable conflict by somehow forcing load order. I replied in the document.
You say that you had a system that was broken by my branch — did it indeed have a readtable conflict while somehow changing the readtable e.g. in a .asd file itself?
My large system is broken, but I have not been able to diagnose the problem. It is turning up *way* downstream. I get a type error where a class is not getting the right METACLASS.
I suspect it has to do with the fact that we have an ASDF extension that binds *READTABLE* in a PERFORM :AROUND method, but I have not been able to determine whether this is, in fact, the problem.
I can't imagine how anyone would guess that this error is a readtable fail, much less that it was caused by a modification to ASDF. This problem will be even worse if the modification to ASDF is quietly given to the poor programmer through an update to his or her CL implementation.
I pretty much *know* this is a readtable fail and I'm not having an easy time figuring out what went wrong.
This is why I'm not enthusiastic about the syntax-control branch. It's fail-obscure, and we can't count on programmers inferring that their code has suddenly gone pear-shaped because of a modification to ASDF, especially if it wasn't a modification to ASDF that they installed themselves.
Let's say one day I get a new SBCL or ACL, and all of a sudden I get an error like this. I'm not an ASDF hacker. I don't even know that the new CL version has updated the bundled ASDF. How would I begin to figure out what went wrong?
Until I have an answer to that question, I cannot merge the syntax-control code.
Best, r
On Fri, Jun 6, 2014 at 5:04 PM, Robert P. Goldman rpgoldman@sift.info wrote:
My large system is broken, but I have not been able to diagnose the problem. It is turning up *way* downstream. I get a type error where a class is not getting the right METACLASS.
Is it possible to look at that system? With an NDA?
I suspect it has to do with the fact that we have an ASDF extension that binds *READTABLE* in a PERFORM :AROUND method, but I have not been able to determine whether this is, in fact, the problem.
This kind of method should actually work cleanly.
I can't imagine how anyone would guess that this error is a readtable fail, much less that it was caused by a modification to ASDF. This problem will be even worse if the modification to ASDF is quietly given to the poor programmer through an update to his or her CL implementation.
I pretty much *know* this is a readtable fail and I'm not having an easy time figuring out what went wrong.
This is why I'm not enthusiastic about the syntax-control branch. It's fail-obscure, and we can't count on programmers inferring that their code has suddenly gone pear-shaped because of a modification to ASDF, especially if it wasn't a modification to ASDF that they installed themselves.
I understand that this is important and requires investigation.
Let's say one day I get a new SBCL or ACL, and all of a sudden I get an error like this. I'm not an ASDF hacker. I don't even know that the new CL version has updated the bundled ASDF. How would I begin to figure out what went wrong?
Until I have an answer to that question, I cannot merge the syntax-control code.
On the other hand, we have made many changes that were backward-incompatible in corner cases (e.g. the way component pathnames are handled, introducing output-translations, making utf-8 the default, removing the crazy :if-component-dep-fails, etc.) for the sake of sensible semantics and more robust behavior. Some of these changes were otherwise silent indeed, and we fixed them by a combination of proactively looking for misbehaving code, making announcements, and documenting the new behavior.
Being able to support most code that change the readtable is worth it.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Please leave the State in the toilets where you found it.
Faré wrote:
Being able to support most code that change the readtable is worth it.
I think part of the problem here is that you are smarter than most of the rest of us (at least me).
I'm still groping after a clear statement of exactly what class of bugs it is that we are proposing to fix. The existing design document is still too vague, proposing to fix
"uncontrolled, unintentional leaking of readtable side effects to systems that depend on such effects not happening,"
I think we need a clear example of a bug (ideally from a real system, although it's fine if that bug has since been fixed) that would be fixed by the change.
Also, we need an argument for why this must be fixed in ASDF.
Here's the devil's advocate argument:
"OK, you have a crummy library that leaks state unintentionally, and happens to do that in the readtable. Why is it ASDF's job to make that library suddenly work *without modification*? Why shouldn't we just tell the maintainer of that library to stop leaking state unintentionally?"
E.g., if I had a library that was annoyingly causing the printer to display integers in hex, why wouldn't I just fix that in the library, instead of trying to hack ASDF to prevent it from happening? The GBBopen library, which makes a sort of expert system shell, was borking my environment in all sorts of ways, but I just stomped on it until it stopped doing that.
A further development of the devil's advocate position:
It's not hard to fix unintentional readtable leakage. You just make your library export a variable holding its readtable and a function that will set the readtable to the one you're exporting. You make sure you don't touch the default readtable.
On the flipside, if you have a library that is intentionally leaking readtable state, it's likely to be an intricately balanced, teetering and delicate edifice, because CL doesn't provide good tools to deal with readtables (as we have discussed, packages have names, so are easy to deal with; readtables don't and are correspondingly fussy). So if we go in there using ASDF as a blunt instrument, we are very likely to mess things up in complex and difficult to debug ways.
So it seems to me that we are making a complex extension to ASDF to fix problems that are easy for library maintainers to debug, and can have side effects that will be difficult to debug. That doesn't seem like a good cost/benefit tradeoff.
I'm willing to be persuaded, but I need an "elevator pitch" that addresses these issues head on. * What exactly are we fixing? and * Why is ASDF the right place to fix this? with the correlate * Isn't an ASDF fix more complicated than just making libraries that Do The Right Thing?
Again, I'm not dead set against this, but I just don't Get It yet.
thanks, r
On Sat, Jun 7, 2014 at 8:03 PM, Robert P. Goldman rpgoldman@sift.info wrote:
I'm still groping after a clear statement of exactly what class of bugs it is that we are proposing to fix. The existing design document is still too vague, proposing to fix
"uncontrolled, unintentional leaking of readtable side effects to systems that depend on such effects not happening,"
I think we need a clear example of a bug (ideally from a real system, although it's fine if that bug has since been fixed) that would be fixed by the change.
I probably should create test cases.
Scenario A: dealing with *readtable* being bound to a new value 1- setup the test environment 2- save current packages 3- load fare-quasiquote (or anything that has its own readtable) 4- while in-readtable fare-quasiquote, load-system :force t something that uses normal quote, and see it corrupt its fasl with fare-quasiquote annotations. 5- delete all new packages since 2 6- repeat 3, 4 — and see whether it breaks [used to]
Scenario B: dealing with the *readtable* object being modified 1- setup the test environment 2- load and compile x1, that define #^ 3- load and compile x2, that uses #^ as per x1 4- load and compile x3, that defines a different #^ 3- load and compile x4, that uses #^ as per x3 4- force recompilation of x2 — it will use the definition from x4 — oops 5- force recompilation of x1 and x4 — it will use the definition from x2 — oops again
Also, we need an argument for why this must be fixed in ASDF.
Because ASDF must be safe when called from a different readtable. And using ASDF safely in presence of
Here's the devil's advocate argument:
"OK, you have a crummy library that leaks state unintentionally, and happens to do that in the readtable. Why is it ASDF's job to make that library suddenly work *without modification*? Why shouldn't we just tell the maintainer of that library to stop leaking state unintentionally?"
It's not *that library*. It's all the innocent libraries that know nothing about readtables, and that will be compiled into calls to functions from systems they don't depend on, which will pollute the build.
E.g., if I had a library that was annoyingly causing the printer to display integers in hex, why wouldn't I just fix that in the library, instead of trying to hack ASDF to prevent it from happening? The GBBopen library, which makes a sort of expert system shell, was borking my environment in all sorts of ways, but I j
Because you can't. You can't control what readtable is used at the REPL. You don't control which of your dependencies will or won't be forced recompiled by some change.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org No one has the right to a position; everyone has the right to positions being well filled. — Ernest Renan
ust stomped on it until it
stopped doing that.
A further development of the devil's advocate position:
It's not hard to fix unintentional readtable leakage. You just make your library export a variable holding its readtable and a function that will set the readtable to the one you're exporting. You make sure you don't touch the default readtable.
On the flipside, if you have a library that is intentionally leaking readtable state, it's likely to be an intricately balanced, teetering and delicate edifice, because CL doesn't provide good tools to deal with readtables (as we have discussed, packages have names, so are easy to deal with; readtables don't and are correspondingly fussy). So if we go in there using ASDF as a blunt instrument, we are very likely to mess things up in complex and difficult to debug ways.
So it seems to me that we are making a complex extension to ASDF to fix problems that are easy for library maintainers to debug, and can have side effects that will be difficult to debug. That doesn't seem like a good cost/benefit tradeoff.
I'm willing to be persuaded, but I need an "elevator pitch" that addresses these issues head on.
- What exactly are we fixing? and
- Why is ASDF the right place to fix this? with the correlate
- Isn't an ASDF fix more complicated than just making libraries that Do
The Right Thing?
Again, I'm not dead set against this, but I just don't Get It yet.
thanks, r
Faré wrote:
On Sat, Jun 7, 2014 at 8:03 PM, Robert P. Goldman rpgoldman@sift.info wrote:
I'm still groping after a clear statement of exactly what class of bugs it is that we are proposing to fix. The existing design document is still too vague, proposing to fix
"uncontrolled, unintentional leaking of readtable side effects to systems that depend on such effects not happening,"
I think we need a clear example of a bug (ideally from a real system, although it's fine if that bug has since been fixed) that would be fixed by the change.
I probably should create test cases.
Scenario A: dealing with *readtable* being bound to a new value 1- setup the test environment 2- save current packages 3- load fare-quasiquote (or anything that has its own readtable) 4- while in-readtable fare-quasiquote, load-system :force t something that uses normal quote, and see it corrupt its fasl with fare-quasiquote annotations. 5- delete all new packages since 2 6- repeat 3, 4 — and see whether it breaks [used to]
OK, so the bug is not that systems leak the readtable *out*, it's that readtables leak *into* systems. I had not understood that.
So unlike *PACKAGE*, the current (REPL) value of *READTABLE* leaks into every system that you load. It seems to me that this is a deficiency in CL: if CL had an IN-READTABLE equivalent to IN-PACKAGE, we would not be having this discussion.
Scenario B: dealing with the *readtable* object being modified 1- setup the test environment 2- load and compile x1, that define #^ 3- load and compile x2, that uses #^ as per x1 4- load and compile x3, that defines a different #^ 3- load and compile x4, that uses #^ as per x3 4- force recompilation of x2 — it will use the definition from x4 — oops 5- force recompilation of x1 and x4 — it will use the definition from x2 — oops again
This is the one I had in mind, and this is the one I don't think is ASDF's problem. If x1 exported *X1-READTABLE* and ADD-X1-SYNTAX (this function adds X1 reader macros to an existing readtable passed as argument), then the Right Thing would simply be for X1 not to alter the global readtable, and for X2 and X4 to explicitly employ the readtables they want.
This problem does not seem worthy of fixing in ASDF.
Scenario A seems like a stronger argument, but I'm inclined to say "instead of limping along with a complex fix in ASDF, let's just push NAMED-READTABLES use and leave ASDF alone."
NAMED-READTABLES seems like a much better fix than complicating ASDF further, because it gets at the root of the problem, instead of band-aiding around it.
The counter-argument, I suppose, is that Scenario A can only be fixed if *EVERYONE* fixes their code to put IN-READTABLE at the top of their source files, just the way we have to put IN-PACKAGE at the head of every source file.
Also, we need an argument for why this must be fixed in ASDF.
Because ASDF must be safe when called from a different readtable. And using ASDF safely in presence of
Here's the devil's advocate argument:
"OK, you have a crummy library that leaks state unintentionally, and happens to do that in the readtable. Why is it ASDF's job to make that library suddenly work *without modification*? Why shouldn't we just tell the maintainer of that library to stop leaking state unintentionally?"
It's not *that library*. It's all the innocent libraries that know nothing about readtables, and that will be compiled into calls to functions from systems they don't depend on, which will pollute the build.
My question here came from thinking only about your Scenario B, and overlooking Scenario A.
I'm starting to reluctantly come around, but I still need to figure out how my system went up the spout when using syntax-control ASDF. If I can't answer on my own behalf, I can hardly claim to understand the patch well enough to apply it....
Best r
On Sun, Jun 8, 2014 at 12:09 PM, Robert P. Goldman rpgoldman@sift.info wrote:
Faré wrote:
On Sat, Jun 7, 2014 at 8:03 PM, Robert P. Goldman rpgoldman@sift.info wrote:
I'm still groping after a clear statement of exactly what class of bugs it is that we are proposing to fix. The existing design document is still too vague, proposing to fix
"uncontrolled, unintentional leaking of readtable side effects to systems that depend on such effects not happening,"
I think we need a clear example of a bug (ideally from a real system, although it's fine if that bug has since been fixed) that would be fixed by the change.
I probably should create test cases.
Scenario A: dealing with *readtable* being bound to a new value 1- setup the test environment 2- save current packages 3- load fare-quasiquote (or anything that has its own readtable) 4- while in-readtable fare-quasiquote, load-system :force t something that uses normal quote, and see it corrupt its fasl with fare-quasiquote annotations. 5- delete all new packages since 2 6- repeat 3, 4 — and see whether it breaks [used to]
OK, so the bug is not that systems leak the readtable *out*, it's that readtables leak *into* systems. I had not understood that.
So unlike *PACKAGE*, the current (REPL) value of *READTABLE* leaks into every system that you load. It seems to me that this is a deficiency in CL: if CL had an IN-READTABLE equivalent to IN-PACKAGE, we would not be having this discussion.
Oh, the outer *package* also leaks in, but people have the discipline to use defpackage and in-package, and to not shadow them away in incompatible ways.
The syntax-control branch ought to *also* set *package* to *shared-package* which should be cl-user for maximum backward compatibility, and asdf-user or something like that for maximum portability. Or maybe, like Stelian once suggested, some package that *only* has defpackage and in-package defined (maybe also uiop:define-package?), for maximum strictness.
Scenario B: dealing with the *readtable* object being modified 1- setup the test environment 2- load and compile x1, that define #^ 3- load and compile x2, that uses #^ as per x1 4- load and compile x3, that defines a different #^ 3- load and compile x4, that uses #^ as per x3 4- force recompilation of x2 — it will use the definition from x4 — oops 5- force recompilation of x1 and x4 — it will use the definition from x2 — oops again
This is the one I had in mind, and this is the one I don't think is ASDF's problem. If x1 exported *X1-READTABLE* and ADD-X1-SYNTAX (this function adds X1 reader macros to an existing readtable passed as argument), then the Right Thing would simply be for X1 not to alter the global readtable, and for X2 and X4 to explicitly employ the readtables they want.
This problem does not seem worthy of fixing in ASDF.
Indeed, and I'm not proposing that ASDF should do anything about it. I'm just explaining what the restriction *already is* for sane programs to never define or use conflicting readtable modifications. This restriction had to be documented (which I did with ASDF 3.1).
Scenario A seems like a stronger argument, but I'm inclined to say "instead of limping along with a complex fix in ASDF, let's just push NAMED-READTABLES use and leave ASDF alone."
NAMED-READTABLES seems like a much better fix than complicating ASDF further, because it gets at the root of the problem, instead of band-aiding around it.
The counter-argument, I suppose, is that Scenario A can only be fixed if *EVERYONE* fixes their code to put IN-READTABLE at the top of their source files, just the way we have to put IN-PACKAGE at the head of every source file.
Exactly. Unless you're demanding that every library should depend on named-readtables and use it, that doesn't work. And even then, you'd forbid any readtable that is incompatible enough with the standard one to interfere with the evaluation of the in-readtable form, e.g. a readtable for Python syntax, etc. — or even something as innocuous as fare-quasiquote, if a file uses ` before in-readtable.
This can't fly. And named-readtables itself can't depend on named-readtables. In the end, you'd have to move it into ASDF, which is more complex than the simple use of *shared-readtable*.
My question here came from thinking only about your Scenario B, and overlooking Scenario A.
I'm starting to reluctantly come around, but I still need to figure out how my system went up the spout when using syntax-control ASDF. If I can't answer on my own behalf, I can hardly claim to understand the patch well enough to apply it....
If I can help with this understanding, I'm at your service to inspect code of discuss behavior.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Guns & bullets don't kill people — blood loss and organ damage kills people.
-
Faré -
Robert P. Goldman