Pre-submission discussion: code walking
Hello. I want to try to do something about portability of universally present features, given that stndard revisions are extremely unlikely, and CDR process seems to be the best way to obtain a stable ID for some version of description. As automatic code processing is a large part of Common Lisp identity as it is presented, and there is a small and easy to define chunk of functionality that is missing, I am preparing a CDR on portable code walking support. Before I formally submit a CDR proposal, I want to try to get some feedback. The current version of the writeup I have is at: https://gitlab.common-lisp.net/mraskin/cdr-walkability/blob/master/walkabili... Please comment on that proposal, here or via issues or maybe even merge requests. I will announce the link to the discussion in the ELS mailing list (as ELS conference seems to be the closest approximation to community discussion we have). I considered including Trucler API as an option, but currently I doubt I am able to do it well: mixing a coherent API with multiple optional additions that are useful even as single functions is a harder task. I want to start somewhere. Separately, if someone has some proposals on how to make CDR process used more — or matter more, please share. Probably on cdr-devel https://mailman.common-lisp.net/listinfo/cdr-devel Thank you, goodbye Michael Raskin
Regarding the feedback, here are some notes I had when I've started writing a test suite before starting implementing cltl2 in ECL, maybe you'll find some of them useful. I plan to tackle this problem again after upcoming 16.2.0 release (with some refactor of the compiler interfces). I'll definetely look into the walkability document before doing so. http://ix.io/1Hbo On that note: I try to implement some clarifications in ECL. What would be useful is to have all extensions which can't be implemented as a user library in a common package (i.e CDR vs CDR-USER). I think this is not covered in any CDR (and having a feature that CDR-X is implemented doesn't give me much if I don't know in which package should I seek the new symbol). Best regards, Daniel On Thu, 2019-04-25 at 16:47 +0200, Michael Raskin wrote:
Hello. I want to try to do something about portability of universally present features, given that stndard revisions are extremely unlikely, and CDR process seems to be the best way to obtain a stable ID for some version of description.
As automatic code processing is a large part of Common Lisp identity as it is presented, and there is a small and easy to define chunk of functionality that is missing, I am preparing a CDR on portable code walking support.
Before I formally submit a CDR proposal, I want to try to get some feedback. The current version of the writeup I have is at: https://gitlab.common-lisp.net/mraskin/cdr-walkability/blob/master/walkabili...
Please comment on that proposal, here or via issues or maybe even merge requests. I will announce the link to the discussion in the ELS mailing list (as ELS conference seems to be the closest approximation to community discussion we have).
I considered including Trucler API as an option, but currently I doubt I am able to do it well: mixing a coherent API with multiple optional additions that are useful even as single functions is a harder task. I want to start somewhere.
Separately, if someone has some proposals on how to make CDR process used more — or matter more, please share. Probably on cdr- devel https://mailman.common-lisp.net/listinfo/cdr-devel
Thank you, goodbye Michael Raskin
Regarding the feedback, here are some notes I had when I've started writing a test suite before starting implementing cltl2 in ECL, maybe you'll find some of them useful. I plan to tackle this problem again after upcoming 16.2.0 release (with some refactor of the compiler interfces). I'll definetely look into the walkability document before doing so.
Hm, interesting. Indeed, I should include at least optional support for declarations. I have a suspicion that SBCL environments have been refactored in the last couple of years (I vaguely remember getting some errors when trying to refer to the environment value after the dynamic extent of the macro function, but I cannot reproduce the problem with fresh SBCL). Maybe in the process some of the bugs you describe in the tests have also been fixed.
On that note: I try to implement some clarifications in ECL. What would be useful is to have all extensions which can't be implemented as a user library in a common package (i.e CDR vs CDR-USER). I think this is not covered in any CDR (and having a feature that CDR-X is implemented doesn't give me much if I don't know in which package should I seek the new symbol).
My plan for walkability is: compliance with CDR-NN requires having a package with name CDR-NN that has all the symbols defined in CDR-NN. My best-case plan includes codifying this approach, and making CDRs for other things from the same point of view. Thank you!
My plan for walkability is: compliance with CDR-NN requires having a package with name CDR-NN that has all the symbols defined in CDR- NN.
IMHO if CDR's are truly meant as a process to augument standarization then all that should be in a single package (with an additional "CDR- USER" package which uses CL and CDR and is not locked). That will ensure that there are no symbol conflicts and it is indeed a possible candidate for inclusion a "base" package. Also too many packages is a tedious disease of many systems.
My plan for walkability is: compliance with CDR-NN requires having a package with name CDR-NN that has all the symbols defined in CDR- NN.
IMHO if CDR's are truly meant as a process to augument standarization then all that should be in a single package (with an additional "CDR- USER" package which uses CL and CDR and is not locked). That will ensure that there are no symbol conflicts and it is indeed a possible candidate for inclusion a "base" package. Also too many packages is a tedious disease of many systems.
I agree with the goal of eventually having a usable CDR/CDR-USER package. I don't think this should be defined by the next coming CDR. Currently the CDR process is defined in a way that is best suited for defining small isolated things. I agree it would be useful to define a decision-making process that could be used for filling a single extension namespace in a consistent and widely respected way. I think defining such a process requires a discussion with buy-in from multiple implementations, and I hope to revive CDR as a recognised discussion place, but my plan starts with discussing something concrete — starting with a meta-discussion about rules has its own failure modes. So I am not ready to start defining a CDR package with the (coordination) tools I see, but I think there is hope to improve the situation. A part of the story is that to get the ball rolling I want walkability extensions to be as cheap as possible to implement. So I want to give the implementations some choice in implementing the CDR; which means that environment handling support is not directly for users as much as for building a portable library on top of it. Agreeing on a single API is more risk and more negotiations than analyzing the existing APIs and defining various sufficient subsets that are already sufficient, as well as identifying the options for improvements when more is available. User APIs on top of that can evolve as Quicklisp-installable libraries before we need to agree on the precise best approach. In other portability-sensitive areas, I think CFFI and Bordeaux-Threads will stay but it is good to have a defined foundation that a new implementation or a fork can provide and immediately get full support.
A part of the story is that to get the ball rolling I want walkability extensions to be as cheap as possible to implement. So I want to give the implementations some choice in implementing the CDR; which means that environment handling support is not directly for users as much as for building a portable library on top of it. Agreeing on a single API is more risk and more negotiations than analyzing the existing APIs and defining various sufficient subsets that are already sufficient, as well as identifying the options for improvements when more is available. User APIs on top of that can evolve as Quicklisp-installable libraries before we need to agree on the precise best approach.
I think that there should be a single spec to avoid incompatible coverage. I agree that having it relatively low-level is fine. I think that a test suite covering various corner cases would be a very good initial step. We could include it in ansi-test library in ansi-beyond suite.
In other portability-sensitive areas, I think CFFI and Bordeaux- Threads will stay but it is good to have a defined foundation that a new implementation or a fork can provide and immediately get full support.
CFFI in ECL's case is baked by ECL's UFFI implementation (ECL implements UFFI protocol and said protocol is fairly simple). Daniel
A part of the story is that to get the ball rolling I want walkability extensions to be as cheap as possible to implement. So I want to give the implementations some choice in implementing the CDR; which means that environment handling support is not directly for users as much as for building a portable library on top of it. Agreeing on a single API is more risk and more negotiations than analyzing the existing APIs and defining various sufficient subsets that are already sufficient, as well as identifying the options for improvements when more is available. User APIs on top of that can evolve as Quicklisp-installable libraries before we need to agree on the precise best approach.
I think that there should be a single spec to avoid incompatible coverage. I agree that having it relatively low-level is fine. I think that a test suite covering various corner cases would be a very good initial step. We could include it in ansi-test library in ansi-beyond suite.
From the point of view of better chances of support from diverse implementations, I expect the suboptimal Javascript-style non-uniform support with easy feature-probing and «polyfill» libraries to be useful for solving chicken-and-egg problem: we need useful libraries to show something is worth implementing well, and we need working support to try writing the useful libraries.
On Apr 26, 2019, at 09:05 , Daniel Kochmański <daniel@turtleware.eu> wrote:
My plan for walkability is: compliance with CDR-NN requires having a package with name CDR-NN that has all the symbols defined in CDR- NN.
IMHO if CDR's are truly meant as a process to augument standarization then all that should be in a single package (with an additional "CDR- USER" package which uses CL and CDR and is not locked). That will ensure that there are no symbol conflicts and it is indeed a possible candidate for inclusion a "base" package. Also too many packages is a tedious disease of many systems.
This has been discussed before and it was not quite deemed a very useful approach, as it would mean that you would have single namespace where different implementations of a given CDR would be dumped, possibly causing unwanted misalignments. Using CDR-XX as a nickname, OTOH would signal a conflict as soon as two possible implementations of a given CDR would be loaded in an image. I don’t remember the details of the discussion but that is the gist of it. All in all the CDR process has been as light weight as possible and even so it has been difficult to have the implementations buy into it. Cheers -- Marco Antoniotti
Hi I looked at the proposal. I agree it would be useful, but I have two concerns. * The proposal requires the presence of a CDR-NN package. Such a proposal should be made separately. FTTB if CDR-14 is used, only the CDR-NN feature should be provided. I would be in favor of a new CDR stating that a CDR-XX nickname could be added to a package implementing a given feature. * The proposal as is may not be portably implemented by a third party without resorting to implementation support which may or may not be there. The problematic operators are WITH-PARENT-ENVIRONMENT and ENVIRONMENT-ENTRY-NAMES. Do you see any way to provide it in a simple (read: non SBCL) way? Finally, COPY-ENVIRONMENT may and LEXICAL-ENVIRONMENT also be problematic. cheers Marco Antoniotti On Apr 25, 2019, at 16:47 , Michael Raskin <38a938c2@rambler.ru<mailto:38a938c2@rambler.ru>> wrote: Hello. I want to try to do something about portability of universally present features, given that stndard revisions are extremely unlikely, and CDR process seems to be the best way to obtain a stable ID for some version of description. As automatic code processing is a large part of Common Lisp identity as it is presented, and there is a small and easy to define chunk of functionality that is missing, I am preparing a CDR on portable code walking support. Before I formally submit a CDR proposal, I want to try to get some feedback. The current version of the writeup I have is at: https://gitlab.common-lisp.net/mraskin/cdr-walkability/blob/master/walkabili... Please comment on that proposal, here or via issues or maybe even merge requests. I will announce the link to the discussion in the ELS mailing list (as ELS conference seems to be the closest approximation to community discussion we have). I considered including Trucler API as an option, but currently I doubt I am able to do it well: mixing a coherent API with multiple optional additions that are useful even as single functions is a harder task. I want to start somewhere. Separately, if someone has some proposals on how to make CDR process used more — or matter more, please share. Probably on cdr-devel https://mailman.common-lisp.net/listinfo/cdr-devel Thank you, goodbye Michael Raskin -- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY Please check: http://cdac2019.lakecomoschool.org Please check: http://troncopackage.org Please check: https://www.frontiersin.org/research-topics/7394/network-bioscience Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
Thank you for your comments.
* The proposal requires the presence of a CDR-NN package. Such a proposal should be made separately. FTTB if CDR-14 is used, only the CDR-NN feature should be provided. I would be in favor of a new CDR stating that a CDR-XX nickname could be added to a package implementing a given feature.
I thought that maybe I don't want to create too many proposals at once. I see, I will split.
* The proposal as is may not be portably implemented by a third party without resorting to implementation support which may or may not be there. The problematic operators are WITH-PARENT-ENVIRONMENT and ENVIRONMENT-ENTRY-NAMES. Do you see any way to provide it in a simple (read: non SBCL) way?
Finally, COPY-ENVIRONMENT may and LEXICAL-ENVIRONMENT also be problematic.
Note that B-level and C-level support requires some properties of underlying implementation that do not currently hold for many popular ones. Also note that I want to provide common names for much more functionality than I want to require. It is enough to have WITH-AUGMENTED-ENVIRONMENT for B-level complieance, and COPY-ENVIRONMENT (or standalone AUGMENT-ENVIRONMENT) is not required even on the C-level. It is purely a «please tell me if you already provide this». Frankly, I think SLIME wants to do something that boils down to ability to define ENVIRONMENT-ENTRY-NAMES, so that shouldn't be too bad. LEXICAL-ENVIRONMENT has very weak requirements, I think on most implementations it can be defined by a third-party library. WITH-PARENT-ENVIRONMENT is definitely completely optional — on the other hand, if the implementation never tracks parent environments, it is OK to always have NIL there.
HI
On May 8, 2019, at 18:24 , Michael Raskin <38a938c2@rambler.ru> wrote:
Thank you for your comments.
* The proposal requires the presence of a CDR-NN package. Such a proposal should be made separately. FTTB if CDR-14 is used, only the CDR-NN feature should be provided. I would be in favor of a new CDR stating that a CDR-XX nickname could be added to a package implementing a given feature.
I thought that maybe I don't want to create too many proposals at once. I see, I will split.
* The proposal as is may not be portably implemented by a third party without resorting to implementation support which may or may not be there. The problematic operators are WITH-PARENT-ENVIRONMENT and ENVIRONMENT-ENTRY-NAMES. Do you see any way to provide it in a simple (read: non SBCL) way?
Finally, COPY-ENVIRONMENT may and LEXICAL-ENVIRONMENT also be problematic.
Note that B-level and C-level support requires some properties of underlying implementation that do not currently hold for many popular ones.
I may have misread the proposal, but, AFAIU, even what you call “Level A” cannot be easily provided without a commitment by the implementations.
Also note that I want to provide common names for much more functionality than I want to require. It is enough to have WITH-AUGMENTED-ENVIRONMENT for B-level complieance, and COPY-ENVIRONMENT (or standalone AUGMENT-ENVIRONMENT) is not required even on the C-level. It is purely a «please tell me if you already provide this».
Frankly, I think SLIME wants to do something that boils down to ability to define ENVIRONMENT-ENTRY-NAMES, so that shouldn't be too bad.
LEXICAL-ENVIRONMENT has very weak requirements, I think on most implementations it can be defined by a third-party library.
WITH-PARENT-ENVIRONMENT is definitely completely optional — on the other hand, if the implementation never tracks parent environments, it is OK to always have NIL there.
LW already has SYS:MAP-ENVIRONMENT which looks like ENVIRONMENT-ENTRY-NAMES, so, yes, it is definitively doable, like everything else, given a “sufficiently supportive implementation/vendor”. I just feel that CDRs should not make too many requests on the implementation. Apart from that, I would also note that listing names should not be sufficient for a CDR. Functions and macros signatures should also be described (*), as well as types. As for the specific content, I would also explicitly refer to the CLtL2 Environment Section 8.5. All the best Marco Antoniotti (*) To the chagrin of the folks at Franz (you-know-why 3:) 3:) 3:) ) -- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY Please check: http://cdac2019.lakecomoschool.org Please check: http://troncopackage.org Please check: https://www.frontiersin.org/research-topics/7394/network-bioscience Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
Finally, COPY-ENVIRONMENT may and LEXICAL-ENVIRONMENT also be problematic.
Note that B-level and C-level support requires some properties of underlying implementation that do not currently hold for many popular ones.
I may have misread the proposal, but, AFAIU, even what you call “Level A” cannot be easily provided without a commitment by the implementations.
OK, then I failed at wording. The intention is: implementation provides an arbitrary _subset_ of symbols described in the document, and symbols that it decides to provide shall match the description. The level-B adds sane expansion requirement, and level-B and level-C make sure at least something is provided. Unless an implementation reuses the names while changing the semantics, a third party can track what names correspond to what functionality and wrap everything that is available into CDR-NN functions (and ignore the rest). This matches exactly the intention of level-A compliance. Do you think that splitting a separate CDR for implementation packages and all-optional functionality would make this easier to understand? In that case what are the best words for optional/required functionality? Walkability level A defines OPTIONAL symbols to provide, but flexible hash-table interface defines MANDATORY functionality?
Also note that I want to provide common names for much more functionality than I want to require. It is enough to have WITH-AUGMENTED-ENVIRONMENT for B-level complieance, and COPY-ENVIRONMENT (or standalone AUGMENT-ENVIRONMENT) is not required even on the C-level. It is purely a «please tell me if you already provide this».
Frankly, I think SLIME wants to do something that boils down to ability to define ENVIRONMENT-ENTRY-NAMES, so that shouldn't be too bad.
LEXICAL-ENVIRONMENT has very weak requirements, I think on most implementations it can be defined by a third-party library.
WITH-PARENT-ENVIRONMENT is definitely completely optional — on the other hand, if the implementation never tracks parent environments, it is OK to always have NIL there.
LW already has SYS:MAP-ENVIRONMENT which looks like ENVIRONMENT-ENTRY-NAMES, so, yes, it is definitively doable, like everything else, given a “sufficiently supportive implementation/vendor”.
I just feel that CDRs should not make too many requests on the implementation.
Well, level-A would be a large compatibility enhancement and it expects only wrapping or aliasing the functionality already there.
Apart from that, I would also note that listing names should not be sufficient for a CDR. Functions and macros signatures should also be described (*), as well as types.
As for the specific content, I would also explicitly refer to the CLtL2 Environment Section 8.5.
I did provide signatures for some but apparently not all of the symbols added compared to CLtL2, and should probably also add signatures for CLtL2 names as well, thanks. PS. Is there anything I can do to help make the links in the bottom of the CDR project page lead to webpages of archives?
participants (4)
-
Antoniotti Marco -
Daniel Kochmański -
Marco Antoniotti -
Michael Raskin