In CLOS, instance remorphing considered useless in practice?
Hello Pros of Common Lisp, Here is my attempt at starting a significant (and hopefully useful) debate on a subject squarely about Common Lisp and its internals. My main stance here is to state that I have yet to see, in a significant application, any use of functions #'cl:make-instance-obsolete and #'cl:update-instance-for-redefined-class and of the underlying machinery that support the remorphing of instances following a class redefinition (through a subsequent cl:defclass most likely). I can state the same thing about #'cl:update-instance-for-different-class and #'cl:change-class. The machinery required of any CL implementation to properly support those functions (mentioned here above) is quite significant in its complexity and usually imposes a sizeable performance penalty on the speed of any instance slot access. A different choice of class redefinition semantics can lead to an implementation of CLOS with much reduced instance slot access overhead. The necessity of supporting instance remorphing should therefore be well motivated by very significant gains in application code performance (in speed and/or size and/or expressiveness power). Can anyone of you point me to some evidence of such application level usefulness? Is there any notorious usecase of this (instance remorphing) since its inclusion into ANSI CL? By the way, at the bottom of the entry about #'cl:change-class in the text of the CL standard, one can find a "Notes:" section that starts with this sentence: "The generic function change-class has several semantic difficulties." And this was written in the context of a single-threaded implementation, as ANSI-CL limited itself. I bet that almost all currently significant CL implementations are now multi-threaded, therefore the "several semantic difficulties" are greatly and gravely compounded. In my opinion, this makes proper motivation of usefulness all the more imperious. What do you think?
Updating instances when classes are redefined is critical to interactive development with the REPL. If you want to see why it's so critical, try working with Python and modifying a class definition. You have to shut down the interpreter and start over, which is a huge pain if your system has a big, complex state you want to preserve. On 6 Dec 2020, at 22:52, Jean-Claude Beaudoin wrote:
Hello Pros of Common Lisp,
Here is my attempt at starting a significant (and hopefully useful) debate on a subject squarely about Common Lisp and its internals.
My main stance here is to state that I have yet to see, in a significant application, any use of functions #'cl:make-instance-obsolete and #'cl:update-instance-for-redefined-class and of the underlying machinery that support the remorphing of instances following a class redefinition (through a subsequent cl:defclass most likely). I can state the same thing about #'cl:update-instance-for-different-class and #'cl:change-class.
The machinery required of any CL implementation to properly support those functions (mentioned here above) is quite significant in its complexity and usually imposes a sizeable performance penalty on the speed of any instance slot access.
A different choice of class redefinition semantics can lead to an implementation of CLOS with much reduced instance slot access overhead.
The necessity of supporting instance remorphing should therefore be well motivated by very significant gains in application code performance (in speed and/or size and/or expressiveness power).
Can anyone of you point me to some evidence of such application level usefulness? Is there any notorious usecase of this (instance remorphing) since its inclusion into ANSI CL?
By the way, at the bottom of the entry about #'cl:change-class in the text of the CL standard, one can find a "Notes:" section that starts with this sentence: "The generic function change-class has several semantic difficulties." And this was written in the context of a single-threaded implementation, as ANSI-CL limited itself. I bet that almost all currently significant CL implementations are now multi-threaded, therefore the "several semantic difficulties" are greatly and gravely compounded. In my opinion, this makes proper motivation of usefulness all the more imperious. What do you think?
I am interested to hear arguments in both directions. But you haven’t outlined the alternative, other than to state that they exist. What are these alternatives? I use these MOP functions indirectly whenever I perform a CHANGE-CLASS on objects, mimicking something akin to Smalltalk BECOME. And yes, many (most?) of my apps are mutlithreaded. - DM
On Dec 6, 2020, at 9:52 PM, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
Hello Pros of Common Lisp,
Here is my attempt at starting a significant (and hopefully useful) debate on a subject squarely about Common Lisp and its internals.
My main stance here is to state that I have yet to see, in a significant application, any use of functions #'cl:make-instance-obsolete and #'cl:update-instance-for-redefined-class and of the underlying machinery that support the remorphing of instances following a class redefinition (through a subsequent cl:defclass most likely). I can state the same thing about #'cl:update-instance-for-different-class and #'cl:change-class.
The machinery required of any CL implementation to properly support those functions (mentioned here above) is quite significant in its complexity and usually imposes a sizeable performance penalty on the speed of any instance slot access.
A different choice of class redefinition semantics can lead to an implementation of CLOS with much reduced instance slot access overhead.
The necessity of supporting instance remorphing should therefore be well motivated by very significant gains in application code performance (in speed and/or size and/or expressiveness power).
Can anyone of you point me to some evidence of such application level usefulness? Is there any notorious usecase of this (instance remorphing) since its inclusion into ANSI CL?
By the way, at the bottom of the entry about #'cl:change-class in the text of the CL standard, one can find a "Notes:" section that starts with this sentence: "The generic function change-class has several semantic difficulties." And this was written in the context of a single-threaded implementation, as ANSI-CL limited itself. I bet that almost all currently significant CL implementations are now multi-threaded, therefore the "several semantic difficulties" are greatly and gravely compounded. In my opinion, this makes proper motivation of usefulness all the more imperious. What do you think?
On Tue, Dec 8, 2020 at 3:21 PM dbm@refined-audiometrics.com < dbm@refined-audiometrics.com> wrote:
I am interested to hear arguments in both directions. But you haven’t outlined the alternative, other than to state that they exist. What are these alternatives?
The main purpose of my original post was to solicit more information from knowledgeable parties in order to do some kind of a survey of the situation on the subject. The subject of alternative semantics of class redefinition is a very closely related subject of the original topic so I dropped a hint of it in there as a teaser for a possible follow-up. Let me elaborate a bit on what I meant. One possible behavior of CL:DEFCLASS could be to simply create a new class instance each time it is invoked, no matter what happened before, and then install this brand new class on the symbol that denotes it as the sole thus "properly named" class from that point on. What would be wrong with that? The class that was the previous "properly named" class would not be so "properly" anymore and could merily go on to live its natural life on its own. Methods would have to be re-loaded since all the previous signatures would still reference the previous class. Is this re-loading not already happening to a large extent since the said class has now seen its internal structure altered?
I use these MOP functions indirectly whenever I perform a CHANGE-CLASS on objects, mimicking something akin to Smalltalk BECOME. And yes, many (most?) of my apps are mutlithreaded.
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization? My Smalltalk days are too far gone (20y ago) to remember what BECOME really did, sorry. Now, about the multi-threaded question, please imagine the following situation. One good morning, two (or more) background threads happen to be executing their business at the same priority level, each in their own private dynamic environment, probably sleeping on some IO most of the time. And then, the master of the REPL decides to redefine a class that has an instance that is part of each of the said thread dynamic environments or that can be easily reached from them. It then so happens that the said threads wake up and access that instance each at "almost the same time" and for the first time since the class redefinition. In that situation, what happens? Do we simply duck, pray and hope for the best? Or do we have some way to ensure the proper order of the world? Currently, all I see for the latter option are very heavy handed solutions like locks on every slot access, or global locks that destroy the purpose of threads. Am I simply wrong on this or what?
On 9 Dec 2020, at 07:42, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
On Tue, Dec 8, 2020 at 3:21 PM dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com> <dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com>> wrote: I am interested to hear arguments in both directions. But you haven’t outlined the alternative, other than to state that they exist. What are these alternatives?
The main purpose of my original post was to solicit more information from knowledgeable parties in order to do some kind of a survey of the situation on the subject. The subject of alternative semantics of class redefinition is a very closely related subject of the original topic so I dropped a hint of it in there as a teaser for a possible follow-up.
Let me elaborate a bit on what I meant. One possible behavior of CL:DEFCLASS could be to simply create a new class instance each time it is invoked, no matter what happened before, and then install this brand new class on the symbol that denotes it as the sole thus "properly named" class from that point on. What would be wrong with that? The class that was the previous "properly named" class would not be so "properly" anymore and could merily go on to live its natural life on its own. Methods would have to be re-loaded since all the previous signatures would still reference the previous class. Is this re-loading not already happening to a large extent since the said class has now seen its internal structure altered?
There is nothing wrong with that. In fact, you can already do that, thanks to (self find-class).
I use these MOP functions indirectly whenever I perform a CHANGE-CLASS on objects, mimicking something akin to Smalltalk BECOME. And yes, many (most?) of my apps are mutlithreaded.
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
My Smalltalk days are too far gone (20y ago) to remember what BECOME really did, sorry.
Now, about the multi-threaded question, please imagine the following situation. One good morning, two (or more) background threads happen to be executing their business at the same priority level, each in their own private dynamic environment, probably sleeping on some IO most of the time. And then, the master of the REPL decides to redefine a class that has an instance that is part of each of the said thread dynamic environments or that can be easily reached from them. It then so happens that the said threads wake up and access that instance each at "almost the same time" and for the first time since the class redefinition. In that situation, what happens? Do we simply duck, pray and hope for the best? Or do we have some way to ensure the proper order of the world? Currently, all I see for the latter option are very heavy handed solutions like locks on every slot access, or global locks that destroy the purpose of threads. Am I simply wrong on this or what?
Praying and hoping is indeed not a good option. I’d rather double-check what the CL implementation of choice says what the semantics are. For example: http://www.lispworks.com/documentation/lw71/LW/html/lw-142.htm <http://www.lispworks.com/documentation/lw71/LW/html/lw-142.htm> Pascal -- Pascal Costanza
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
I have used CHANGE-CLASS sparingly over the years. My first use was in a graphical DSP algorithm prototyping environment, and I believe it was related to graphical display objects. Almost a decade ago, so my memory is rusty. But most recently I have a class hierarchy of objects, where some more refined subclass instances can act one way through an initial mixin class on their first execution of a principal method, and then revert back to other superclass behavior thereafter. On CHANGE-CLASS, there is elision of slots in dropping back the the principal superclass structure. But every other slot remains intact. I cannot simply re-MAKE-INSTANCE on these objects as their identity is referenced in many places elsewhere. And what I need is a change in behavior, not identity. The only way to accomplish this change along the lines of re-making them, would require yet another layer of indirection. That might be interesting to contemplate. - DM
Just as an aside… that DSP prototyping environment still runs today as well as it did back in 2008 (perhaps a bit faster on todays CPUs). That is one thing about Common Lisp that I value highly. I have code that I wrote almost 30 years ago and have never needed to change, while it is still very much in use today. I contrast that experience with programming in Rust just a year ago, while the entire language was changing beneath me almost monthly. - DM
On Dec 9, 2020, at 7:55 AM, dbm@refined-audiometrics.com wrote:
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
I have used CHANGE-CLASS sparingly over the years. My first use was in a graphical DSP algorithm prototyping environment, and I believe it was related to graphical display objects. Almost a decade ago, so my memory is rusty.
But most recently I have a class hierarchy of objects, where some more refined subclass instances can act one way through an initial mixin class on their first execution of a principal method, and then revert back to other superclass behavior thereafter.
On CHANGE-CLASS, there is elision of slots in dropping back the the principal superclass structure. But every other slot remains intact.
I cannot simply re-MAKE-INSTANCE on these objects as their identity is referenced in many places elsewhere. And what I need is a change in behavior, not identity. The only way to accomplish this change along the lines of re-making them, would require yet another layer of indirection. That might be interesting to contemplate.
- DM
On Wed, Dec 9, 2020 at 10:25 AM David McClain <dbm@refined-audiometrics.com> wrote:
Just as an aside… that DSP prototyping environment still runs today as well as it did back in 2008 (perhaps a bit faster on todays CPUs). That is one thing about Common Lisp that I value highly. I have code that I wrote almost 30 years ago and have never needed to change, while it is still very much in use today.
That is indeed a very valuable property of Common Lisp.
I contrast that experience with programming in Rust just a year ago, while the entire language was changing beneath me almost monthly.
I currently have a case where the additional layer of indirection is what I'm doing. Essentially an interface with multiple internal representations. I did it that way because, while you can technically get information from the instance for any of the internal representations, it's not exact if you get it from a different representation. I did it that way for some of the same reasons you're contemplating it, mostly because of references elsewhere. Neil Gilmore raito@raito.com On 2020-12-09 07:55, dbm@refined-audiometrics.com wrote:
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
I have used CHANGE-CLASS sparingly over the years. My first use was in a graphical DSP algorithm prototyping environment, and I believe it was related to graphical display objects. Almost a decade ago, so my memory is rusty.
But most recently I have a class hierarchy of objects, where some more refined subclass instances can act one way through an initial mixin class on their first execution of a principal method, and then revert back to other superclass behavior thereafter.
On CHANGE-CLASS, there is elision of slots in dropping back the the principal superclass structure. But every other slot remains intact.
I cannot simply re-MAKE-INSTANCE on these objects as their identity is referenced in many places elsewhere. And what I need is a change in behavior, not identity. The only way to accomplish this change along the lines of re-making them, would require yet another layer of indirection. That might be interesting to contemplate.
- DM
The answer to this question is that I *don't* invoke `change-class` -- the environment does it for me. That's the beauty of the environment. Working with Python, where I have a machine learning model that I want to tweak without retraining has been very instructive for me in reminding me what I would lose without `change-class`. On 9 Dec 2020, at 8:55, dbm@refined-audiometrics.com wrote:
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
I once used update-instance-for-redefined-class in ASDF, when I was trying to make a smooth upgrade from ASDF 1 to ASDF 2. The problem I found is that proper support for class redefinition requires the cooperation not only of the underlying object system, but also of all the parts of the implementation. It also requires more deterministic guarantees on what code gets inlined and what code doesn't get inlined. In ASDF 3, I threw the towel, and decided I could never get the OTHER parts of the system to cooperate (aka ASDF extensions for me, but also all the extensions to all existing programs being redefined), so I stopped using update-instance-for-redefined-class and instead went for deleting/overwriting/resetting functions, generic functions, and, in extreme cases, packages. I would love to see languages with better support for redefinition. Like unison, or darklang. Unhappily, update-instance-for-redefined-class, while very cool, is *not enough*. But in relatively small systems that are long-running, with everyone cooperating, like they had in the 1980s, it was probably sufficient. And it is a marvelous tool. If you control the entire application and want images to survive code evolution. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Taxonomy is the death of science — A. N. Whitehead On Tue, Dec 8, 2020 at 2:48 PM Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
Hello Pros of Common Lisp,
Here is my attempt at starting a significant (and hopefully useful) debate on a subject squarely about Common Lisp and its internals.
My main stance here is to state that I have yet to see, in a significant application, any use of functions #'cl:make-instance-obsolete and #'cl:update-instance-for-redefined-class and of the underlying machinery that support the remorphing of instances following a class redefinition (through a subsequent cl:defclass most likely). I can state the same thing about #'cl:update-instance-for-different-class and #'cl:change-class.
The machinery required of any CL implementation to properly support those functions (mentioned here above) is quite significant in its complexity and usually imposes a sizeable performance penalty on the speed of any instance slot access.
A different choice of class redefinition semantics can lead to an implementation of CLOS with much reduced instance slot access overhead.
The necessity of supporting instance remorphing should therefore be well motivated by very significant gains in application code performance (in speed and/or size and/or expressiveness power).
Can anyone of you point me to some evidence of such application level usefulness? Is there any notorious usecase of this (instance remorphing) since its inclusion into ANSI CL?
By the way, at the bottom of the entry about #'cl:change-class in the text of the CL standard, one can find a "Notes:" section that starts with this sentence: "The generic function change-class has several semantic difficulties." And this was written in the context of a single-threaded implementation, as ANSI-CL limited itself. I bet that almost all currently significant CL implementations are now multi-threaded, therefore the "several semantic difficulties" are greatly and gravely compounded. In my opinion, this makes proper motivation of usefulness all the more imperious. What do you think?
Hello Faré, I had you and ASDF somewhat in mind when I wrote the original post. On Tue, Dec 8, 2020 at 3:23 PM Faré <fahree@gmail.com> wrote:
I once used update-instance-for-redefined-class in ASDF, when I was trying to make a smooth upgrade from ASDF 1 to ASDF 2. The problem I found is that proper support for class redefinition requires the cooperation not only of the underlying object system, but also of all the parts of the implementation. It also requires more deterministic guarantees on what code gets inlined and what code doesn't get inlined.
Inlining, what a can or worms!
In ASDF 3, I threw the towel, and decided I could never get the OTHER parts of the system to cooperate (aka ASDF extensions for me, but also all the extensions to all existing programs being redefined), so I stopped using update-instance-for-redefined-class and instead went for deleting/overwriting/resetting functions, generic functions, and, in extreme cases, packages.
I followed from somewhat afar your "acharnement" in that area, with some bewilderment I'd say.
I would love to see languages with better support for redefinition. Like unison, or darklang. Unhappily, update-instance-for-redefined-class, while very cool, is *not enough*.
My point is that it is already too much.
But in relatively small systems that are long-running, with everyone cooperating, like they had in the 1980s, it was probably sufficient. And it is a marvelous tool. If you control the entire application and want images to survive code evolution.
In the late 1980s, all the Lisp Machines I touched were running ZetaLisp with Flavors as OO extension. And we were dropping CLTL1 version of Common Lisp on top of them before our application code. So, no CLOS. That came later...
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Taxonomy is the death of science — A. N. Whitehead
BTW, I just received the latest book by Denis Robert titled "LARRY ET MOI, comment BLACKROCK nous aime, nous surveille et nous détruit" and I dropped it on my pile of books to read soon. Thought you'd be amused to know, somehow.
Hello there. I don't normally chime in on this list, but thought I could explain my usage of CHANGE-CLASS in a large Common Lisp project (~20k lines of code). Said application is a game engine, as anyone familiar with my work would probably have guessed as much (I have several, this being the latest one I have developed over the past 2 years). CLOS, and most notably CHANGE-CLASS is an integral part of the entire system. During a running game, there can exist multiple "entities" in the game world. They do nothing on their own however, until a "component" is attached to one. Entities and components are both represented as standard-object's. At any point in a game developer's code, they can create an entity, and either immediately or some other time later, attach a new component, or detach a previously attached component to/from an entity. In actuality, there is no "has-a" relationship between entities and components. Upon attaching or detaching of a component from an entity, what effectively happens is the entity's class is changed by means of CHANGE-CLASS. It is done so in a way that it's class precedence list is topologically sorted according to an ordering the game developer defines. For example, they could specify that COMPONENT-FOO should be before or after COMPONENT-BAR. At various points during the execution of each game frame, various hooks are called. For example, there are "on attach" and "on detach" hooks that are called when a component is attached or detached from an entity. There are also an "on render" hook that is fired if a particular component should be rendered to the screen, and many others. However, since these hooks are defined by the programmer specializing on a component class, an invocation of any one of these hooks would only affect the most applicable object. And this is why we topologically sort the class precedence list as mentioned above; instead of the user-definable hooks using standard method combination, they actually all use PROGN method combination. This means any time a hook is fired for an entity, it will affect all of the components it has attached (with a no-op method for those the programmer does not define). This whole system works beautifully for us, even though I probably didn't explain it very well. I personally use it for games with thousands of entities with various components each, that all have to be processed each frame, where 1 frame is 1/60 of a second. I am bottle-necked by CLOS somewhere around 2000 entities with a few components each for my personal game in progress, but I don't find this to be a problem for what I need currently.
Hi I used CHANGE-CLASS some time ago to ensure that I could read new data and reshape it. It was a much cleaner - and "built in" - solution than writing ad-hoc code which should eventually be "generalized". I understand people get worked up about things they feel may be in their way of "efficient code". But guys! The world I see from my perch is running on Python and R. At the risk of sounding obnoxious (I am getting old; I can afford it :) ) there are other trees to bark on in the CL forest: mostly at the outskirts. Not that many at the core. All the best Marco On Wed, Dec 9, 2020 at 9:10 AM Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:
Hello Faré,
I had you and ASDF somewhat in mind when I wrote the original post.
On Tue, Dec 8, 2020 at 3:23 PM Faré <fahree@gmail.com> wrote:
I once used update-instance-for-redefined-class in ASDF, when I was trying to make a smooth upgrade from ASDF 1 to ASDF 2. The problem I found is that proper support for class redefinition requires the cooperation not only of the underlying object system, but also of all the parts of the implementation. It also requires more deterministic guarantees on what code gets inlined and what code doesn't get inlined.
Inlining, what a can or worms!
In ASDF 3, I threw the towel, and decided I could never get the OTHER parts of the system to cooperate (aka ASDF extensions for me, but also all the extensions to all existing programs being redefined), so I stopped using update-instance-for-redefined-class and instead went for deleting/overwriting/resetting functions, generic functions, and, in extreme cases, packages.
I followed from somewhat afar your "acharnement" in that area, with some bewilderment I'd say.
I would love to see languages with better support for redefinition. Like unison, or darklang. Unhappily, update-instance-for-redefined-class, while very cool, is *not enough*.
My point is that it is already too much.
But in relatively small systems that are long-running, with everyone cooperating, like they had in the 1980s, it was probably sufficient. And it is a marvelous tool. If you control the entire application and want images to survive code evolution.
In the late 1980s, all the Lisp Machines I touched were running ZetaLisp with Flavors as OO extension. And we were dropping CLTL1 version of Common Lisp on top of them before our application code. So, no CLOS. That came later...
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Taxonomy is the death of science — A. N. Whitehead
BTW, I just received the latest book by Denis Robert titled "LARRY ET MOI, comment BLACKROCK nous aime, nous surveille et nous détruit" and I dropped it on my pile of books to read soon. Thought you'd be amused to know, somehow.
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
On Wed, Dec 9, 2020 at 3:55 AM Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Hi
I used CHANGE-CLASS some time ago to ensure that I could read new data and reshape it. It was a much cleaner - and "built in" - solution than writing ad-hoc code which should eventually be "generalized".
I understand people get worked up about things they feel may be in their way of "efficient code".
But guys! The world I see from my perch is running on Python and R.
No contest from here.
At the risk of sounding obnoxious (I am getting old; I can afford it :) ) there are other trees to bark on in the CL forest: mostly at the outskirts. Not that many at the core.
I'd be curious to see your laundry list if you care to share it. As far as priorities go, that is an other issue.
Am Mi., 9. Dez. 2020 um 09:10 Uhr schrieb Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com>:
I would love to see languages with better support for redefinition.
Like unison, or darklang. Unhappily, update-instance-for-redefined-class, while very cool, is *not enough*.
My point is that it is already too much.
"I don't want to silence a lively discussion, but" several people have made the point that they found CLOS's mechanisms to deal with class redefinition useful. I can add myself to the list: When I was writing a CLOS based database system, I used CHANGE-CLASS and the associated helper methods to support schema evaluation. It did not work perfectly for all cases, but it worked well enough and felt like the MOP API was made exactly for this kind of thing. Given the discussion until now, Jean-Claude, what is it that you're trying to accomplish with this debate? Are you up for proving that the MOP is "wrong"? You have repeatedly indicated that this would be your opinion. If so, are you trying to create a new version of the MOP that would not include class redefinition hooks? Or are you up for creating a new Lisp or a new Lisp object system that is inspired by the MOP but somehow different? I think the ship has sailed anyway. Common Lisp, CLOS and the MOP are what they are. I still think they're beautiful, but they are also a product of the era in which they were created. Time has not stood still, and 20 years have passed without CL having changed or adapted to any new developments in computing. Even though in many respects, Common Lisp has aged well, there are areas in which the language itself is in the way of adapting to the modern world. Multiple threads come to mind, as do cons cells and CLOS, too. What else is there? I'd be interested in reading opinions. I like using CL when I know that I don't need scalability and when I know that I'll be the only developer of the program that I'm writing. Under these circumstances, I enjoy it very much because it caters for this setting. -Hans
On Wed, Dec 9, 2020 at 4:36 AM Hans Hübner <hans.huebner@gmail.com> wrote:
Am Mi., 9. Dez. 2020 um 09:10 Uhr schrieb Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com>:
I would love to see languages with better support for redefinition.
Like unison, or darklang. Unhappily, update-instance-for-redefined-class, while very cool, is *not enough*.
My point is that it is already too much.
"I don't want to silence a lively discussion, but" several people have made the point that they found CLOS's mechanisms to deal with class redefinition useful. I can add myself to the list: When I was writing a CLOS based database system, I used CHANGE-CLASS and the associated helper methods to support schema evaluation. It did not work perfectly for all cases, but it worked well enough and felt like the MOP API was made exactly for this kind of thing.
Given the discussion until now, Jean-Claude, what is it that you're trying to accomplish with this debate?
I am curious to survey the state-of-the-experience-so-far on this issue.
Are you up for proving that the MOP is "wrong"?
No, not "wrong" on any "essential" point at least.
You have repeatedly indicated that this would be your opinion.
Had no awareness that I did so up to this point.
If so, are you trying to create a new version of the MOP that would not include class redefinition hooks?
No, I still consider AMOP to be part of the holy scriptures of Common Lisp.
Or are you up for creating a new Lisp or a new Lisp object system that is inspired by the MOP but somehow different?
I am not that ambitious. (And make it fly on the top of it, man! I would be out of my mind, wouldn't I?)
I think the ship has sailed anyway. Common Lisp, CLOS and the MOP are what they are. I still think they're beautiful, but they are also a product of the era in which they were created. Time has not stood still, and 20 years have passed without CL having changed or adapted to any new developments in computing. Even though in many respects, Common Lisp has aged well, there are areas in which the language itself is in the way of adapting to the modern world. Multiple threads come to mind, as do cons cells and CLOS, too. What else is there? I'd be interested in reading opinions.
I like using CL when I know that I don't need scalability and when I know that I'll be the only developer of the program that I'm writing. Under these circumstances, I enjoy it very much because it caters for this setting.
And I'd like it with scalability too, how about that? The "only developer" part I simply do not get. What is the deal with the solitary state here?
-Hans
Am Mi., 9. Dez. 2020 um 11:03 Uhr schrieb Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com>:
On Wed, Dec 9, 2020 at 4:36 AM Hans Hübner <hans.huebner@gmail.com> wrote:
I like using CL when I know that I don't need scalability and when I know that I'll be the only developer of the program that I'm writing. Under these circumstances, I enjoy it very much because it caters for this setting.
And I'd like it with scalability too, how about that? The "only developer" part I simply do not get. What is the deal with the solitary state here?
My personal conclusion, having used Lisp for many years, is that it caters very well for the individual. After all, REPL based, exploratory programming is all about the interaction of an individual with the machine. This nature of Lisp goes far, and then Lisp itself wants to be explored. As an individual programmer, Lisp provides me not only with a set of tooling that I know how to use, it also provides me with its own exploratory space: There is this set of Lisp features that I have not found a good use in the past, but that I would like to try out in the future. Now, this might not be that far from other large languages, yet due to the fact that Lisp carries a lot of baggage from older Lisp dialects, a lot of these seldom-used features would not have made it into a new language, had it not been to appease different parts of the language committee who had their own baggage to carry over into the new, unified Common Lisp dialect. As an example, I would like to point at symbol plists, which were integral to earlier LISPs lacking better, dictionary-like data structures. They sleep dormant in the spec, but rest assured: If Common Lisp is your community, you will eventually run into someone who insists that symbol plists are great and uses them all over. Looking at you, Ken. Now, this is all good and fine for the solitary experience of having a tool that you can both use to create solutions and that is in itself something that you can endlessly explore. It is less useful in the world of industrial software development, where maintainability of systems written by other people is one of the key properties that should be supported by the implementation language. While this usually makes languages boring and often causes features that "one" would "like" to to have be missing, it also is a property that makes one language successful and the other language less so. This is also where scalability comes into play, and that word certainly needs to be qualified before it can be discussed. If scalability simply describes the raw size of the problem that can be addressed with a tool, then there are limits to what a given language can do based on its inherent support for threads that execute in parallel. Parallelism is something that requires proper abstractions. Some say that these can be just on the library level, but my experience tells me a different story. I find it difficult to create robust parallel solutions in languages like Lisp, as the language itself assumes a single thread of control in how it makes variable bindings accessible and how information is passed between invocations of functions by reference. If scalability is about the number of people that one can bring into a team that develops a single system, Common Lisp is not a particular nice environment either. Due to its size, age and heritage, every program that is written in it basically subsets the language and, worse, often also extends it because the subset was found to be insufficient and extending the language is easy. The result is that unless a new person has already mastered a lot of Common Lisp, they will usually have a hard time finding their way around in a new system. This is not to say that the Common Lisp is the only language with that problem - It is a property of many old languages. I would still argue that CL is particularly affected because it was an amalgam of several languages to begin with, because it had very strict performance goals in the beginning and because the model of development that it always catered for is not that of a team, but that of a person interacting with a long-running program. -Hans
On Wed, Dec 9, 2020 at 7:13 AM Hans Hübner <hans.huebner@gmail.com> wrote:
Am Mi., 9. Dez. 2020 um 11:03 Uhr schrieb Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com>:
On Wed, Dec 9, 2020 at 4:36 AM Hans Hübner <hans.huebner@gmail.com> wrote:
I like using CL when I know that I don't need scalability and when I know that I'll be the only developer of the program that I'm writing. Under these circumstances, I enjoy it very much because it caters for this setting.
And I'd like it with scalability too, how about that? The "only developer" part I simply do not get. What is the deal with the solitary state here?
My personal conclusion, having used Lisp for many years, is that it caters very well for the individual. After all, REPL based, exploratory programming is all about the interaction of an individual with the machine. This nature of Lisp goes far, and then Lisp itself wants to be explored. As an individual programmer, Lisp provides me not only with a set of tooling that I know how to use, it also provides me with its own exploratory space: There is this set of Lisp features that I have not found a good use in the past, but that I would like to try out in the future. Now, this might not be that far from other large languages, yet due to the fact that Lisp carries a lot of baggage from older Lisp dialects, a lot of these seldom-used features would not have made it into a new language, had it not been to appease different parts of the language committee who had their own baggage to carry over into the new, unified Common Lisp dialect.
As an example, I would like to point at symbol plists, which were integral to earlier LISPs lacking better, dictionary-like data structures. They sleep dormant in the spec, but rest assured: If Common Lisp is your community, you will eventually run into someone who insists that symbol plists are great and uses them all over. Looking at you, Ken.
Now, this is all good and fine for the solitary experience of having a tool that you can both use to create solutions and that is in itself something that you can endlessly explore. It is less useful in the world of industrial software development, where maintainability of systems written by other people is one of the key properties that should be supported by the implementation language. While this usually makes languages boring and often causes features that "one" would "like" to to have be missing, it also is a property that makes one language successful and the other language less so.
This is also where scalability comes into play, and that word certainly needs to be qualified before it can be discussed. If scalability simply describes the raw size of the problem that can be addressed with a tool, then there are limits to what a given language can do based on its inherent support for threads that execute in parallel. Parallelism is something that requires proper abstractions. Some say that these can be just on the library level, but my experience tells me a different story. I find it difficult to create robust parallel solutions in languages like Lisp, as the language itself assumes a single thread of control in how it makes variable bindings accessible and how information is passed between invocations of functions by reference.
If scalability is about the number of people that one can bring into a team that develops a single system, Common Lisp is not a particular nice environment either. Due to its size, age and heritage, every program that is written in it basically subsets the language and, worse, often also extends it because the subset was found to be insufficient and extending the language is easy. The result is that unless a new person has already mastered a lot of Common Lisp, they will usually have a hard time finding their way around in a new system.
This is not to say that the Common Lisp is the only language with that problem - It is a property of many old languages. I would still argue that CL is particularly affected because it was an amalgam of several languages to begin with, because it had very strict performance goals in the beginning and because the model of development that it always catered for is not that of a team, but that of a person interacting with a long-running program.
-Hans
Much to think about, thank you Hans
Hello.
Am 09.12.2020 um 13:13 schrieb Hans Hübner <hans.huebner@gmail.com>:
Am Mi., 9. Dez. 2020 um 11:03 Uhr schrieb Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com>: On Wed, Dec 9, 2020 at 4:36 AM Hans Hübner <hans.huebner@gmail.com> wrote:
This is also where scalability comes into play, and that word certainly needs to be qualified before it can be discussed. If scalability simply describes the raw size of the problem that can be addressed with a tool, then there are limits to what a given language can do based on its inherent support for threads that execute in parallel. Parallelism is something that requires proper abstractions. Some say that these can be just on the library level, but my experience tells me a different story. I find it difficult to create robust parallel solutions in languages like Lisp, as the language itself assumes a single thread of control in how it makes variable bindings accessible and how information is passed between invocations of functions by reference.
I think parallelism is certainly possible in CL. The abstractions may not be given by the CL spec but library exist that have them. Like lparallel, and even „message passing“ frameworks like cl-gserver and others. The maturity of these may be a different thing, but they exist. From a runtime perspective CL uses OS threads in the same way as the JVM does. So whatever is possible in the JVM is possible in CL.
If scalability is about the number of people that one can bring into a team that develops a single system, Common Lisp is not a particular nice environment either. Due to its size, age and heritage, every program that is written in it basically subsets the language and, worse, often also extends it because the subset was found to be insufficient and extending the language is easy. The result is that unless a new person has already mastered a lot of Common Lisp, they will usually have a hard time finding their way around in a new system.
I wouldn’t see this as hindrance. Using a language in a company always comes with restrictions, rules and disciplines for which libraries to use, which style of code, etc. Cheers, Manfred
Hans Hübner <hans.huebner@gmail.com> writes:
Parallelism is something that requires proper abstractions. Some say that these can be just on the library level, but my experience tells me a different story. I find it difficult to create robust parallel solutions in languages like Lisp, as the language itself assumes a single thread of control in how it makes variable bindings accessible and how information is passed between invocations of functions by reference.
If "how information is passed between invocations of functions by reference" is the problem, then it sounds like you really do not want a Lisp, but Lisp Flavoured Erlang. You have to ask: is parallelism the problem, or concurrency? Good approaches to parallelism have already been worked out for Lisp a long time ago: Connection Machine Lisp, *Lisp, Paralations, NESL. What is not here now is an implementation of any of those for GPUs. Kelly Murray worked on a multi-processor Common Lisp (Top Level Common Lisp) in the late 1980s/early 1990s, and I think his opinion about Common Lisp is still true today: "This is one reason why I believed Common Lisp was good for multiprocessing, since you can use macros to create new higher-level parallel constructs, which is not possible with C/C++. Stuff like parallel-do's and parallel-maps, and sequence operations can be macros that expand into "low-level" code." -- Vladimir Sedach Software engineering services in Los Angeles https://oneofus.la
On Wed, Dec 9, 2020 at 4:36 AM Hans Hübner <hans.huebner@gmail.com> wrote:
Given the discussion until now, Jean-Claude, what is it that you're trying to accomplish with this debate? Are you up for proving that the MOP is "wrong"? You have repeatedly indicated that this would be your opinion. If so, are you trying to create a new version of the MOP that would not include class redefinition hooks? Or are you up for creating a new Lisp or a new Lisp object system that is inspired by the MOP but somehow different?
Sorry Hans but I just got a "come to Jesus" moment and therefore had to consult the scriptures directly. What do I find in the AMOP, #'cl:update-instance-for-redefined-class and #'cl:make-instances-obsolete are nowhere to be found in it! I see this as a sign. One finds #'cl:change-class and #'cl:update-instance-for-different-class inside AMOP but not the other two. I would then be tempted to drop almost half of my case after all.
Hi! It is used all the time while working on a live system (image). If you want raw performance, you can often substitute structs, if you can live with the downsides: all objects from a class must be thrown away on redefinition (which might be OK if you want to treat e. g. a production system as unchangeable except through re-deployment), you only get single inheritance (which is often not needed anyway), and most of the MOP goodness is gone, starting with that you can't have a different metaclass than structure-class. Also, because of the mentioned inflexibility, you have to dismiss several warnings if you do re-define a struct. On the other hand, structs have a lot of other, more low-level convenience tooling, e. g. automatic and portable serialization and deserialization. I _think_ that one might be able to create a new macro (let's say def-struct-class) that has exactly the same semantics as defstruct, but in development mode uses defclass underneath, while being an alias for defstruct in production mode. It might get a bit hairy to adhere to :type :list or :vector, though. The result is a bit limited, but just might be what you need. The folk at ITA also did quite some work on compile-time metaprogramming in order to get fast objects etc., so maybe they can chime in. Anyway, my stance is that the incredibly flexible MOP classes provide general semantic usefulness, but Common Lisp does offer more constrained options that trade general usefulness for speed. Yours aye Svante Jean-Claude Beaudoin writes:
Hello Pros of Common Lisp,
Here is my attempt at starting a significant (and hopefully useful) debate on a subject squarely about Common Lisp and its internals.
My main stance here is to state that I have yet to see, in a significant application, any use of functions #'cl:make-instance-obsolete and #'cl:update-instance-for-redefined-class and of the underlying machinery that support the remorphing of instances following a class redefinition (through a subsequent cl:defclass most likely). I can state the same thing about #'cl:update-instance-for-different-class and #'cl:change-class.
The machinery required of any CL implementation to properly support those functions (mentioned here above) is quite significant in its complexity and usually imposes a sizeable performance penalty on the speed of any instance slot access.
A different choice of class redefinition semantics can lead to an implementation of CLOS with much reduced instance slot access overhead.
The necessity of supporting instance remorphing should therefore be well motivated by very significant gains in application code performance (in speed and/or size and/or expressiveness power).
Can anyone of you point me to some evidence of such application level usefulness? Is there any notorious usecase of this (instance remorphing) since its inclusion into ANSI CL?
By the way, at the bottom of the entry about #'cl:change-class in the text of the CL standard, one can find a "Notes:" section that starts with this sentence: "The generic function change-class has several semantic difficulties." And this was written in the context of a single-threaded implementation, as ANSI-CL limited itself. I bet that almost all currently significant CL implementations are now multi-threaded, therefore the "several semantic difficulties" are greatly and gravely compounded. In my opinion, this makes proper motivation of usefulness all the more imperious. What do you think?
On Tue, Dec 8, 2020 at 4:17 PM Svante Carl v. Erichsen < svante.v.erichsen@web.de> wrote:
Hi!
It is used all the time while working on a live system (image).
If you want raw performance, you can often substitute structs, if you can live with the downsides: all objects from a class must be thrown away on redefinition (which might be OK if you want to treat e. g. a production system as unchangeable except through re-deployment), you only get single inheritance (which is often not needed anyway), and most of the MOP goodness is gone, starting with that you can't have a different metaclass than structure-class. Also, because of the mentioned inflexibility, you have to dismiss several warnings if you do re-define a struct.
On the other hand, structs have a lot of other, more low-level convenience tooling, e. g. automatic and portable serialization and deserialization.
I _think_ that one might be able to create a new macro (let's say def-struct-class) that has exactly the same semantics as defstruct, but in development mode uses defclass underneath, while being an alias for defstruct in production mode. It might get a bit hairy to adhere to :type :list or :vector, though. The result is a bit limited, but just might be what you need.
Yep. Got that.
The folk at ITA also did quite some work on compile-time metaprogramming in order to get fast objects etc., so maybe they can chime in.
This tells me that ITA had a pretty big itch to scratch in that area. Very interesting clue...
Anyway, my stance is that the incredibly flexible MOP classes provide general semantic usefulness, but Common Lisp does offer more constrained options that trade general usefulness for speed.
I agree with one caveat, in this area ANSI-CL pretty gratuitously went one notch too far I'd say. Regards, Jean-Claude
-------- Date: Tue, 08 Dec 2020 22:17:10 +0100 From: "Svante Carl v. Erichsen" <svante.v.erichsen@web.de> Hi! [Sorry to be coming late to the game.] It is used all the time while working on a live system (image). If you want raw performance, you can often substitute structs, if you can live with the downsides: all objects from a class must be thrown away on redefinition (which might be OK if you want to treat e. g. a production system as unchangeable except through re-deployment), you only get single inheritance (which is often not needed anyway), and most of the MOP goodness is gone, starting with that you can't have a different metaclass than structure-class. Also, because of the mentioned inflexibility, you have to dismiss several warnings if you do re-define a struct. The only reason why structs must be thrown away after a change in definition is because the objects have identity, and thus cannot be modified internally and under-the-table. CLOS objects, on the other hand, have base objects that are tiny and their guts can be modified without losing their identity, because these guts have been indirected through the base objects that provide that identity. I'm guessing that most CL implementations use similar techniques, because it would be hard to implement CLOS semantics without having this single indirection. A struct-like CLOS reference is therefore possible with one more indirection than a struct, and that extra register load need not be per-reference - the indirection could be pre-loaded for any sections of code which can afford to (or which must) treat the object consistently. Conversely, a struct, if implemented with an indirection, could be set up to be lazily updated, just like CLOS instances, if the implementation so chose to pay the extra load penalty. On the other hand, structs have a lot of other, more low-level convenience tooling, e. g. automatic and portable serialization and deserialization. Yes, structs have an implied make-load-form, although m-l-f is itself much more powerful than the implicit one for structs. In almost every way, structs are a subset of CLOS semantically, and we can get fairly close to struct performance with CLOS objects. I _think_ that one might be able to create a new macro (let's say def-struct-class) that has exactly the same semantics as defstruct, but in development mode uses defclass underneath, while being an alias for defstruct in production mode. It might get a bit hairy to adhere to :type :list or :vector, though. The result is a bit limited, but just might be what you need. Bringing out defstruct semantics as a wrapper for a CLOS implementation seems counterproductive; the (CLOS) implementation would always be fighting the semantic limitations of structs, whereas voluntary restrictions on CLOS semantics would allow the full CLOS semantics to be brought out and extensions would be trivial; instead of inventing new extensions on defstruct semantics, a relaxing of the voluntary restrictions would bring back the CLOS smantics as needed. The folk at ITA also did quite some work on compile-time metaprogramming in order to get fast objects etc., so maybe they can chime in. Anyway, my stance is that the incredibly flexible MOP classes provide general semantic usefulness, but Common Lisp does offer more constrained options that trade general usefulness for speed. Elsewhere in this thread, both inlining and sealing have been mentioned. In fact, they are different parts of the same elephant, which may look different but which can all be placed under the category of early vs late binding. That is, where do you package up a code base to the point where the api is solid enough to perform optimizations on that code? Often, it is the difference between interpreted code and compiled code - the binding occurs at compile-time when macros are expanded once and then commited to assembler code (whereas interpreted code tends to macroexpand for each invocation). In the case of CLOS, the commitment never need be permanent; even after an object has been created, it can be modified without destroying its identity, thanks to AMOP. Allegro CL provides a different cross-section of binding-time; a CLOS slot can be given a "fixed-index" attribute, which holds a slot to a particular index. It is one of those restrictons I mentioned that can be used or not - a class can specify fixed-index and/or regular (non-fixed-index) slots. The semantics of fixed-index slots are the same as regular slots, except that trying to define or mixin two slots with the same index and a different name will cause an error. This has a few remifications, but is usable in practice. A fixed-index slot is used exactly as a regular slot, but a separate macro, slot-value-using-class-name, provides syntax for defining extremely fast accesses (one or two instructions, depending on how it is used). This feature has been around for many years, but was mostly undocumented until recently. For the documentation and examples, see https://franz.com/support/documentation/current/doc/implementation.htm#ef-sl... Note that in fixed-index (and especially slot-value-using-class-name) semantics, the binding time is macroexpand/compile time, whuch means that redefinition will require recompilation, but _only_ if the actual fixed-index values have changed; slots that have unchanged fixed index locations will remain at the same place, allowing the accessors to remain in place, and other slot-value usages for the non-fixed slots will be taken care of lazily by the update-instance-for-{different,redefined}-class mechanisms. Duane Rettig Jean-Claude Beaudoin writes: > Hello Pros of Common Lisp, > > Here is my attempt at starting a significant (and hopefully useful) debat e > on a subject squarely about Common Lisp and its internals. > > My main stance here is to state that I have yet to see, in a significant > application, any use of functions #'cl:make-instance-obsolete and > #'cl:update-instance-for-redefined-class and of the underlying machinery > that support the remorphing of instances following a class redefinition > (through a subsequent cl:defclass most likely). I can state the same thin g > about #'cl:update-instance-for-different-class and #'cl:change-class. > > The machinery required of any CL implementation to properly support those > functions (mentioned here above) is quite significant in its complexity a nd > usually imposes a sizeable performance penalty on the speed of any instan ce > slot access. > > A different choice of class redefinition semantics can lead to an > implementation of CLOS with much reduced instance slot access overhead. > > The necessity of supporting instance remorphing should therefore be well > motivated by very significant gains in application code performance (in > speed and/or size and/or expressiveness power). > > Can anyone of you point me to some evidence of such application level > usefulness? > Is there any notorious usecase of this (instance remorphing) since its > inclusion into ANSI CL? > > > By the way, at the bottom of the entry about #'cl:change-class in the tex t > of the CL standard, one can find a "Notes:" section that starts with this > sentence: "The generic function change-class has several semantic > difficulties." And this was written in the context of a single-threaded > implementation, as ANSI-CL limited itself. I bet that almost all currentl y > significant CL implementations are now multi-threaded, therefore the > "several semantic difficulties" are greatly and gravely compounded. In my > opinion, this makes proper motivation of usefulness all the more imperiou s. > What do you think? --------
I see it and use it a lot. Maybe that's not significant to you, but it's significant to me. Zach On Tue, Dec 8, 2020 at 2:51 PM Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:
Hello Pros of Common Lisp,
Here is my attempt at starting a significant (and hopefully useful) debate on a subject squarely about Common Lisp and its internals.
My main stance here is to state that I have yet to see, in a significant application, any use of functions #'cl:make-instance-obsolete and #'cl:update-instance-for-redefined-class and of the underlying machinery that support the remorphing of instances following a class redefinition (through a subsequent cl:defclass most likely). I can state the same thing about #'cl:update-instance-for-different-class and #'cl:change-class.
The machinery required of any CL implementation to properly support those functions (mentioned here above) is quite significant in its complexity and usually imposes a sizeable performance penalty on the speed of any instance slot access.
A different choice of class redefinition semantics can lead to an implementation of CLOS with much reduced instance slot access overhead.
The necessity of supporting instance remorphing should therefore be well motivated by very significant gains in application code performance (in speed and/or size and/or expressiveness power).
Can anyone of you point me to some evidence of such application level usefulness? Is there any notorious usecase of this (instance remorphing) since its inclusion into ANSI CL?
By the way, at the bottom of the entry about #'cl:change-class in the text of the CL standard, one can find a "Notes:" section that starts with this sentence: "The generic function change-class has several semantic difficulties." And this was written in the context of a single-threaded implementation, as ANSI-CL limited itself. I bet that almost all currently significant CL implementations are now multi-threaded, therefore the "several semantic difficulties" are greatly and gravely compounded. In my opinion, this makes proper motivation of usefulness all the more imperious. What do you think?
On 7 Dec 2020, at 05:52, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
Hello Pros of Common Lisp,
Here is my attempt at starting a significant (and hopefully useful) debate on a subject squarely about Common Lisp and its internals.
My main stance here is to state that I have yet to see, in a significant application, any use of functions #'cl:make-instance-obsolete and #'cl:update-instance-for-redefined-class and of the underlying machinery that support the remorphing of instances following a class redefinition (through a subsequent cl:defclass most likely). I can state the same thing about #'cl:update-instance-for-different-class and #'cl:change-class.
The machinery required of any CL implementation to properly support those functions (mentioned here above) is quite significant in its complexity and usually imposes a sizeable performance penalty on the speed of any instance slot access.
A different choice of class redefinition semantics can lead to an implementation of CLOS with much reduced instance slot access overhead.
The necessity of supporting instance remorphing should therefore be well motivated by very significant gains in application code performance (in speed and/or size and/or expressiveness power).
Can anyone of you point me to some evidence of such application level usefulness? Is there any notorious usecase of this (instance remorphing) since its inclusion into ANSI CL?
The implementation of ContextL relies heavily on the CLOS MOP, and among other things on runtime class redefinition and corresponding updates of class instances. You can find the details how the class redefinition machinery is used in cx-partial-class.lisp at https://github.com/pcostanza/contextl <https://github.com/pcostanza/contextl> To the best of my knowledge, ContextL is one of the most efficient implementations of context-oriented programming (as compared to other implementations of COP for Java, Smalltalk, and a few other languages). This is because there is enough machinery in the CLOS MOP that provides the necessary dynamism and efficiency at the same time. ContextL has been used in production, so should count as relevant. Removal of features can lead to more efficient implementation strategies, but also leads to less expressivity. It’s always difficult to draw a good line between what is good for most cases, and what is necessary for some. I, for one, am happy that CLOS MOP provides this amount of flexibility. There are enough OOP systems / languages with reduced expressivity (including ISLISP, Dylan, and some Scheme implementations) that we don’t necessarily need another one. It has already been pointed out in this thread that you can always revert to the much more restricted defstruct. Maybe that one is too restrictive for your taste, but again, it is difficult to draw a good line between restrictive and expressive, and no compromise will make everybody happy. I think CLOS could be even more flexible, without loss of efficiency, but that’s a different discussion. Pascal -- Pascal Costanza
On Tue, Dec 8, 2020 at 8:02 PM Pascal Costanza <pc@p-cos.net> wrote:
The implementation of ContextL relies heavily on the CLOS MOP, and among other things on runtime class redefinition and corresponding updates of class instances. You can find the details how the class redefinition machinery is used in cx-partial-class.lisp at https://github.com/pcostanza/contextl
To the best of my knowledge, ContextL is one of the most efficient implementations of context-oriented programming (as compared to other implementations of COP for Java, Smalltalk, and a few other languages). This is because there is enough machinery in the CLOS MOP that provides the necessary dynamism and efficiency at the same time.
ContextL has been used in production, so should count as relevant.
Thank you for the pointer to ContextL. I just cloned the repo and I'll browse the code to see if I can get properly educated thusly.
Removal of features can lead to more efficient implementation strategies, but also leads to less expressivity. It’s always difficult to draw a good line between what is good for most cases, and what is necessary for some. I, for one, am happy that CLOS MOP provides this amount of flexibility. There are enough OOP systems / languages with reduced expressivity (including ISLISP, Dylan, and some Scheme implementations) that we don’t necessarily need another one.
It has already been pointed out in this thread that you can always revert to the much more restricted defstruct. Maybe that one is too restrictive for your taste, but again, it is difficult to draw a good line between restrictive and expressive, and no compromise will make everybody happy.
I am indeed of the opinion that the "line" has been drawn at the wrong level/place. That this choice has brought no real gain of expressiveness over what could have been provided in, say, a library. And that this choice has also brought very real overhead and "semantic difficulties" (as alluded in the ANSI-CL entry on #'CL:CHANGE-CLASS).
I think CLOS could be even more flexible, without loss of efficiency, but that’s a different discussion.
How about as flexible with improved efficiency for marginal inconvenience?
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
Because you’d have to go find all the pointers to the old instance. Maybe you don’t want to do that. Or maybe you don’t care, but that’s ok because what CLOS gives you is possibilities. Class redefinition is cheap, in the sense that until you touch each instance (i.e. passing it to a method) no work is done on it. I suspect — but can’t remember the details — that cl:change-class recalculated slots on the spot. - nick
About multithreading, *all *kinds of redefinition have an impact. If I redefine a widely-used, low-level function, with hundreds of call sites – will each thread immediately call the new one without the bug, or will some still call the old one? Again, imposing a proper order would require protecting each function call with a lock, which is even worse for performance than protecting each slot access. Still, we consider function redefinition a key feature of Common Lisp. So, redefinition of classes is in accordance with the spirit of the language. Anecdotally, implementations that don't allow to redefine what Common Lisp doesn't mandate (e.g., in ABCL you cannot really redefine packages), in certain situations are painful to use, as they force you to delete & recreate everything, possibly even quitting Lisp and restarting. On Wed, 9 Dec 2020 at 09:19, Nick Levine <nick@nicklevine.org> wrote:
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
Because you’d have to go find all the pointers to the old instance. Maybe you don’t want to do that. Or maybe you don’t care, but that’s ok because what CLOS gives you is possibilities.
Class redefinition is cheap, in the sense that until you touch each instance (i.e. passing it to a method) no work is done on it. I suspect — but can’t remember the details — that cl:change-class recalculated slots on the spot.
- nick
On Wed, Dec 9, 2020 at 3:59 AM Alessio Stalla <alessiostalla@gmail.com> wrote:
About multithreading, *all *kinds of redefinition have an impact. If I redefine a widely-used, low-level function, with hundreds of call sites – will each thread immediately call the new one without the bug, or will some still call the old one? Again, imposing a proper order would require protecting each function call with a lock, which is even worse for performance than protecting each slot access.
Isn't "inline/not-inline" doing just what is needed in that area? Still, we consider function redefinition a key feature of Common Lisp. So,
redefinition of classes is in accordance with the spirit of the language.
Redefinition of function is not "in situ". Why should redefinition of classes have to be so? I am not advocating against class redefinition! Anecdotally, implementations that don't allow to redefine what Common Lisp
doesn't mandate (e.g., in ABCL you cannot really redefine packages), in certain situations are painful to use, as they force you to delete & recreate everything, possibly even quitting Lisp and restarting.
On Wed, 9 Dec 2020 at 10:50, Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:
On Wed, Dec 9, 2020 at 3:59 AM Alessio Stalla <alessiostalla@gmail.com> wrote:
About multithreading, *all *kinds of redefinition have an impact. If I redefine a widely-used, low-level function, with hundreds of call sites – will each thread immediately call the new one without the bug, or will some still call the old one? Again, imposing a proper order would require protecting each function call with a lock, which is even worse for performance than protecting each slot access.
Isn't "inline/not-inline" doing just what is needed in that area?
I'm talking about non inlined functions, for which the implementation may nevertheless have done type inference or other optimizations that could be invalidated by a redefinition.
Still, we consider function redefinition a key feature of Common Lisp. So,
redefinition of classes is in accordance with the spirit of the language.
Redefinition of function is not "in situ". Why should redefinition of classes have to be so? I am not advocating against class redefinition!
It is. We can redefine a function without having to track and recompile/redefine all uses of that function. In the simple case, it's just updating a pointer. But what if the compiler had performed cross-function optimizations? Or, what if the compiler *couldn't *perform such optimizations because they would have gotten in the way of redefinition? It's not just CLOS to have this issue – it's the whole of Common Lisp.
On Wed, Dec 9, 2020 at 3:16 AM Nick Levine <nick@nicklevine.org> wrote:
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
Because you’d have to go find all the pointers to the old instance. Maybe you don’t want to do that. Or maybe you don’t care, but that’s ok because what CLOS gives you is possibilities.
Yes, preserving instance identity is at the core of the question. It could even be the only question here. Is there any other? But what looks like an occasional convenience comes at a cost.
Class redefinition is cheap, in the sense that until you touch each instance (i.e. passing it to a method) no work is done on it. I suspect — but can’t remember the details — that cl:change-class recalculated slots on the spot.
- nick
BTW, I just remembered that PCL (that venerable demonstration implementation of CLOS) contains all the machinery needed to implement that identity preservation feature as application level code expressed in CLTL1 compatible code. So this shows that the whole thing could be implemented as a library/package all from the beginning. It is a proof of concept of some kind I would say.
Jean-Claude the point is that CLOS is not a "library", although it was certainly possible to implement it that way.. It is an integral part of CL "as is". I see that the discussion has gone ahead, but here is my 2 cents. I don't have much time to work on it due to my day job (and other - retrocomputing - distractions). First of all there are several issues with CL which need to be clarified. - Some time ago, I started looking at reviewing the condition hierarchy. The motivation being the fact that (read-from-string "I-AM-NOT-A-PACKAGE::FOO") yields different conditions in different implementations. This is just an example: there are many, many more. - If your really want to help a "smart enough compiler" to produce fast code, what about writing up a precise explanation, wrt the current CL standard of what an extension like (defclass foo (a1 a2 ... an) (... slots...) ... (:sealed t)) should actually do? - I still would like to be able to write an interval arithmetic package in Common Lisp (yes; I may have hooked up a student to finish the grunt work of producing an initial spec that could be discussed in detail. - Can we have a common and Common Lispy network interface? - Did I mention (pathname-device some-pathname) ? The list can go on. But what I have in mind are all clarifications and extension to the CL spec as is. Not that there aren't libraries out there that do some or most of what I have in mind, but that's' the way things are right now. All the best Marco On Wed, Dec 9, 2020 at 10:11 AM Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:
On Wed, Dec 9, 2020 at 3:16 AM Nick Levine <nick@nicklevine.org> wrote:
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
Because you’d have to go find all the pointers to the old instance. Maybe you don’t want to do that. Or maybe you don’t care, but that’s ok because what CLOS gives you is possibilities.
Yes, preserving instance identity is at the core of the question. It could even be the only question here. Is there any other?
But what looks like an occasional convenience comes at a cost.
Class redefinition is cheap, in the sense that until you touch each instance (i.e. passing it to a method) no work is done on it. I suspect — but can’t remember the details — that cl:change-class recalculated slots on the spot.
- nick
BTW, I just remembered that PCL (that venerable demonstration implementation of CLOS) contains all the machinery needed to implement that identity preservation feature as application level code expressed in CLTL1 compatible code. So this shows that the whole thing could be implemented as a library/package all from the beginning. It is a proof of concept of some kind I would say.
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
... I forgot. Let's get the CLtL2 "Environment" API back in the "extended" spec. Which brings me to the following statement. The issue here is to *extend *the spec; not to cut pieces out of it. All the best. Marco PS. Nag, nag. :) ABCL crowd! You said you were going to bring the environment API in "Very Soon Now" (tm). I have not checked the latest and greatest ABCL, but is it in now? :) :) :) On Wed, Dec 9, 2020 at 12:07 PM Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Jean-Claude
the point is that CLOS is not a "library", although it was certainly possible to implement it that way.. It is an integral part of CL "as is".
I see that the discussion has gone ahead, but here is my 2 cents. I don't have much time to work on it due to my day job (and other - retrocomputing - distractions).
First of all there are several issues with CL which need to be clarified.
- Some time ago, I started looking at reviewing the condition hierarchy. The motivation being the fact that (read-from-string "I-AM-NOT-A-PACKAGE::FOO") yields different conditions in different implementations. This is just an example: there are many, many more. - If your really want to help a "smart enough compiler" to produce fast code, what about writing up a precise explanation, wrt the current CL standard of what an extension like (defclass foo (a1 a2 ... an) (... slots...) ... (:sealed t)) should actually do? - I still would like to be able to write an interval arithmetic package in Common Lisp (yes; I may have hooked up a student to finish the grunt work of producing an initial spec that could be discussed in detail. - Can we have a common and Common Lispy network interface? - Did I mention (pathname-device some-pathname) ?
The list can go on. But what I have in mind are all clarifications and extension to the CL spec as is. Not that there aren't libraries out there that do some or most of what I have in mind, but that's' the way things are right now.
All the best
Marco
On Wed, Dec 9, 2020 at 10:11 AM Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:
On Wed, Dec 9, 2020 at 3:16 AM Nick Levine <nick@nicklevine.org> wrote:
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
Because you’d have to go find all the pointers to the old instance. Maybe you don’t want to do that. Or maybe you don’t care, but that’s ok because what CLOS gives you is possibilities.
Yes, preserving instance identity is at the core of the question. It could even be the only question here. Is there any other?
But what looks like an occasional convenience comes at a cost.
Class redefinition is cheap, in the sense that until you touch each instance (i.e. passing it to a method) no work is done on it. I suspect — but can’t remember the details — that cl:change-class recalculated slots on the spot.
- nick
BTW, I just remembered that PCL (that venerable demonstration implementation of CLOS) contains all the machinery needed to implement that identity preservation feature as application level code expressed in CLTL1 compatible code. So this shows that the whole thing could be implemented as a library/package all from the beginning. It is a proof of concept of some kind I would say.
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
On Wed, Dec 9, 2020 at 6:18 AM Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
... I forgot.
Let's get the CLtL2 "Environment" API back in the "extended" spec.
Which brings me to the following statement. The issue here is to *extend *the spec; not to cut pieces out of it.
CLtL2 "Environment" API is second from the top of my list of things to do. But I am of the opinion that this API needs one or two corrections before being good enough for primetime. But currently I need first to finish my C99-complete FFI thing.
On Wed, Dec 9, 2020 at 12:07 PM Marco Antoniotti < marco.antoniotti@unimib.it> wrote:
Jean-Claude
the point is that CLOS is not a "library", although it was certainly possible to implement it that way.. It is an integral part of CL "as is".
I see that the discussion has gone ahead, but here is my 2 cents. I don't have much time to work on it due to my day job (and other - retrocomputing - distractions).
First of all there are several issues with CL which need to be clarified.
- Some time ago, I started looking at reviewing the condition hierarchy. The motivation being the fact that (read-from-string "I-AM-NOT-A-PACKAGE::FOO") yields different conditions in different implementations. This is just an example: there are many, many more.
You have to hit directly on it to see what could be wrong with it. I cannot do much about other implementations though.
- If your really want to help a "smart enough compiler" to produce fast code, what about writing up a precise explanation, wrt the current CL standard of what an extension like (defclass foo (a1 a2 ... an) (... slots...) ... (:sealed t)) should actually do?
I tried myself at this one quite a while back and got a very big
headache. If I recall correctly my working conclusion was that this :sealed option of cl:defclass was targeting the wrong thing. The CPL of the class was the real key instead and I could not see how to nail that one. But that has been quite a while ago so I could be fabulating here.
- I still would like to be able to write an interval arithmetic package in Common Lisp (yes; I may have hooked up a student to finish the grunt work of producing an initial spec that could be discussed in detail.
Looks like a "also nice to have".
- Can we have a common and Common Lispy network interface?
Such a nice candidate for a (standard?) library.
- - Did I mention (pathname-device some-pathname) ?
Specifically pathname-device alone or the whole pathname business?
-
The list can go on. But what I have in mind are all clarifications and extension to the CL spec as is. Not that there aren't libraries out there that do some or most of what I have in mind, but that's' the way things are right now
Thank you very much for all this. Regards, Jean-Claude
Marco Antoniotti <marco.antoniotti@unimib.it> writes:
- Some time ago, I started looking at reviewing the condition hierarchy. The motivation being the fact that (read-from-string "I-AM-NOT-A-PACKAGE::FOO") yields different conditions in different implementations. This is just an example: there are many, many more.
What would also be nice is having conditions be CLOS classes (I recall reading somewhere that they already are in most implementations). Then we can have fun doing CHANGE-CLASS inside condition handlers. -- Vladimir Sedach Software engineering services in Los Angeles https://oneofus.la
On 9 Dec 2020, at 10:11, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
On Wed, Dec 9, 2020 at 3:16 AM Nick Levine <nick@nicklevine.org <mailto:nick@nicklevine.org>> wrote:
Can I ask why you invoke #'CL:CHANGE-CLASS on an object instead of simply creating a new instance of the second class with adequate initialization?
Because you’d have to go find all the pointers to the old instance. Maybe you don’t want to do that. Or maybe you don’t care, but that’s ok because what CLOS gives you is possibilities.
Yes, preserving instance identity is at the core of the question. It could even be the only question here. Is there any other?
But what looks like an occasional convenience comes at a cost.
The cost of implementing the mechanisms involved has already been payed. Changing the semantics would imply changing the implementations, and would have an impact on all existing libraries and applications. So you would be incurring an additional cost. (That alone is not a good argument for keeping the feature, it’s just to remind you that the cost/benefit analysis in this regard is not straightforward.) I doubt that the cost in terms of runtime performance really matters that much in practice. Class redefinitions are rare, I suppose, and I doubt that slot accesses have a high overhead because of them. (I would think that unbound slots, for example, cost more, but that’s just guessing.) In my experience, the two most important feature for improving performance in Common Lisp are inlining and stack allocation. If you could work on a way how to inline generic functions while maintaining the ability to add and remove methods at runtime, and to make dynamic-extent declarations on CLOS instances effective, that would probably buy us a lot more in terms of performance than restricting class redefinition capabilities. Pascal -- Pascal Costanza
Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
My main stance here is to state that I have yet to see, in a significant application, any use of functions #'cl:make-instance-obsolete and #'cl:update-instance-for-redefined-class and of the underlying machinery that support the remorphing of instances following a class redefinition (through a subsequent cl:defclass most likely). I can state the same thing about #'cl:update-instance-for-different-class and #'cl:change-class.
At least from a theoretical point of view, CHANGE-CLASS is a simple, natural, and elegant solution to situations where you can't come up with a design that gets behavioral subtyping (LSP) right in all situations. Simple example: the famous square/rectangle (or circle/ellipse) problem. (defclass rectangle () ((width :initarg :width :accessor width) (height :initarg :height :accessor height))) (defun make-rectangle (width height) (if (= width height) (make-instance 'square :width width) (make-instance 'rectangle :width width :height height))) (defmethod (setf width) :after (width (rectangle rectangle)) (declare (ignore width)) (when (= (width rectangle) (height rectangle)) (change-class rectangle 'square))) (defmethod (setf height) :after (height (rectangle rectangle)) (declare (ignore height)) (when (= (width rectangle) (height rectangle)) (change-class rectangle 'square))) (defclass square () ((width :initarg :width :reader width :reader height :accessor side))) (defun make-square (width) (make-instance 'square :width width)) (defmethod (setf width) (width (square square)) (let ((side (side square))) (unless (= width side) (change-class square 'rectangle :width width :height side))) width) (defmethod (setf height) (height (square square)) (unless (= height (side square)) (change-class square 'rectangle :height height)) height) I would hate to see it go. I have 450 new students every year that learn about CHANGE-CLASS :-D -- ¡En Seguida! -- New album: https://www.didierverna.com/records/en-seguida.php Available on all digital platforms now! Lisp, Jazz, Aïkido: http://www.didierverna.info
participants (18)
-
Alessio Stalla -
David McClain -
dbm＠refined-audiometrics.com
-
Didier Verna -
Duane Rettig -
Faré -
Hans Hübner -
Jean-Claude Beaudoin -
Manfred Bergmann -
Marco Antoniotti -
Michael Fiano -
Neil Gilmore -
Nick Levine -
Pascal Costanza -
Robert Goldman -
Svante Carl v. Erichsen -
Vladimir Sedach -
Zach Beane