Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me in contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a hash-table implementation (PCL and its derivatives all seem to do just that).
The problem I see with this is that it will be a "for ever growing" hash-table with not upper bound in sight. And the "purpose" of such a mandatory built-in memory leak also completely escapes me. Could any of you share some insight on this question, please?
Thank you,
JCB
Not sure where I see either "forever growing" or "memory leak". Come to think of it, not sure what you mean by "mandatory". It is a spec of how the MOP should work internally. Do you have some other way in mind for things to work?
I mean, it sounds like you might be talking about forever adding and removing eql-specialized methods, but I'd rather not guess. Even if so, nothing stops the implementation from GCing unused specializer metaobjects.
-hk
On Fri, Dec 26, 2014 at 2:10 AM, Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:
Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me in contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a hash-table implementation (PCL and its derivatives all seem to do just that).
The problem I see with this is that it will be a "for ever growing" hash-table with not upper bound in sight. And the "purpose" of such a mandatory built-in memory leak also completely escapes me. Could any of you share some insight on this question, please?
Thank you,
JCB
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
The specializers are probably interned for the sake of allowing better optimization of EQL dispatch. I don't think it'll grow without bounds; it probably gets big enough to hold all of the methods with EQL dispatch, and then grows no further.
Honestly, I don't use EQL methods much myself, if they can be avoided by using SELECT.
--S
On Fri, Dec 26, 2014 at 10:33 AM, Kenneth Tilton ken@tiltontec.com wrote:
Not sure where I see either "forever growing" or "memory leak". Come to think of it, not sure what you mean by "mandatory". It is a spec of how the MOP should work internally. Do you have some other way in mind for things to work?
I mean, it sounds like you might be talking about forever adding and removing eql-specialized methods, but I'd rather not guess. Even if so, nothing stops the implementation from GCing unused specializer metaobjects.
-hk
On Fri, Dec 26, 2014 at 2:10 AM, Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:
Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me in contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a hash-table implementation (PCL and its derivatives all seem to do just that).
The problem I see with this is that it will be a "for ever growing" hash-table with not upper bound in sight. And the "purpose" of such a mandatory built-in memory leak also completely escapes me. Could any of you share some insight on this question, please?
Thank you,
JCB
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
-- Kenneth Tilton Fort Lauderdale, FL http://tiltontec.com "In a class by itself." *-Macworld*
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
Scott (long time no see):
This reply doesn't seem correct at all. Interning an EQL specializer does not directly have anything to do with methods (although it is most often performed by the implementation when an EQL method is defined) nor would methods be remembered in the hash table or whatever the implementation uses to intern them.
An EQL specializer must exist in order for METHOD-SPECIALIZERS to have something to return, and as an argument to be passed to functions like SPECIALIZER-DIRECT-GENERIC-FUNCTIONS SPECIALIZER-DIRECT-METHODS EQL-SPECIALIZER-OBJECT
Specializers are also necessary if one wants to manipulate generic functions and methods directly using the MOP instead of with the DEFGENERIC and DEFMETHOD macros. If for no other reason, specializers exist because the MOP must be self-reflexive.
True, there is no way to remove an interned EQL specializer, which may be an oversight in the MOP. But there will be exactly one entry for each unique specializer in the EQL hash table or whatever else is used for their internment. I suppose an implementation that features weak hashtables could use one for EQL specializers, since if there were no other references to one there would be no portable way to ask if it was still there.
"Growing without bounds" is no different than what happens when a new symbol is interned, or a new function or method is defined on an interned symbol. Rarely any problem in typical programming.
On Fri, Dec 26, 2014 at 7:38 AM, Scott McKay swmckay@gmail.com wrote:
The specializers are probably interned for the sake of allowing better optimization of EQL dispatch. I don't think it'll grow without bounds; it probably gets big enough to hold all of the methods with EQL dispatch, and then grows no further.
Honestly, I don't use EQL methods much myself, if they can be avoided by using SELECT.
--S
On Fri, Dec 26, 2014 at 10:33 AM, Kenneth Tilton ken@tiltontec.com wrote:
Not sure where I see either "forever growing" or "memory leak". Come to think of it, not sure what you mean by "mandatory". It is a spec of how the MOP should work internally. Do you have some other way in mind for things to work?
I mean, it sounds like you might be talking about forever adding and removing eql-specialized methods, but I'd rather not guess. Even if so, nothing stops the implementation from GCing unused specializer metaobjects.
-hk
On Fri, Dec 26, 2014 at 2:10 AM, Jean-Claude Beaudoin jean.claude.beaudoin@gmail.com wrote:
Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me in contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a hash-table implementation (PCL and its derivatives all seem to do just that).
The problem I see with this is that it will be a "for ever growing" hash-table with not upper bound in sight. And the "purpose" of such a mandatory built-in memory leak also completely escapes me. Could any of you share some insight on this question, please?
Thank you,
JCB
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
-- Kenneth Tilton Fort Lauderdale, FL http://tiltontec.com "In a class by itself." -Macworld
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
Why is there no way to remove an interned EQL specializer meta-object? They get defined in the context of a method definition, so one just needs to do some good old-fashioned engineering: method-specializer reference tracking leveraged at method removal time to know when to toss the hash table entry.
I am more interested in why this is perceived as a problem, but if the OP is doing some dynamic metaprogramming I can imagine a use case.
On Fri, Dec 26, 2014 at 11:58 AM, Steve Haflich shaflich@gmail.com wrote:
Scott (long time no see):
This reply doesn't seem correct at all. Interning an EQL specializer does not directly have anything to do with methods (although it is most often performed by the implementation when an EQL method is defined) nor would methods be remembered in the hash table or whatever the implementation uses to intern them.
An EQL specializer must exist in order for METHOD-SPECIALIZERS to have something to return, and as an argument to be passed to functions like SPECIALIZER-DIRECT-GENERIC-FUNCTIONS SPECIALIZER-DIRECT-METHODS EQL-SPECIALIZER-OBJECT
Specializers are also necessary if one wants to manipulate generic functions and methods directly using the MOP instead of with the DEFGENERIC and DEFMETHOD macros. If for no other reason, specializers exist because the MOP must be self-reflexive.
True, there is no way to remove an interned EQL specializer, which may be an oversight in the MOP. But there will be exactly one entry for each unique specializer in the EQL hash table or whatever else is used for their internment. I suppose an implementation that features weak hashtables could use one for EQL specializers, since if there were no other references to one there would be no portable way to ask if it was still there.
"Growing without bounds" is no different than what happens when a new symbol is interned, or a new function or method is defined on an interned symbol. Rarely any problem in typical programming.
On Fri, Dec 26, 2014 at 7:38 AM, Scott McKay swmckay@gmail.com wrote:
The specializers are probably interned for the sake of allowing better optimization of EQL dispatch. I don't think it'll grow without bounds; it probably gets big enough to hold all of the methods with EQL dispatch, and then grows no further.
Honestly, I don't use EQL methods much myself, if they can be avoided by using SELECT.
--S
On Fri, Dec 26, 2014 at 10:33 AM, Kenneth Tilton ken@tiltontec.com
wrote:
Not sure where I see either "forever growing" or "memory leak". Come to think of it, not sure what you mean by "mandatory". It is a spec of how
the
MOP should work internally. Do you have some other way in mind for
things to
work?
I mean, it sounds like you might be talking about forever adding and removing eql-specialized methods, but I'd rather not guess. Even if so, nothing stops the implementation from GCing unused specializer
metaobjects.
-hk
On Fri, Dec 26, 2014 at 2:10 AM, Jean-Claude Beaudoin jean.claude.beaudoin@gmail.com wrote:
Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me in contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a hash-table implementation (PCL and its derivatives all seem to do just that).
The problem I see with this is that it will be a "for ever growing" hash-table with not upper bound in sight. And the "purpose" of such a mandatory built-in memory leak also completely escapes me. Could any
of you
share some insight on this question, please?
Thank you,
JCB
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
-- Kenneth Tilton Fort Lauderdale, FL http://tiltontec.com "In a class by itself." -Macworld
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
On Fri, Dec 26, 2014 at 11:38 AM, Kenneth Tilton ken@tiltontec.com wrote:
Why is there no way to remove an interned EQL specializer meta-object?
Because no way to do this was defined in the MOP. It's unclear whether you suggest there should be some programmatic way to unintern an EQL specializer, or whether the system should do it automatically. But neither makes a lot of sense.
If a user call uninterned an EQL specializer while there were still methods specialized upon it, then the MOP would become inconsistent.
They get defined in the context of a method definition,
They are also interned by an explicit user-code call to INTERN-EQL-SPECIALIZER, which would be a reasonably thing to do it using the MOP directly to install new methods, or even to test whether any method or gf exists specialized on that EQL object.
so one just needs to do some good old-fashioned engineering: method-specializer reference tracking leveraged at method removal time to know when to toss the hash table entry.
(defparameter .kenny. (intern-eql-specializer 'tilton))
How would the MOP do reference counting on this metaobject. If the implementation spontaneously uninterned it, then a subsequent call to i-e-s would return a different metaobject, in violation of the MOP specification.
I am more interested in why this is perceived as a problem, but if the OP is doing some dynamic metaprogramming I can imagine a use case.
Yes, indeed. I expect this thread is a lot of owrrying about nothing important. But as I suggested previously, an implementation with weak hash tables could unintern EQL specializers safely if it wanted to bother.
On Fri, Dec 26, 2014 at 10:10 PM, Steve Haflich shaflich@gmail.com wrote:
On Fri, Dec 26, 2014 at 11:38 AM, Kenneth Tilton ken@tiltontec.com wrote:
Why is there no way to remove an interned EQL specializer meta-object?
Because no way to do this was defined in the MOP.
Wait. Are we in the religious zone now? This is engineering.
It's unclear whether you suggest there should be some programmatic way to unintern an EQL specializer,
Yeah, it is trivial if you want to make it work. No help of course for the religious zone.
or whether the system should do it automatically. But neither makes a lot of sense.
If a user call uninterned an EQL specializer while there were still methods specialized upon it, then the MOP would become inconsistent.
What is the user up to, inside the MOP? Why are they doing something so perverse? Whatever reason they have, they are inside the mop they are authoring. (Guessing internal-eql-specializer is not exported). If their mop worries about IES leaks, they need to enforce GCability.
Should we not decide the angels atop pin headcount first?
I cannot believe the nonsese into which language lawyers forgetting they are endineers get themselves.
They get defined in the context of a method definition,
They are also interned by an explicit user-code call to INTERN-EQL-SPECIALIZER, which would be a reasonably thing to do it using the MOP directly to install new methods,
Dude. They are INSIDE a MOP that frets over IES leaks, they need to cope.
or even to test whether any method or gf exists specialized on that EQL object.
OK, I see. We are not in Kansas anymore.
I gotta say, no longer interested until the OP offers motivation for their concerns.
One thing we know in programming is that the abstract is largely unaddressable. bring me a sensible use case and I can help you.
Given that the OP has disappeared, i think this wise.
-hk
On Fri, Dec 26, 2014 at 11:04 PM, Kenneth Tilton ken@tiltontec.com wrote:
... Given that the OP has disappeared, i think this wise.
What do you mean "disappeared". I am just busy thinking through my reply/ies. Sorry if I am not as quick as you are in these matters, Kenny. And don't forget that English is a second language for me on the top of it...
BTW, thank you all for your thoughtful replies/reactions. They are quite useful.
Regards,
JCB
Bonjour, Jean Claude!
I look forward to your reappearance! No, this does not count! :)
-hk
On Fri, Dec 26, 2014 at 10:10 PM, Steve Haflich shaflich@gmail.com wrote:
On Fri, Dec 26, 2014 at 11:38 AM, Kenneth Tilton ken@tiltontec.com wrote:
Why is there no way to remove an interned EQL specializer meta-object?
Because no way to do this was defined in the MOP. It's unclear whether you suggest there should be some programmatic way to unintern an EQL specializer, or whether the system should do it automatically. But neither makes a lot of sense.
If a user call uninterned an EQL specializer while there were still methods specialized upon it, then the MOP would become inconsistent.
They get defined in the context of a method definition,
They are also interned by an explicit user-code call to INTERN-EQL-SPECIALIZER, which would be a reasonably thing to do it using the MOP directly to install new methods, or even to test whether any method or gf exists specialized on that EQL object.
so one just needs to do some good old-fashioned engineering: method-specializer reference
tracking
leveraged at method removal time to know when to toss the hash table
entry.
(defparameter .kenny. (intern-eql-specializer 'tilton))
How would the MOP do reference counting on this metaobject. If the implementation spontaneously uninterned it, then a subsequent call to i-e-s would return a different metaobject, in violation of the MOP specification.
I am more interested in why this is perceived as a problem, but if the
OP is
doing some dynamic metaprogramming I can imagine a use case.
Yes, indeed. I expect this thread is a lot of owrrying about nothing important. But as I suggested previously, an implementation with weak hash tables could unintern EQL specializers safely if it wanted to bother.
pro mailing list pro@common-lisp.net http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
I am ready to concede that the issue here is minor and peripheral but it is not inexistent.
From Steve's mention of specializer-direct-methods and friend it is clear
to me now that intern-eql-specializer is part of a specializer specific dependency tracking facility. And, looking again at the source code of PCL, I can see that facility used at least in the optimization of make-instance and compute-applicable-methods (probably as Scott suspected). So, this is reasonable purpose for me and I consider my original (subject line) question properly answered.
Implementing intern-eql-specializer by means of a weak hash-table as suggested by Steve is exactly what CCL does, clisp uses something similar (weak sets I think), but not SBCL. But somehow I don't think it fair from MOP to require every implementation to use weak hash-tables in this case.
I am more of the opinion that the specification is incomplete in this area. The parallel with the situation of symbols in packages had also struck me and I think it should be pushed somewhat further with the addition of unintern-eql-specializer (granted this one is as dangerous as unintern is for symbols) and of something like map-eql-specializers (à la do-symbols). (BTW, in PCL you can see its optimization code use a map-specializers.) With these two it becomes possible to implement something like scrub-unused-eql-specializers if one wants to, without them it is simply impossible.
I think this is just good principled engineering being applied here. And, no Kenny, principled engineering, as I understand it, is not a religion but rather a philosophy, a subdivision of pragmatism I would say even if you don't find it pragmatic enough.
And the principles at work here would be:
1) Respect clearly and completely defined interfaces.
2) Inaccessible internal state is a very bad thing, avoid it.
I admit that 2) comes from my hardware design days (way back) but I think it also applies to software, pretty much for the same reasons it imposed itself on the hardware side as central to the "design for testability" methodology.
Thank you all again for your replies. They have been of great help.
Cheers,
JCB
Hi,
1. intern-eql-specializer is not strictly necessary. eql-specializer metaobjects are actually not strictly necessary. Case in point: LispWorks doesn’t have eql-specializer metaobjects in its MOP implementation at all. In LispWorks, eql specializers are just lists of the form `(eql ,object). I don’t have the impression that this has a negative impact on performance, or so. When eql specializers are just lists like this, then I believe the whole issue of getting rid of ‘garbage’ eql specializers is a non issue: Once a method with a particular eql specializer is removed, the corresponding object can be freed (unless other references exist, of course). (You still have the issue of not easily being able to get rid of ‘garbage’ classes.)
2. Don’t take the CLOS MOP specification as some kind of final word on how CLOS should be implemented. The CLOS MOP was still work in progress when the specification was published, and this is acknowledged in “The Art of the Metaobject Protocol.” It’s perfectly possible that there are better ways to implement CLOS with better protocols. In practice, almost actual CLOS implementations actually deviate in one way or another from the CLOS MOP specification, which is partially for historical reasons, partially because there are holes in the CLOS MOP specification, partially because parts of the spec are not implementable, and partially because certain aspects can actually be implemented better.
3. I’m not convinced that eql specializers are the most pressing concern. For example, the total lack of a good specification for method combination metaobjects is much more worrisome. Also, the generic function invocation protocol is too restrictive, in that it doesn’t allow for more modern inlining techniques (like polymorphic inline caches, or trace-based JIT compilation, for example). There is probably more.
4. If you are really concerned about eql specializers using up too much memory (and I’m really wondering what kind of use case that might be), then it’s probably better to roll your own approach to make that work. You don’t even need to go through the MOP, just implement your own dispatch mechanism, maybe even completely independent from CLOS. It’s not that hard, and you can come up with rules for your own dispatch mechanism that don’t have to be in line with what CLOS or the CLOS MOP require at all.
Pascal
-- Pascal Costanza The views expressed in this email are my own, and not those of my employer.
On Sat, Dec 27, 2014 at 5:50 AM, Pascal Costanza pc@p-cos.net wrote:
Hi,
- intern-eql-specializer is not strictly necessary. ...
I had somewhat noticed ;-)
- Don’t take the CLOS MOP specification as some kind of final word on how
CLOS should be implemented. ...
Usage as made it more or less mandatory by now I think.
- I’m not convinced that eql specializers are the most pressing concern.
For example, the total lack of a good specification for method combination metaobjects is much more worrisome.
Yep! That one was a bit freakish when I redid it lately in the thing I tinker with.
Also, the generic function invocation protocol is too restrictive, in that it doesn’t allow for more modern inlining techniques (like polymorphic inline caches, or trace-based JIT compilation, for example).
This is almost precisely the subject I wanted to address next on this list, in a couple of weeks, when I'll think I'd be ready...
- If you are really concerned about eql specializers
I am not that much concerned about them and, in fact, I consider the matter settled now.
Thanks,
JCB
On Sat, Dec 27, 2014 at 4:59 AM, Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:
I think this is just good principled engineering being applied here. And, no Kenny, principled engineering, as I understand it, is not a religion but rather a philosophy, a subdivision of pragmatism I would say even if you don't find it pragmatic enough.
Doh! I am in the wrong meet-up. I was looking for the one for Simple Lisp *Application* Programmers!
Cool. In this "where the Hell is unintern-eql-specializer" fretting I see yet another reason CL rules. And agreed: details matter.
Sorry for the noise.
-hk
-
Jean-Claude Beaudoin -
Kenneth Tilton -
Pascal Costanza -
Scott McKay -
Steve Haflich