On 2 Aug 2014, at 06:35, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:

I think I found the subsection you refer to: AMOP 5.3.1 (pp. 142-144) "Implementation and User Specialization", which contains two sub-subsections with titles starting with "Restrictions on". I will try to decipher those and their implications now that I have renewed motivation to do so.

There are similar restrictions defined in http://www.lispworks.com/documentation/HyperSpec/Body/11_abab.htm - bullet 19. It is in general a good idea when defining methods that at least one of the specializers should only match instances of your own classes. (The phrasing “your own” is a bit ambiguous here, but I believe it should be clear what is meant.) This is a good rule of thumb because this ensures that when using different libraries in a project, they won’t step on each other’s toes. This doesn’t mean there can’t be exceptions to this rule, but if you break it, you should better document it well and be aware of the potential implications. Pascal -- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

Jean-Claude, the basic point that the other, more knowledgeable responders made, was: You cannot define a method on slot-value-using-class specializing on standard-class and expect the same behavior across implementations. This is because standard-class may be used within the lisp implementation itself and/or be treated by the compiler differently to achieve better performance. If you want to use slot-value-using-class, you need to define your own metaclass and specialize the methods on that. -Hans 2014-08-03 9:40 GMT+02:00 Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com>:

On Sat, Aug 2, 2014 at 7:38 AM, Pascal Costanza <pc@p-cos.net> wrote:

...

On 2 Aug 2014, at 06:35, Jean-Claude Beaudoin < jean.claude.beaudoin@gmail.com> wrote:

...

I think I found the subsection you refer to: AMOP 5.3.1 (pp. 142-144) "Implementation and User Specialization", which contains two sub-subsections with titles starting with "Restrictions on". I will try to decipher those and their implications now that I have renewed motivation to do so.

There are similar restrictions defined in http://www.lispworks.com/documentation/HyperSpec/Body/11_abab.htm - bullet 19.

I missed that one too. Thank you for pointing it out.

But after modifying my test code to take account of it I saw no difference in the results. It turns out that it is the last bullet of the sub-subsection "Restrictions on Portable Programs" that is the key factor in the issue I am interested in here (last bullet of page 144 in AMOP, or here <http://franz.com/support/documentation/current/doc/mop/concepts.html#portable> as pointed by Steve). It basically states that the defmethod on slot-value-using-class must happen before any call to make-instance on a class the method uses as a specializer. (And then goes on to motivate this definition order restriction). If I modify my test code to comply with this rule (see slot_mop_strict.lsp here attached) then sbcl stops being the odd man out and behaves like clisp and ACL (which I think is the proper behavior).

Further, if I agree to stop tickling the system where it does not like to be tickled and define a class tracked_class as a subclass of standard-class to be used as a metaclass of the classes I want to track (see slot_mop_strict_tracked_class.lsp here attached), then all of sbcl, clisp, ccl and ACL show the same behavior, properly calling slot-value-using-class both from slot-value or from the slot accessor. Only LispWorks persists in its (erroneous, I think) behavior of calling s-v-u-c only from slot-value. I think there is a consensus emerging here.

Thank you all for your help on this matter with a special emphasis toward Steve.

_______________________________________________ pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro

On 3 Aug 2014, at 09:40, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:

Further, if I agree to stop tickling the system where it does not like to be tickled and define a class tracked_class as a subclass of standard-class to be used as a metaclass of the classes I want to track (see slot_mop_strict_tracked_class.lsp here attached), then all of sbcl, clisp, ccl and ACL show the same behavior, properly calling slot-value-using-class both from slot-value or from the slot accessor. Only LispWorks persists in its (erroneous, I think) behavior of calling s-v-u-c only from slot-value. I think there is a consensus emerging here.

LispWorks deviates from the slot access protocol in two ways: - AMOP states that the third argument to s-v-u-c and friends is a slot definition metaobject, not a slot name. In LispWorks, it is in fact a slot name. The CLOS MOP specification specified a slot name as the third argument for a long time, until the last minute before the AMOP book was published, when it was changed to being a slot definition metaobject. LispWorks still adheres to an older version of the specification in this regard. - One reason why the change from slot name to slot definition metaobject was made is that it allows for better optimization of standard slot accesses that don’t need to go through user-defined methods on s-v-u-c and friends. In LispWorks, you need one extra step to invoke user-defined methods - you have to specify that slot access should not be optimized. In your example, the class definition for ‘foo should look like this to achieve this: (defclass foo () ((a :accessor foo-a :initarg :a) (b :accessor foo-b :initarg :b)) (:metaclass tracked-class) (:optimize-slot-access nil)) Then your modified protocol also works in LispWorks. It’s usually better to specialize s-v-u-c and friends not only on the first, but also on the third argument, on one of your own slot definition metaobject classes. To make this work, you usually also need to define methods on direct-slot-definition-class and effective-slot-definition-class. You may be interested in the Closer to MOP project - see http://common-lisp.net/project/closer/ and also available via quicklisp. This is a compatibility library which fixes incompatibilities of the CLOS MOP across different Common Lisp implementations, and makes LispWorks adhere better to the slot access protocol in AMOP. There your code would work unchanged, without the optimize-slot-access flag, and you would not need to define methods several times for different Common Lisp implementations.

It would be interesting to see if the situation has really changed by now. Is there a "full" MOP implementation now in use and what lessons has it taught? Most implementations for which I have source code are derived from PCL for their CLOS part and I think that PCL is probably the "not yet full" implementation that was available to the committee back then, wasn't it?

Closer to MOP also has a test suite to check how well a CLOS MOP implementation adheres to AMOP. According to that test suite, ABCL and SBCL are the only CL implementations that fully adhere to AMOP. All other implementations deviate in some aspects. However, two caveats: - This is only a statement about how well these implementations fulfill the test suite. There may still be other ways in which they deviate. So take this statement with a grain of salt. (In my experience, SBCL for example still adheres slightly better in practice than ABCL.) - Many areas in which CLOS MOP implementations don’t adhere don’t seem to matter that much in practice. The most often used protocol seems to be the slot access protocol. Other protocols are either not that useful - for example I see no really useful way how to reliably specialize ensure-generic-function-using-class or ensure-class-using-class, and you can typically achieve the desired effects also otherwise; or they are simply not widely used, like for example make-method-lambda. My impression is that with Closer to MOP, the most important areas are actually well covered. (However, I may be biased, being the author of Closer to MOP.) Also keep in mind that it is often useful to consider ways other than using the CLOS MOP to achieve what you want to do. For example, ANSI Common Lisp itself already specifies ways to modify slot accesses through the generic functions slot-missing and slot-unbound, and if this is sufficient for whatever you are trying to do, you might as well stick to that.

BTW, I noticed in the clisp documentation that they mention a problem with "forward-referenced-class", a case of misdesign they say. At first sight, they seem to have a case. Do they?

I’m not sure if they have a case or not. Specializing ensure-class-using-class is not so useful anyway, and tracking changes from forward-referenced-class to some subclass of standard-class also doesn’t seem to be that useful either, as far as I can tell. If you want to keep track of definitions of classes, it’s better to focus on initialize-instance and reinitialize-instance on subclasses of standard-class, and on finalize-inheritance.

If they do then the CLOS subcommittee would be vindicated.

That’s exaggerated. The AMOP book has a foreword to the specification which clearly states that this is preliminary work. Unfortunately, the foreword is not reproduced in the HTML version. Here is the text, to put things in perspective: "In this part of the book, we provide the detailed specification of a metaobject protocol for CLOS. Our work with this protocol has always been rooted in our own implementation of CLOS, PCL. This has made it possible for us to have a user community, which in turn has provided us with feedback on this protocol as it has evolved. As a result, much of the design presented here is well-tested and stable. As this is being written, those parts have been implemented not only in PCL, but in at least three other CLOS implementations we know of. Other parts of the protocol, even though they have been implemented in one form or another in PCL and other implementations, are less well worked out. Work remains to improve not only the ease of use of these protocols, but also the balance they provide between user extensibility and implementor freedom. "In preparing this specification, it is our hope that it will provide a basis for the users and implementors who wish to work with a metaobject protocol for CLOS. This document should not be construed as any sort of final word or standard, but rather only as documentation of what has been done so far. We look forward to seeing the improvements, both small and large, which we hope this publication will catalyze. "To this end, for Part II only (chapters 5 and 6), we grant permission to prepare revisions or other derivative works including any amount of the original text. We ask only that you properly acknowledge the source of the original text and explicitly allow subsequent revisions and derivative works under the same terms. To further facilitate improvements in this work, we have made the electronic source for these chapters publicly available; it can be accessed by anonymous FTP from the /pcl/mop directory an arisia.xerox.com. "In addition to the valuable feedback from users of PCL, the protocol design presented here has benefited from detailed comments and suggestions by the following people: Kim Barrett, Scott Cyphers, Harley Davis, Patrick Dussud, John Foderaro, Richard P. Gabriel, David Gray, David A. Moon, Andreas Paepcke, Chris Richardson, Walter van Roggen, and Jon L. White. We are very grateful to each of them. Any remaining errors, inconsistencies or design deficiencies are the responsibility of the authors alone." Personally, I don’t believe that forward-referenced-class or ensure-class-using-class are the most important areas that need fixing. There are other issues that are more relevant. Pascal -- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

On 4 Aug 2014, at 10:44, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:

On Sun, Aug 3, 2014 at 12:50 PM, Pascal Costanza <pc@p-cos.net> wrote:

- One reason why the change from slot name to slot definition metaobject was made is that it allows for better optimization of standard slot accesses that don’t need to go through user-defined methods on s-v-u-c and friends. In LispWorks, you need one extra step to invoke user-defined methods - you have to specify that slot access should not be optimized. In your example, the class definition for ‘foo should look like this to achieve this:

(defclass foo () ((a :accessor foo-a :initarg :a) (b :accessor foo-b :initarg :b)) (:metaclass tracked-class) (:optimize-slot-access nil))

I see. And are you cool with the pervasive opt-out nature of this :optimize-slot-access option? Looks pretty much like a quite clear rejection of the MOP slot protocol to me.

It’s a trade off, speed vs flexibility. It doesn’t really restrict you in any way, because you can always opt in. So I think it’s perfectly ok. If you don’t like it, there is Closer to MOP, which leaves optimize-slot-access at t when it can, but sets it to nil if necessary, so you don’t really have to worry that much.

Personally, I don’t believe that forward-referenced-class or ensure-class-using-class are the most important areas that need fixing. There are other issues that are more relevant.

Quite of a cliffhanger you've done here! Are we going to be treated with a sequel on the very subject of those issues?

Maybe. ;) Pascal -- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

On Fri, Aug 1, 2014 at 10:00 PM, Pascal J. Bourguignon < pjb@informatimago.com> wrote:

Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> writes:

...

So my question is: Which one is right?

I'd note that this is a major problem of how OO libraries or frameworks are defined. They very rarely specify or give any guarantee of when or whom will send a given message to a given object.

This makes indeed difficult to subclass and override methods in a sturdy way.

This is probably a reason why OO programmers nowadays tend to distance themselves from inheritance (using so called "flat" hierarchies), and like "final" methods a lot, which basically denies OO itself.

Which is a real shame, because Gregor et al wrote some excellent rules on how to design protocols well in the face of things like inheritance and method combination. I think this is all in AMOP.

Otherwise, for your question, you didn't mention any metaclass. I'm not sure about it, but I would expect such methods from AMOP to be used consistently only when you define your own metaclasses.

-- __Pascal Bourguignon__ http://www.informatimago.com/ "The factory of the future will have only two employees, a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment." -- Carl Bass CEO Autodesk

_______________________________________________ pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro

On Fri, Aug 1, 2014 at 10:16 PM, Kenneth Tilton <ken@tiltontec.com> wrote:

You are puzzliing over undefined behavior, meaning what you have discovered:is that implementations are free to go as crazy as, well, canines in a dog park.

As usual Kenneth you are entertainingly sarcastic almost to a fault.

This can be excitingly crazy, because the CL standardizers were too exhausted to include the MOP in the standard. (They made nice excuses about not wanting to inhibit implementers, but dollars to donuts they just ran up the white flag in the face of the idea of standardizing something as vast in itself as a complete HLL).

Musing over this ever since the official publication the CL standard, I am still not convinced it was a bad and unwise decision.

What were actually trying to build?

Just pushing my Sisyphus' rock (aka. MKCL) in a somewhat better direction.

Note that the MOP is not part of the spec. They explicitly punted on making it so. Ergo, you are likely fishing in unspecified waters, or dog parks as the case may be. Note also that no one has asked you what problem you are trying to solve. Until now. hth, hk On Sat, Aug 2, 2014 at 5:26 PM, Attila Lendvai <attila.lendvai@gmail.com> wrote:

...

I got different results in pretty much each CL implementation on the tracing of calls to slot-value-using-class.

make sure you're not using CL:TRACE for this because it may not work on SVUC due to various optimizations.

-- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “A true intellectual is a man who, after reading a book and being convinced by its arguments, will shoot someone or, more likely, order someone shot.” — John McCarthy

_______________________________________________ pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro

-- Kenneth Tilton Fort Lauderdale, FL http://tiltontec.com "In a class by itself." *-Macworld*

That reminds me of the time I did (trace format) That _did_ crash the LispM, very, very hard. -- Scott On Sat, Aug 2, 2014 at 8:52 PM, Steve Haflich <shaflich@gmail.com> wrote:

OK, now I'm sitting in front of the front end of a computer rather than the back end of a smelly wet large long-haired dog.

The restriction on "portable programs" extending (redefining, whatever) MOP functions is here:

http://franz.com/support/documentation/current/doc/mop/concepts.html#portabl...

The motivation for this particular restriction is twofold.

First, the CL language implementation as well as the MOP itself may depend upon the MOP itself. The intention is that the language and MOP can use CLOS and even the MOP in their own implementation, and if they do, and use only "specified" classes or classes specified on class (and operator) names in private packages or otherwise unexported and unknown to properly-written portable programs, those programs won't break the implementation or affect the efficiency of its implementation. This is no different than the ANS prohibition against redefining cons to take its arguments in reverse order, expecting that if the portable user application code was written against this specification, the result would be harmless. But global definitions Lisp worlds are indeed global (*).

The second reason is less obvious. The implementation of ANS CL, CLOS, and the MOP obviously all depend upon themselves. Thus their implementations are metacircular and need protection. Furthermore, the full MOP protocol even where not metaciculr is expensive. If it must be obeyed for "specified" operators on objects of "specified" classes, execution efficiency may be unacceptable. The CLOS and MOP subcommittee of X3J13 IMO did an exquisite job making it possible to implement these subspecifications into a usable "industrial-strength language."

The CLOS subcommittee recommended that CLOS, but not MOP, be accepted into the standard, because the MOP had never yet been fully implemented, and was not yet ready for standardization. That is exactly what we (X3J13) voted.

(*) Back around 1986 just before I intended to reboot my Lisp Machine anyway I redefining car and cdr with the other's definition. I expected an immediate flaming crash, however, the machine kept running just fine. This is because the implementation had already compiled all the important functions that called csr and cdr, and those calls obviously would be inlined, making my redefinitions ineffectual. I'm sure if I had searched for some code that passed #'car or #'cdr as a :key argument to some other function, I might have provoked some failure, but otherwise the machine and OS didn't care.

On Sat, Aug 2, 2014 at 2:59 PM, Kenneth Tilton <ken@tiltontec.com> wrote:

...

Note that the MOP is not part of the spec. They explicitly punted on making it so. Ergo, you are likely fishing in unspecified waters, or dog parks as the case may be.

Note also that no one has asked you what problem you are trying to solve.

Until now.

hth, hk

On Sat, Aug 2, 2014 at 5:26 PM, Attila Lendvai <attila.lendvai@gmail.com> wrote:

...

...

I got different results in pretty much each CL implementation on the tracing of calls to slot-value-using-class.

make sure you're not using CL:TRACE for this because it may not work on SVUC due to various optimizations.

-- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “A true intellectual is a man who, after reading a book and being convinced by its arguments, will shoot someone or, more likely, order someone shot.” — John McCarthy

_______________________________________________ pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro

-- Kenneth Tilton Fort Lauderdale, FL http://tiltontec.com "In a class by itself." *-Macworld*

_______________________________________________ pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro

_______________________________________________ pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro