Anton,

Thank you very much for your mail. It prompted me to explain what I'm thinking of in a much more clear (I hope!!) way. Here we go:

Anton Vodonosov wrote:

What is the difference for modularity in case of subclassing, from other cases were we apply

modularity?

I can't find the difference.

Here's the greater idea.

If you have a module, by using CL packages (without any change to CL), it is possible to have more than one set of exports. So, you can offer different interfaces to different callers. By interfaces in this context I mean the set of symbols that are exported.

If you offer interface I1 to callers C1, C2, and C2, and interface I2 to callers C4, C5, and C6, then you can be sure that C1, C2, and C3 will keep working as long as you do not make incompatible changes to interface I1. Similarly with I2.

Now, moving specifically to OOP:

Traditionally, e.g. in Smalltalk 80, it was a big deal that the caller of an instance of a class does not know the internals of the class. It only uses methods, and does not look internally at the instance variables (slots).

(I can't remember whether ST-80 had the concept of private methods that are only intended to be used from methods of the class and not from callers.)

OOP was often compared to, or even conflated with, the concept of "abstract datatypes" (ADT's), in the sense meant by Barbara Liskov. In CLU and such languages, similarly there was a sharp distinction between the advertised interface presented to callers of the ADT (abstract datatype, corresponding to a class), and whatever was going on inside it.

This was a modularity boundary that allowed a separation of concerns between the users of the ADT and the implementation of the ADT.

Recently, this was considered such a big deal that Liskov was awarded a Turing Award.

ADT's did not, per se, contain the concept of inheritance. Inheritance was first exposed to the general public with ST-80.

However, there wasn't even a semblance of an effort to provide any modular separation between a class and its subclasses. So, whenever the implementation of a class changed in any way at all, you risked breaking subclasses. This was particularly a problem when library 1, under control of group-of-people-1, provided a class, and library-2 subclassed it. If a new release of library-1 came out, there was in general no way for g-o-p-1 to find everybody who had subclassed their classes; if the lib-2 people installed the new release of lib-1, they had no idea whether anything would work.

Now, you could just say that there should be internals and externals, and the externals exposed to the callers and the externals exposed to the subclassers are exactly the same set.

This does not work for most non-trivial cases. I will assume that you know why, since I can't explain it briefly.

So, what I am proposing is to be explicit about what is being exposed to the subclassers, so that if the base class is changed in a new release but the subclasser interface is compatible, the lib-2 people can rest assured that their library will continue to work. (Unless there are bugs, of course.)

In C++ and Java, this isn't done with Lisp packages, of course. The "public" members are the interface to the caller, and the union of the "public" and "protected" members are the interface provided to the subclassers.

Whether the subclassers should be given permission to override the public things, in all cases, is an interesting but separate discussion. With Lisp packages, you can control this any way you want.

We may note that "protected" is actually public, in the sense that when we specify something as "protected", we specify that the library has external clients which can rely on this protected method or field.

In that sense, yes, absolutely.

Anyone may inherit from our public class, and access the "protected" members, therefore these members are available to public. If we change the protected member, we affect the library clients.

What I'm doing, however, is to distinguish between two sets of "public", as explained above. The callers, which would be a "big public" (lots of callers) must cope with incompatible changes relatively less often, whereas the "small public" (those who subclass) must cope relatively more often since more is revealed.

This drives me to a conclusion, that a single language feature is

sufficient

which allows to separate a protocol for interaction with a module from the module part we expect to change without affecting clients.

Taking you literally, we do not need a new CL feature. But translating what you're saying into the terms I am using, you're saying that we need only one protocol, whereas I am saying that it's better to have two. There is no "language feature" at the CL level, but there is a practice, or a "design pattern", that I am recommending. Having :documentation in the defpackage to explain what's going on, especially who is intended to use this package, would be very helpful. Design patterns are essentially a way to extend a language (in a nutshell).

I made observation on many java systems, that addition to packages the public/protected/private for classes has bad influence on the system modularity.

Many java programmers tend to hide attributes of every single class behind get/set accessors, even for simple classes which just hold values passed between methods and do not contain any logic; to create hierarchies of interface/abstract class/concrete for various minor classes, employing protected methods.

There are at least two reasons to put in such methods.

First, changing the implementation while keeping compatible behavior for the public methods.

Imagine that you have a class called Point representing a point in two-space. You could use Cartesian representation: two data members x and y, and getX and getY (and, if desired, setX and setY). These are the simple getters (and setters) that you mention above.

But what if, for some reason (numerical methods, I don't know) you decide that internally using a polar representation, with two data members storing an angle and a distance from the origin, is better. You can make getX continue to work using the appropriate trigonometry operations.

Second, these days there are lots of tools that use "dependency injection" and whatnot that only work if you have such methods. Extending "ant", using Spring, and so on work this way, so it has become part of the usual Java design pattern for many classes. (See "Java Bean".)

But at the same time, people do not care to create structure at larger level, to divide the system into modules. It is not uncommon to see large systems where all these classes, subclasses in all the packages are public: everything is available to everything.

Sure.

Separate access control for classes obscures the idea of modularity (especially in case of such an OO centric language like java, the programmers are mostly concentrated on classes/object, and often don't even think about packages;).

I don't know what you mean by this. It depends what you mean by "access control". But if you mean limiting the access that callers of a module have, that is exactly the way modularity is usually realized.

Class is too small entity to form a module. A module API (protocol) usually consists of set of classes, methods, factory functions, maybe some global variables.

Ah, yes. That's why Common Lisp has packages and Java has namespaces; to be able to make groups exactly as you are saying.

Another thought is that the interface between base class and its subclasses is not a second interface, but a part of the first

interface.

As you said:

...

So we could have one package for the code that calls into our library, which would have the usual protocol, which we can call the "external protocol". Then we could have a *second package*, which presents a second protocol that subclases would use!

The second package IS for the second interface!

What would be the name for the second protocol? I.e. what is the interaction the base class and the subclass perform via this protocol?

I'm not sure what the name would be. If we were to adopt the practice I'm advocating, it would indeed be a good idea to have a standard naming practice.

If you look at some of the major Java libraries, you can see them dealing with that. For example, look at JNDI, the standard Java interface for dealing with any kind of naming service, particularly hierarchical ones such as file system directories and LDAP servers.

First, there is what they call the "API". This is used for modules that want to do things like "look up this name and tell me the contents" or "for this name, tell me all the children".

Then there is what they call the "SPI", the "Service Provider Interface". In the API, the calling module calls JNDI. For the "SPI", JNDI calls the service provider module. A service provider module is what is often called a "driver". You'd have one for file systems, one for LDAP, and so on. Java comes with some useful ones like those two, and you can add your own if you want to allow people who use the JNDI API to access some previously unsupported naming system.

This is very much like what I'm talking about. The API and the SPI are in different Java namespaces (well, as far as I know), and writing a module that calls the API is extremely different from writing a service provider.

The SPI might or might not work by overriding classes or adding :before methods or whatever.

In fact, the whole thing that I am advocating here is NOT ONLY for OOP and subclassing. I just presented it that way because that's where I see the main "pain point". What REALLY matters is the distinction between the API and the SPI.

As you point, it may be reusing of the implementation (derived streams reuse write-string implementation from the base class).

Looks like a thing intended for reusing can rightly be called an "external protocol" too.

Well, this is something different, I think; a utility class whose behavior has to be documented. But this is a side issue and this email is alarmingly long already. :)

Another view on the interaction in subclassing is that the base class

serves

as a template of some functionality, and we can customize its logic by overriding some methods.

In Java, however, the two are separated: there is an Interface that specifies the contract, and then there might or might not be a standard abstract base class that provides some helpful stuff. You are not required to use the utility abstract base class, although in practice I think I've always seen it used.

Java allows you to skip having the Interface, which is too bad from the point of view of clean and simple language semantics, and for clarity of code. It's a lot like CLOS's not requiring defgeneric, and probably for the same reason; to let you be lazy or more succinct. But I think it's less clear. I try to always use explicit defgenerics. In our own code here, we particularly stress using defgenerics for functions that are felt to be exposed to other modules. (I say "felt" because our code does not isolate modules as well as it ought to.)

The output stream example may be viewed in this perspective: we plug-in our write-character implementation into the functionality implemented by the base class in order to customise it to work with different character source.

Yes.

The means to parametrize/customize the protocol behavior can also be considered as a part of the protocol. In simple case it's method

parameters,

in more complex case we provide, for example, custom strategies, and

can do it

by defining a method for the sublcass.

OK.

Therefore it looks to me that in many cases we deal with a single

interface,

not two distinct ones.

Well, I think I've explained my point about that above.

Sometimes a module can interact with the outside world via several

protocols, but

it not necessary relates to sublcassing, and it's not necessary that

both (all)

the protocols are defined by this module.

Yes, indeed! That goes beyond the scope of what I've said so far. You can indeed have more than one external API, regardless of what you do with SPI's.

I believe Ada lets you do all of this. More about Ada and the experience with compatible upgrades, below.

...

Now, there's a great thing about CL packages: you can have more than one defpackage for the *same* symbols. \

Exactly. You can have more than one API, you can have a separate SPI, you can have multiple SPI's and so on. This is crucial to everything I am proposing. (By "proposing" I do not mean to claim that what I'm saying is novel and original. It may well have been done before. I'm not trying to take credit for anything here.)

It's a great feature. I think you mean something like when we want to

define one

protocol as an extension of another protocol. Although in many cases

I suppose

it's possible instead of extending interface to have a separate

interfaces. I.e.

if one API is (func1, func2, func3), and another API is (finc1,

func2, func3, extra-func),

we could have the second API just as (extra-func).

Exactly.

A good example is needed. Maybe something like database connection

API, where implementation

for particular database is provided by a pluggable "driver"; or maybe

swank, and swank

backends for different lisps; or something simpler.

Right, exactly, as I discussed above.

For your reading pleasure, here's is some stuff about Ada. It's from Tucker Taft, one of the world's foremost experts on Ada (no kidding). Note that Ada 95 is a later version of Ada, with improvements to the original Ada definition. Tucker was very heavily involved with the definition of Ada 95, for years.

The point he makes in the first paragraph corresponds to what I've been saying: the fact that "print" calls "to_string" is NOT apparent to the caller, but IS apparent to anyone writing a subclass.

----------------------------------------

Here is the mail from Tucker Taft. I have added a few comments in brackets [like this].

Hi Dan,

Ada 95 avoids some of the heartache associated with OOP subclassing. First a little background...

I remember going to an OOPSLA conference when we were just starting the work on Ada 95, and the panel was bemoaning the maintenance headaches they were facing. Someone like Microsoft would try to release a new version of some object-oriented library, and the users would scream bloody murder about incompatibilities. Microsoft would claim that all of the classes still had the same interface, and they just fiddled with the implementation. The problem was that in the implementation, there were calls from one operation to another, and by default in C++ (and Java and almost all OOP languages), those inter-operation calls would "redispatch" based on the run-time type of the operand. In C++ you can prevent this by using "class::operation", but that is more work and almost no one bothers.

[In other words, Microsoft provided compatibility for the "caller" interface but not the "subclass" interface. The people screaming were the subclassers, not the callers.]

So now what happens when you release a new version of the class library? Well for performance (or other reasons), some operations rather than calling other operations, now implement their functionality directly. A perfect example is a "print" operation and a "to_string" operation. It would make sense for the "print" operation to call the "to_string" operation and take the result and send it to the output device. But it might be more efficient to put the result directly to the output device one character at a time, and never bother creating a string. But what happens if someone had already "subclassed" your class and taken advantage of the fact that "print" called "to_string"? They quite likely would inherit the "print" operation as is, and only bother to re-implement "to_string". Now you release your new higher performance version with its great, more efficient "print" routine and bang, the subclassers code no longer works as expected (the inherited print routine no longer redispatches, so it only prints the contents of the parent class).

[What I'm proposing is that the fact that print calls to_string is documented. If a new release changes that, and says that print might or might not call to_string, then the subclassers have to address that. They would still be angry because they have to do extra work just because of Microsoft, but they would KNOW what work they need to do. (Assuming Microsoft documented everything properly and made clear the protocol (interface) that the subclassers see.]

So how did Ada 95 address this? It was pretty simple. Basically, when one operation of a class calls another, the default is a "static" binding -- there is no redispatch. You can explicitly force a redispatch, but that is more work, and presumably you only do that if you really want that. And if you are a good doobie, you will document that as part of the semantics of your operation.

Hence, in our earlier example, if you want "print" to work by redispatching to "to_string", then you work a little harder to force that, and you document that as part of the semantics of the "print" operation. If in a later version of the class library you want to provide a more efficient "print" operation, well clearly that has different semantics, and deserves a new name like "direct_print" or equivalent.

[I am 95% sure that what he's saying here, using different jargon than I do, is the same as what I am proposing.]

The net effect is that a subclasser can treat operations of a class in Ada 95 as "black boxes," in that you don't care whether one operation calls another to gets its job done, so long as the programmer didn't explicitly decide to "redispatch." If they did redispatch, then presumably they documented that fact, and you can safely take advantage of it. If they didn't redispatch, then for all of the operations, "what you see is what you get," so if you choose to inherit some and override others, you won't get any nasty surprises when a new release of the class library comes out.

I hope some of this makes sense!

[In exchange for Tucker's having provided us the service of sharing his experience above, I am going to pass on his solicitation for people to comment on his new work. If you don't want to read that, you can safely stop here. But I think it's cool stuff.]

By the way, some of these same issues came up in the work I have been doing designing "ParaSail" -- Parallel Specification and Implementation Language. I would love to come by some day and give you guys a short talk on ParaSail. It has some interesting stuff in it. I started a "blog" a year or so ago about the design process, and have started giving some talks in the past few months at conferences and to some local companies. Here is the blog:

http://parasail-programming-language.blogspot.com

If you are interested in seeing some slides from a talk I gave about it, see this page on our website:

http://www.sofcheck.com/news/sate_2010.html

The "blog" entry that most directly relates to this issue is:

http://parasail-programming-language.blogspot.com/2009/09/parasail-extension...

and here is the relevant paragraph:

One important thing to note about how ParaSail implementation inheritance works.

It is a completely black box. Each implicitly provided inherited operation calls the corresponding operation of the underlying interface, passing it the underlying objects of any operands of the type being defined. The underlying interface operation operates only on the underlying object(s), having no knowledge that it was called "on behalf" of some extended class.

If the underlying operation calls other operations, they too are operating only on the underlying object(s). There is no "redispatch" on these internal calls, so the extended class can treat these underlying operations as black boxes, and not worry that if it explicitly defines some operations while inheriting others, that that might somehow interact badly with how the underlying operations are implemented.

Take care, -Tuck