Anton,

Thank you very much for your mail. It prompted me to explain what I'm thinking of in a much more clear (I hope!!) way. Here we go:

Anton Vodonosov wrote:

What is the difference for modularity in case of subclassing, from other cases were we apply modularity? I can't find the difference.

Here's the greater idea.

If you have a module, by using CL packages (without any change to CL), it is possible to have more than one set of exports. So, you can offer different interfaces to different callers. By interfaces in this context I mean the set of symbols that are exported.

If you offer interface I1 to callers C1, C2, and C2, and interface I2 to callers C4, C5, and C6, then you can be sure that C1, C2, and C3 will keep working as long as you do not make incompatible changes to interface I1. Similarly with I2.

Now, moving specifically to OOP:

Traditionally, e.g. in Smalltalk 80, it was a big deal that the caller of an instance of a class does not know the internals of the class. It only uses methods, and does not look internally at the instance variables (slots).

(I can't remember whether ST-80 had the concept of private methods that are only intended to be used from methods of the class and not from callers.)

OOP was often compared to, or even conflated with, the concept of "abstract datatypes", in the sense meant by Barbara Liskov. In CLU and such languages, similarly there was a sharp distinction between the advertised interface presented to callers of the ADT (abstract datatype, corresponding to a class), and whatever was going on inside it.

This was a modularity boundary that allowed a separation of concerns between the users of the ADT and the implementation of the ADT.

Recently, this was considered such a big deal that Liskov was awarded a Turing Award.

ADT's did not, per se, contain the concept of inheritance. Inheritance was first exposed to the general public with ST-80.

However, there wasn't even a semblance of an effort to provide any modular separation between a class and its subclasses. So, whenever the implementation of a class changed in any way at all, you risked breaking subclasses. This was particularly a problem when library 1, under control of group-of-people-1, provided a class, and library-2 subclassed it. If a new release of library-1 came out, there was in general no way for g-o-p-1 to find everybody who had subclassed their classes; if the lib-2 people installed the new release of lib-1, they had no idea whether anythig would work.

Now, you could just say that there should be internals and externals, and the externals exposed to the callers and the externls exposed to the subclassers are exactly the same set.

This does not work for most non-trivial cases. I will assume that you know why, since I can't explain it briefly.

So, what I am proposing is to be explicit about what is being exposed to the subclassers, so that if the base class is changed in a new release but the subclasser interface is compatible, the lib-2 people can rest assured that their library will continue to work. (Unless there are bugs, of course.)

In C++ and Java, this isn't done with Lisp packages, of course. The "public" members are the interface to the caller, and the union of the "public" and "protected" members are the interface provided to the subclassers.

Whether the subclassers should be given permission to override the public things, in all cases, is an interesting but separate discussion. With Lisp packages, you can control this any way you want.

We may note that "protected" is actually public, in the sense that when we specify something as "protected", we specify that the library has external clients which can rely on this protected method or field.

In that sense, yes, absolutely.

Anyone may inherit from our public class, and access the "protected" members, therefore these members are available to public. If we change the protected member, we affect the library clients.

What I'm doing, however, is to distinguish between two sets of "public", as explained above. The callers, which would be a "big public" (lots of callers) must cope with incompatible changes relatively less often, whereas the "small public" (those who subclass) must cope relatively more often since more is revealed.

This drives me to a conclusion, that a single language feature is sufficient which allows to separate a protocol for interaction with a module from the module part we expect to change without affecting clients.

Taking you literally, we do not need a new CL feature. But translating what you're saying into the terms I am using, you're saying that we need only one protocol, whereas I am saying that it's better to have two. There is no "language feature" at the CL level, but there is a practice, or a "design pattern", that I am recommending. Having :documentation in the defpackage to explain what's going on, especially who is intended to use this packge, would be very hepful. Design patterns are essentially a way to extend a language (in a nutshell).

I made observation on many java systems, that addition to packages the public/protected/private for classes has bad influence on the system modularity.

Many java programmers tend to hide attributes of every single class behind get/set accessors, even for simple classes which just hold values passed between methods and do not contain any logic; to create hierarchies of interface/abstract class/concrete for various minor classes, employing protected methods.

There are at least two reasons to put in such methods.

First, changing the implementation while keeping compatible behavior for the public methods.

Imagine that you have a class called Point representing a point in two-space. You could use Cartesian representation: two data members x and y, and getX and getY (and, if desired, setX and setY). These are the simple getters (and setters) that you mention above.

But what if, for some reason (numerical methods, I don't know) you decide that internally using a polar representation, with two data members storing an angle and a distance from the origin, is better. You can make getX continue to work using the appropriate trigonometry operations.

Second, these days there are lots of tools that use "dependency injection" and whatnot that only work if you have such methods. Extending "ant", using Spring, and so on work this way, so it has become part of the usual Java design pattern for many classes. (See "Java Bean".)

But at the same time, people do not care to create structure at larger level, to divide the system into modules. It is not uncommon to see large systems where all these classes, subclasses in all the packages are public: everything is available to everything.

Sure.

Separate access control for classes obscures the idea of modularity (especially in case of such an OO centric language like java, the programmers are mostly concentrated on classes/object, and often don't even think about packages;).

I don't know what you mean by this. It depends what you mean by "access control". But if you mean limiting the access that callers of a module have, that is exactly the way modularity is usually realized.

Class is too small entity to form a module. A module API (protocol) usually consists of set of classes, methods, factory functions, maybe some global variables.

Ah, yes. That's why Common Lisp has packages and Java has namespaces; to be able to make groups exactly as you are saying.

Another thought is that the interface between base class and its subclasses is not a second interface, but a part of the first interface. As you said:

...

So we could have one package for the code that calls into our library, which would have the usual protocol, which we can call the "external protocol". Then we could have a *second package*, which presents a second protocol that subclases would use!

The second package IS for the second interface!

What would be the name for the second protocol? I.e. what is the interaction the base class and the subclass perform via this protocol?

I'm not sure what the name would be. If we were to adopt the practice I'm advocating, it would indeed be a good idea to have a standard naming practice.

If you look at some of the major Java libraries, you can see them dealing with that. For example, look at JNDI, the standard Java interface for dealing with any kind of naming service, particularly hierarchical ones such as file system directories and LDAP servers.

First, there is what they call the "API". This is used for modules that want to do things like "look up this name and tell me the contents" or "for this name, tell me all the children".

Then there is what they call the "SPI", the "Service Provider Interface". In the API, the calling module calls JNDI. For the "SPI", JNDI calls the service provider module. A service provider module is what is often called a "driver". You'd have one for file systems, one for LDAP, and so on. Java comes with some useful ones like those two, and you can add your own if you want to allow people who use the JNDI API to access some previously unsupprted naming system.

This is very much like what I'm talking about. The API and the SPI are in different Java namespaces (well, as far as I know), and writing a module that calls the API is extremely different from writing a service provider.

The SPI might or might not work by overriding classes or adding :before methods or whatever.

In fact, the whole thing that I am advocating here is NOT ONLY for OOP and subclassing. I just presented it that way because that's where I see the main "pain point". What REALLY matters is the distinction between the API and the SPI.

As you point, it may be reusing of the implementation (derived streams reuse write-string implementation from the base class).

Looks like a thing intended for reusing can rightly be called an "external protocol" too.

Well, this is something different, I think; a utility class whose behavior has to be documented. But this is a side issue and this email is alarmingly long already. :)

Another view on the interaction in subclassing is that the base class serves as a template of some functionality, and we can customize its logic by overriding some methods.

In Java, however, the two are separated: there is an Interface that specifies the contract, and then there might or might not be a standard abstract base class that provides some helpful stuff. You are not required to use the utility abstract base class, although in practice I think I've always seen it used.

Java allows you to skip having the Interface, which is too bad from the point of view of clean and simple language semantics, and for clarity of code. It's a lot like CLOS's not requiring defgeneric, and probably for the same reason; to let you be lazy or more succinct. But I think it's less clear. I try to always use explicit defgenerics. In our own code here, we particularly stress using defgenerics for functions that are felt to be exposed to other modules. (I say "felt" because our code does not isolate modules as well as it ought to.)

The output stream example may be viewed in this perspective: we plug-in our write-character implementation into the functionality implemented by the base class in order to customise it to work with different character source.

Yes.

The means to parametrize/customize the protocol behavior can also be considered as a part of the protocol. In simple case it's method parameters, in more complex case we provide, for example, custom strategies, and can do it by defining a method for the sublcass.

OK.

Therefore it looks to me that in many cases we deal with a single interface, not two distinct ones.

Well, I think I've explained my point about that above.

Sometimes a module can interact with the outside world via several protocols, but it not necessary relates to sublcassing, and it's not necessary that both (all) the protocols are defined by this module.

Yes, indeed! That goes beyond the scope of what I've said so far. You can indeed have more than one external API, regardless of what you do with SPI's.

I belive Ada lets you do all of this. More about Ada and the experience with compatible upgrades, below.

...

Now, there's a great thing about CL packages: you can have more than one defpackage for the *same* symbols. \

Exactly. You can have more than one API, you can have a separate SPI, you can have multiple SPI's and so on. This is crucial to everything I am proposing. (By "proposing" I do not mean to claim that what I'm saying is novel and original. It may well have been done before. I'm not trying to take credit for anything here.)

It's a great feature. I think you mean something like when we want to define one protocol as an extension of another protocol. Although in many cases I suppose it's possible instead of extending interface to have a separate interfaces. I.e. if one API is (func1, func2, func3), and another API is (finc1, func2, func3, extra-func), we could have the second API just as (extra-func).

Exactly.

A good example is needed. Maybe something like database connection API, where implementation for particular database is provided by a pluggable "driver"; or maybe swank, and swank backends for different lisps; or something simpler.

Right, exactly, as I discussed above.

For your reading pleasure, here's is some stuff about Ada. It's from Tucker Taft, one of the world's foremost experts on Ada (no kidding). Note that Ada 95 is a later version of Ada, with improvements to the original Ada definition. Tucker was very heavily involved with the definition of Ada 95, for years.

The point he makes in the first paragraph corresponds to what I've been saying: the fact that "print" calls "to_string" is NOT apparent to the caller, but IS apparent to anyone writing a subclass.

----------------------------------------