[snipped some context]

...

Yes, avoiding portability layers is a goal.. so when you state " it is just a matter of creating a small layer on top of them (like usocket)", we're directly against that goal. usocket and the like exist already, so there's not much value in working to standardize the interface... just use usocket! :)

Well, maybe I didn't expressed myself clearly. And I didn't quite get your point. If you go down that road, we could say that we don't need to standardize FFI or threads as well ("just use CFFI or bordaux-threads" - usocket is just as much of a portability layer as CFFI or bordeaux-threads is). In the part of "just create a small layer on top of them", I meant internally in each implementation. Each implementation will take the functions to manipulate sockets that it already has and create the standard functions on top of them or make small modifications appropriately to make those functions conform with the standard.

Right, i getcha, and i don't necessarily disagree. But, why require that an implementation have sockets when you don't need them to be implementation specific?

Sure, given sockets you can create a crude FFI, but it will not be fast or pretty. However, the socket interface i can create via FFI and extensible streams is just as good, if not better, than those provided by individual implementations.

What if i want more functionality for our sockets then those provided by an implementation? do i ask the implementation to implement my extension? Again, if we try to force implementors to do anything, we've lost. We need to gain traction and momentum first.

The point i'm trying to make is that including sockets and networking in the CLtL3 specification (it's not a standard, lets not call it one) will require work, and potentially a lot of work, on both our parts and the parts of implementors. I see a good argument for excluding them (a perfectly good implementation can be built on the features we provide), but i've yet to hear a good argument for providing them (in the description rather than the library) beyond 'because some implementations have sockets already'.

This is what I believe that will be done by implementations to conform FFI, thread and any other standard we create here, isn't it?

That remains to be seen.. this is not ANSI, is not funded by ARPA, and has little chance of getting anywhere if we start making demands of implementors. Keep that in mind... nobody has to listen to anything we come up with. It's our job to make it acceptable.

...

I'm not sure what you mean by 'changing the functions that deal with them' in this context, but a goal of cltl3 (i will add this to the charter) is to require as little effort on the part of implementors as possible. In the case of sockets and networking, we can provide library code from a single source... so why duplicate that effort across every distribution or implementation of lisp? Seems like a wasted effort and a barrier to adoption from my POV.

My point is that every implementation already support sockets, it should be much better if there was a standard interface so they could implement socket interface uniformly across implementations.

what do you mean by 'standard' and who is the 'they' that will implement this interface? If the they is us (which is the only they it could be), then FFI and Extensible Streams are all we require from our implementations. If the they is implementors, who will pay, who will convince them it's worthwhile, etc? This is where the main thrust of my argument lies. We don't need sockets supported by an implementation if we're given the proper tools to write them ourselves.

What difference does it make if they are in CLtL3 or in the 'standard library'?

...

It is possible, given FFI and gray streams, to implement sockets without using _any_ implementation defined socket code. Rather than a compatibility layer that attempts to paper over the differences in implementations (trouble with such layers being a large motivation for the creation of CLtL3), i'd prefer a single canonical implementation based on constructs provided by CLtL3.

I believe it would also be possible to implement multiple threading that way (using FFI, if it support callbacks (as CFFI does)). But I'm not going further on this topic.

You'd better not, because that is demonstrably false. Threading requires implementation support.. think of the simple case of two threads mutzing about with a hash table. Either you lock the world (then why bother with threads!), or the implementation must have primitives that support the notion of multiple strands of execution in order to avoid race conditions and data corruption.

...

The prior art here (as for a lot of things we will be discussing) is IOlib. IOlib uses its own implementation of sockets and networking based on the OS's underlying implementation (through FFI). It is mostly portable and uniform across implementations, and does not require any support for networking in the implementation itself (fe[nl]ix had better correct me if i'm wrong).

Using IOlib has its drawbacks. It is a big library, one more dependency and it will only add functionality that is already present in all Lisp implementations.

All lisp implementations? are you sure that all lisp implementations support all that's in IOlib? Hint: they don't, and that's part of the reason we're doing CLtL3 :).

It also uses grays streams, which are not optimized streams (so implementation specific's sockets should be faster for good implementations). And it uses FFI, which is a barrier when you need to save and load lisp images.

Hence a good reason for describing a FFI and an extensible stream implementation (with a specific stated goal of supporting sockets and networking, on those platforms where they are available). FFI is hardly a barrier to saving images... just save the .so in the image as an array and write it out at run time if you can't find it.

If you think that implementors are going to go ahead and code to anything we can come up with, you are barking up the wrong tree. However, if we define a minimal set of features as 'CLtL3', and provide a reference implementation, there is a chance that it might gain hold, and users might demand CLtL3 compliance from vendors.

...

I'm still not dismissing inclusion of networking and sockets, but i've not heard a convincing argument for them, and i think our time would be better spent elsewhere.

I still can't really see why standardizing sockets is so different from other topics that were included.

You cannot extend sequences without breaking ANSI conformance. You cannot have FFI and extensible streams without implementation support. Threads require runtime support. Lisps should have some access to the OS and filesystem in order to support things like ASDF, file compilation, etc.

Sockets and networking, OTOH, can be implemented without implementation support.

If we implement it as a library, an implementation is free to clone the API using fast native streams etc. If we try to force something on implementors, we will get nowhere.

I agree with you when you say our time will be better spent in other subjects for now, but I really think we should discuss this at some point. Lack of standard sockets is one of the topics that are most oftenly mentioned when the subject is "what are CL's negative points".

Let me reiterate the point that CLtL3 is not a 'standard', and we will not be standardising anything. CLtL3 is merely a description of features that modern lisps should provide in order to support our eventual standard library.

Well, I guess I am the one playing the devil's advocate now :)

It goes back and forth :). Discussion is good!

By the way, a few more topics that can be discussed:

• Features (as in trivial-features - e.g. when should x86-64 or x86 or

darwin features be provided)

Excellent.. it's a simple one that needs little discussion but has a lot of benefit,

• (Per-package ?) Readtables (I believe we really should discuss about

readtables, it would avoid many of the hacks that are needed for using read macros - clsql is the canonical example)

I can't believe i forgot readtables! It'll be on the next draft for sure.

• Declarations and other optimizations - For instance, ECL has a declaration

to freeze some generic function to avoid redefinition or including new methods, which is used for optimization. Freezing functions and types may be useful as well. Something like a standard sb-ext:defglobal, including sb-ext:global and sb-ext:always-bound proclamations, are welcome. I don't know if these are good ideas, but it would be very simple and easy to both include and implement (since actually using the declarations is optional, and defglobal can be implemented as an usual defvar).

I'll add this to the list as well, in some form or another.

Thanks for all the feedback! Your input has been quite valuable so far.

Cheers,

drewc

---

cltl3-devel mailing list cltl3-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cltl3-devel