Re: [cltl3-devel] RFC: CLtL3 Charter
2009/9/1 Drew Crampsie <drewc@tech.coop>
2009/9/1 Gustavo <gugamilare@gmail.com>:
2009/9/1 Drew Crampsie <drewc@tech.coop>
2009/9/1 Gustavo <gugamilare@gmail.com>:
Hello,
I also like to keep things simple. Here are a few suggestions, though.
In paragraph 4, you didn't mention "sockets" explicitly in the list. I don't know if that is intended to be included in "Networking", because Unix sockets are local to the computer.
To be quite honest, i don't know if CLtL3 actually needs sockets or networking. If we have FFI and extensible streams, we can build sockets and networking as library code.. right? I'm interested in hearing dissent on this one.
Well, yes, but all implementations that I know about have some interface
sockets. So, I believe that this is a very small task for implementors, it is just a matter of creating a small layer on top of them (like usocket) or changing the functions that deal with them. Isn't Cltl3 meant to avoid
to the
need for portability layers?
Yes, avoiding portability layers is a goal.. so when you state " it is just a matter of creating a small layer on top of them (like usocket)", we're directly against that goal. usocket and the like exist already, so there's not much value in working to standardize the interface... just use usocket! :)
Well, maybe I didn't expressed myself clearly. And I didn't quite get your point. If you go down that road, we could say that we don't need to standardize FFI or threads as well ("just use CFFI or bordaux-threads" - usocket is just as much of a portability layer as CFFI or bordeaux-threads is). In the part of "just create a small layer on top of them", I meant internally in each implementation. Each implementation will take the functions to manipulate sockets that it already has and create the standard functions on top of them or make small modifications appropriately to make those functions conform with the standard. This is what I believe that will be done by implementations to conform FFI, thread and any other standard we create here, isn't it?
I'm not sure what you mean by 'changing the functions that deal with them' in this context, but a goal of cltl3 (i will add this to the charter) is to require as little effort on the part of implementors as possible. In the case of sockets and networking, we can provide library code from a single source... so why duplicate that effort across every distribution or implementation of lisp? Seems like a wasted effort and a barrier to adoption from my POV.
My point is that every implementation already support sockets, it should be much better if there was a standard interface so they could implement socket interface uniformly across implementations.
It is possible, given FFI and gray streams, to implement sockets without using _any_ implementation defined socket code. Rather than a compatibility layer that attempts to paper over the differences in implementations (trouble with such layers being a large motivation for the creation of CLtL3), i'd prefer a single canonical implementation based on constructs provided by CLtL3.
I believe it would also be possible to implement multiple threading that way (using FFI, if it support callbacks (as CFFI does)). But I'm not going further on this topic.
The prior art here (as for a lot of things we will be discussing) is IOlib. IOlib uses its own implementation of sockets and networking based on the OS's underlying implementation (through FFI). It is mostly portable and uniform across implementations, and does not require any support for networking in the implementation itself (fe[nl]ix had better correct me if i'm wrong).
Using IOlib has its drawbacks. It is a big library, one more dependency and it will only add functionality that is already present in all Lisp implementations. It also uses grays streams, which are not optimized streams (so implementation specific's sockets should be faster for good implementations). And it uses FFI, which is a barrier when you need to save and load lisp images.
Maybe we could make the support for sockets optional for each implementation, but standardize the interface for sockets for implementations that want to provide them (something like "These functions may signal an error if the implementations doesn't have support for sockets.").
This is exactly what i'd like to avoid.
Fair enough.
[snipped]
I'm still not dismissing inclusion of networking and sockets, but i've not heard a convincing argument for them, and i think our time would be better spent elsewhere.
I still can't really see why standardizing sockets is so different from other topics that were included. I agree with you when you say our time will be better spent in other subjects for now, but I really think we should discuss this at some point. Lack of standard sockets is one of the topics that are most oftenly mentioned when the subject is "what are CL's negative points". Well, I guess I am the one playing the devil's advocate now :)
[snipped bit about hash tables]
I believe that we could discuss it later, when we discuss the CDR documents (we are going to do that, aren't we?).
You bet. :)
Cheers,
drewc
By the way, a few more topics that can be discussed: * Features (as in trivial-features - e.g. when should x86-64 or x86 or darwin features be provided) * (Per-package ?) Readtables (I believe we *really* should discuss about readtables, it would avoid many of the hacks that are needed for using read macros - clsql is the canonical example) * Declarations and other optimizations - For instance, ECL has a declaration to freeze some generic function to avoid redefinition or including new methods, which is used for optimization. Freezing functions and types may be useful as well. Something like a standard sb-ext:defglobal<http://www.sbcl.org/manual/Global-and-Always_002dBound-variables.html#Global-and-Always_002dBound-variables>, including sb-ext:global and sb-ext:always-bound proclamations, are welcome. I don't know if these are good ideas, but it would be very simple and easy to both include and implement (since actually using the declarations is optional, and defglobal can be implemented as an usual defvar).
On 1-Sep-09, at 11:41 PM, Gustavo wrote:
I believe it would also be possible to implement multiple threading that way (using FFI, if it support callbacks (as CFFI does)). But I'm not going further on this topic.
No. Even if it were technically possible to implement threading (and strictly that) portably using FFI, threads are not a library feature. Threads are a language feature. For threads to be useful and trustworthy, you must fully specify a boat load of things, from atomicness of stores to memory barriers and observable store/load ordering. Paul Khuong
[snipped some context]
Yes, avoiding portability layers is a goal.. so when you state " it is just a matter of creating a small layer on top of them (like usocket)", we're directly against that goal. usocket and the like exist already, so there's not much value in working to standardize the interface... just use usocket! :)
Well, maybe I didn't expressed myself clearly. And I didn't quite get your point. If you go down that road, we could say that we don't need to standardize FFI or threads as well ("just use CFFI or bordaux-threads" - usocket is just as much of a portability layer as CFFI or bordeaux-threads is). In the part of "just create a small layer on top of them", I meant internally in each implementation. Each implementation will take the functions to manipulate sockets that it already has and create the standard functions on top of them or make small modifications appropriately to make those functions conform with the standard.
Right, i getcha, and i don't necessarily disagree. But, why require that an implementation have sockets when you don't need them to be implementation specific? Sure, given sockets you can create a crude FFI, but it will not be fast or pretty. However, the socket interface i can create via FFI and extensible streams is just as good, if not better, than those provided by individual implementations. What if i want more functionality for our sockets then those provided by an implementation? do i ask the implementation to implement my extension? Again, if we try to force implementors to do anything, we've lost. We need to gain traction and momentum first. The point i'm trying to make is that including sockets and networking in the CLtL3 specification (it's not a standard, lets not call it one) will require work, and potentially a lot of work, on both our parts and the parts of implementors. I see a good argument for excluding them (a perfectly good implementation can be built on the features we provide), but i've yet to hear a good argument for providing them (in the description rather than the library) beyond 'because some implementations have sockets already'.
This is what I believe that will be done by implementations to conform FFI, thread and any other standard we create here, isn't it?
That remains to be seen.. this is not ANSI, is not funded by ARPA, and has little chance of getting anywhere if we start making demands of implementors. Keep that in mind... nobody has to listen to anything we come up with. It's our job to make it acceptable.
I'm not sure what you mean by 'changing the functions that deal with them' in this context, but a goal of cltl3 (i will add this to the charter) is to require as little effort on the part of implementors as possible. In the case of sockets and networking, we can provide library code from a single source... so why duplicate that effort across every distribution or implementation of lisp? Seems like a wasted effort and a barrier to adoption from my POV.
My point is that every implementation already support sockets, it should be much better if there was a standard interface so they could implement socket interface uniformly across implementations.
what do you mean by 'standard' and who is the 'they' that will implement this interface? If the they is us (which is the only they it could be), then FFI and Extensible Streams are all we require from our implementations. If the they is implementors, who will pay, who will convince them it's worthwhile, etc? This is where the main thrust of my argument lies. We don't need sockets supported by an implementation if we're given the proper tools to write them ourselves. What difference does it make if they are in CLtL3 or in the 'standard library'?
It is possible, given FFI and gray streams, to implement sockets without using _any_ implementation defined socket code. Rather than a compatibility layer that attempts to paper over the differences in implementations (trouble with such layers being a large motivation for the creation of CLtL3), i'd prefer a single canonical implementation based on constructs provided by CLtL3.
I believe it would also be possible to implement multiple threading that way (using FFI, if it support callbacks (as CFFI does)). But I'm not going further on this topic.
You'd better not, because that is demonstrably false. Threading requires implementation support.. think of the simple case of two threads mutzing about with a hash table. Either you lock the world (then why bother with threads!), or the implementation must have primitives that support the notion of multiple strands of execution in order to avoid race conditions and data corruption.
The prior art here (as for a lot of things we will be discussing) is IOlib. IOlib uses its own implementation of sockets and networking based on the OS's underlying implementation (through FFI). It is mostly portable and uniform across implementations, and does not require any support for networking in the implementation itself (fe[nl]ix had better correct me if i'm wrong).
Using IOlib has its drawbacks. It is a big library, one more dependency and it will only add functionality that is already present in all Lisp implementations.
All lisp implementations? are you sure that all lisp implementations support all that's in IOlib? Hint: they don't, and that's part of the reason we're doing CLtL3 :).
It also uses grays streams, which are not optimized streams (so implementation specific's sockets should be faster for good implementations). And it uses FFI, which is a barrier when you need to save and load lisp images.
Hence a good reason for describing a FFI and an extensible stream implementation (with a specific stated goal of supporting sockets and networking, on those platforms where they are available). FFI is hardly a barrier to saving images... just save the .so in the image as an array and write it out at run time if you can't find it. If you think that implementors are going to go ahead and code to anything we can come up with, you are barking up the wrong tree. However, if we define a minimal set of features as 'CLtL3', and provide a reference implementation, there is a chance that it might gain hold, and users might demand CLtL3 compliance from vendors.
I'm still not dismissing inclusion of networking and sockets, but i've not heard a convincing argument for them, and i think our time would be better spent elsewhere.
I still can't really see why standardizing sockets is so different from other topics that were included.
You cannot extend sequences without breaking ANSI conformance. You cannot have FFI and extensible streams without implementation support. Threads require runtime support. Lisps should have some access to the OS and filesystem in order to support things like ASDF, file compilation, etc. Sockets and networking, OTOH, can be implemented without implementation support. If we implement it as a library, an implementation is free to clone the API using fast native streams etc. If we try to force something on implementors, we will get nowhere.
I agree with you when you say our time will be better spent in other subjects for now, but I really think we should discuss this at some point. Lack of standard sockets is one of the topics that are most oftenly mentioned when the subject is "what are CL's negative points".
Let me reiterate the point that CLtL3 is not a 'standard', and we will not be standardising anything. CLtL3 is merely a description of features that modern lisps should provide in order to support our eventual standard library.
Well, I guess I am the one playing the devil's advocate now :)
It goes back and forth :). Discussion is good!
By the way, a few more topics that can be discussed:
* Features (as in trivial-features - e.g. when should x86-64 or x86 or darwin features be provided)
Excellent.. it's a simple one that needs little discussion but has a lot of benefit,
* (Per-package ?) Readtables (I believe we really should discuss about readtables, it would avoid many of the hacks that are needed for using read macros - clsql is the canonical example)
I can't believe i forgot readtables! It'll be on the next draft for sure.
* Declarations and other optimizations - For instance, ECL has a declaration to freeze some generic function to avoid redefinition or including new methods, which is used for optimization. Freezing functions and types may be useful as well. Something like a standard sb-ext:defglobal, including sb-ext:global and sb-ext:always-bound proclamations, are welcome. I don't know if these are good ideas, but it would be very simple and easy to both include and implement (since actually using the declarations is optional, and defglobal can be implemented as an usual defvar).
I'll add this to the list as well, in some form or another. Thanks for all the feedback! Your input has been quite valuable so far. Cheers, drewc
_______________________________________________ cltl3-devel mailing list cltl3-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cltl3-devel
If you think that implementors are going to go ahead and code to anything we can come up with, you are barking up the wrong tree. However, if we define a minimal set of features as 'CLtL3', and provide a reference implementation, there is a chance that it might gain hold, and users might demand CLtL3 compliance from vendors.
Slightly off topic, but if I might enquire.. when you say provide provide a reference implementation, does that mean picking one of the open source CL implementations and developing it to include everything that CLtL3 specifies? Malcolm
2009/9/2 Drew Crampsie <drewc@tech.coop>
[snipped some context]
Yes, avoiding portability layers is a goal.. so when you state " it is just a matter of creating a small layer on top of them (like usocket)", we're directly against that goal. usocket and the like exist already, so there's not much value in working to standardize the interface... just use usocket! :)
Well, maybe I didn't expressed myself clearly. And I didn't quite get your point. If you go down that road, we could say that we don't need to standardize FFI or threads as well ("just use CFFI or bordaux-threads" - usocket is just as much of a portability layer as CFFI or bordeaux-threads is). In the part of "just create a small layer on top of them", I meant internally in each implementation. Each implementation will take the functions to manipulate sockets that it already has and create the standard functions on top of them or make small modifications appropriately to make those functions conform with the standard.
Right, i getcha, and i don't necessarily disagree. But, why require that an implementation have sockets when you don't need them to be implementation specific?
Sure, given sockets you can create a crude FFI, but it will not be fast or pretty. However, the socket interface i can create via FFI and extensible streams is just as good, if not better, than those provided by individual implementations.
What if i want more functionality for our sockets then those provided by an implementation? do i ask the implementation to implement my extension? Again, if we try to force implementors to do anything, we've lost. We need to gain traction and momentum first.
The point i'm trying to make is that including sockets and networking in the CLtL3 specification (it's not a standard, lets not call it one) will require work, and potentially a lot of work, on both our parts and the parts of implementors. I see a good argument for excluding them (a perfectly good implementation can be built on the features we provide), but i've yet to hear a good argument for providing them (in the description rather than the library) beyond 'because some implementations have sockets already'.
All right, I guess you won this discussion. I won't die for not having a per-implementation interface for sockets. I guess I did not had in mind that implementors might or might not like Cltl3 and choose to comply with it or not to care at all. But, in any case, as I see in usocket's page, all currently alive implementations that I am aware of (except perhaps Corman CL) have at least the basics for sockets.
[snipped]
Thanks for all the feedback! Your input has been quite valuable so far.
You are welcome. I'm just having some fun. There are still a bunch of minor things that could be be included in Cltl3, but I don't know yet how much effort this might take, so this is just a suggestion. Hooks (hooks for when the gc runs, for when Lisp initializes and finishes (useful for when you save and load images and need to do some setup or teardown) and possibly others). There are also functions like quitting lisp, running an external program, access to a shell (like trivial-shell<http://common-lisp.net/project/trivial-shell/>) a way of getting the so-called "posix-argv". The last three might fall in the "OS and Filesystem access" topic. Cheers, Gustavo.
On Wednesday 02 September 2009 07:41:15 Gustavo wrote:
I believe it would also be possible to implement multiple threading that way (using FFI, if it support callbacks (as CFFI does)). But I'm not going further on this topic.
It is certainly not possible to implement multithreading with FFI. Multithreading requires substantial support from the runtime.
On Sep 2, 2009, at 19:11 , Kalyanov Dmitry wrote:
On Wednesday 02 September 2009 07:41:15 Gustavo wrote:
I believe it would also be possible to implement multiple threading that way (using FFI, if it support callbacks (as CFFI does)). But I'm not going further on this topic.
It is certainly not possible to implement multithreading with FFI. Multithreading requires substantial support from the runtime.
Do you mean the underlying the OS? Cheers Marco -- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 Viale Sarca 336 I-20126 Milan (MI) ITALY Please note that I am not checking my Spam-box anymore.
participants (6)
-
Drew Crampsie -
Gustavo -
Kalyanov Dmitry -
Malcolm Reynolds -
Marco Antoniotti -
Paul Khuong