Re: [usocket-devel] Portably supporting :external-format
So, to stay backward compatible, this is what I'd like to do:
- If no external-format is given: just return the raw stream returned by the implementation (as we do now) - If an external-format is specified, create a raw stream with 'octet' element-type and wrap the returned stream with a flexi-stream, exposing the flexi-stream to the caller.
How about that?
Flexi-streams depends on trivial-gray-streams, which appears to support the following implementations:
- Allegro - CMUCL - LispWorks - OpenMCL - CLISP - SBCL
It doesn't appear to support ABCL or Scieneer CL. For that reason, we can't simply make it a mandatory dependency, unless we want to give up on those implementations. How do you propose we make it optional?
Well, there's another option: I talked to David Lichteblau and he will accept patches to support more implementations. I already had ABCL installed on my system, so I've been working to add ABCL support today. ABCL users probably won't want to use the facility, because (as the authors say themselves) "CLOS is dead slow" and Gray streams use CLOS... As far as Scieneer is concerned, well given their prices, I won't be able to afford to have an installation, but maybe Douglas Crosher (who I CC'ed in with this message) can provide a patch to trivial-gray-streams too?
One idea that comes to mind is to only use flexi-streams if it has already been loaded when package.lisp is loaded. If it has, we set a *use-flexi-streams* variable (or something), and then test that whenever we need to deal with :external-format. In addition, we could expose this variable to the user, allowing them to test or set it.
Well, I'd like to check and see if it's possible to extend trivial-gray-streams (and possibly flexi-streams) to support our use-case. Looking at ABCL and flexi-streams though, I do see a problem: ABCL doesn't have a generic function close, but defines STREAM-CLOSE instead... At least flexi-streams 0.5.10 seems to depend on close, open-stream-p and maybe others too, to be generic functions, not just defuns... Well, ABCL isn't in wide use anyway and not going very fast development, but, well, I could ask around with the developers what to expect there. It would indeed be too bad not to be able to support :external-format there, but that's an option too (just ignore the external-format in case of ABCL). We need to make sure to document that though. bye, Erik. PS: Douglas, do you have interface documentation for Scieneer Gray streams somewhere for me? Or could you have a look at trivial-gray-streams (and flexi-streams) and tell us if it's possible to add code for Scieneer to them?
On 10/22/06, Erik Huelsmann <ehuels@gmail.com> wrote:
Looking at ABCL and flexi-streams though, I do see a problem: ABCL doesn't have a generic function close, but defines STREAM-CLOSE instead... At least flexi-streams 0.5.10 seems to depend on close, open-stream-p and maybe others too, to be generic functions, not just defuns...
Well, ABCL isn't in wide use anyway and not going very fast development, but, well, I could ask around with the developers what to expect there. It would indeed be too bad not to be able to support :external-format there, but that's an option too (just ignore the external-format in case of ABCL). We need to make sure to document that though.
That sounds good to me. One minor concern, though: does it make sense to add a whole new mandatory dependency (two, in fact) for a relatively obscure feature like this? Now, I realize I'm backpedaling here, but how about we require the user, if they want to use flexi-streams, to just create one themselves from the raw streams that we return? This is a trade-off: it makes things easier for people who don't need :external-format or bivalent streams, but it makes things harder for people who do. It would also mean that the :external-format flag to OPEN-STREAM would remain useless, but, in trivial-sockets itself, it only works under CLISP, anyway. Anyway, feel free to ignore me; your way would work fine, as well. I'm just throwing out alternatives here.
On 10/23/06, nick thomas <jesuswaffle@gmail.com> wrote:
On 10/22/06, Erik Huelsmann <ehuels@gmail.com> wrote:
Looking at ABCL and flexi-streams though, I do see a problem: ABCL doesn't have a generic function close, but defines STREAM-CLOSE instead... At least flexi-streams 0.5.10 seems to depend on close, open-stream-p and maybe others too, to be generic functions, not just defuns...
Well, ABCL isn't in wide use anyway and not going very fast development, but, well, I could ask around with the developers what to expect there. It would indeed be too bad not to be able to support :external-format there, but that's an option too (just ignore the external-format in case of ABCL). We need to make sure to document that though.
That sounds good to me.
Anyway, feel free to ignore me; your way would work fine, as well. I'm just throwing out alternatives here.
Oh! No! I'm not ignoring you! Having these discussions with myself is a lot harder than having them with you ;-) I do have my own opinions, but talking about pros and cons is a lot better than hitting a wall. You're really helping usocket forward.
One minor concern, though: does it make sense to add a whole new mandatory dependency (two, in fact) for a relatively obscure feature like this?
I think that's funny: :external-format should have become a major issue for all lisp implementations by now. i18n is really a 'must' for any self-respecting application these days. So, any platform worth developing on should support it.
Now, I realize I'm backpedaling here, but how about we require the user, if they want to use flexi-streams, to just create one themselves from the raw streams that we return?
Well, that does simplify matters: if we do that, we only need to support :element-type to make sure people can create the binary streams which flexi-streams needs for its input.
This is a trade-off: it makes things easier for people who don't need :external-format or bivalent streams, but it makes things harder for people who do. It would also mean that the :external-format flag to OPEN-STREAM would remain useless, but, in trivial-sockets itself, it only works under CLISP, anyway.
I do like Douglas's idea too: those implementations which do not support external-format or bivalent streams, will require flexi-streams. Implementations which *do* support that can use their internal system. However that would loose portable external format specifications. It should be quite trivial to write a mapping from a general spec to implementation defined ones though. The other thing we will probably loose is bivalence. On IRC I was told SBCL will allow stream-read-byte/read-byte on character streams, effectively making them bivalent, but I'm not sure for any of the other implementations. There is (in contrast with external-format) no (setf element-type) method in the standard, so this behaviour will be implementation specific... Maybe we can find out by creating a simple test-case to run on all supported platforms and see if each platform effectively provides bivalent streams? bye, Erik. PS: I created a generic function in ABCL and it took a measurable amount of time on my P4-3Ghz 1GB RAM, so I indeed think the ABCL developers are right: ABCL is dead slow.
Erik Huelsmann wrote:
So, to stay backward compatible, this is what I'd like to do:
- If no external-format is given: just return the raw stream returned by the implementation (as we do now) - If an external-format is specified, create a raw stream with 'octet' element-type and wrap the returned stream with a flexi-stream, exposing the flexi-stream to the caller.
How about that?
Flexi-streams depends on trivial-gray-streams, which appears to support the following implementations:
- Allegro - CMUCL - LispWorks - OpenMCL - CLISP - SBCL
It doesn't appear to support ABCL or Scieneer CL. For that reason, we can't simply make it a mandatory dependency, unless we want to give up on those implementations. How do you propose we make it optional?
Could you simply add :external-format and :element-type arguments to the 'socket-connection function and allow implementations to return a suitable stream? The Scieneer CL has a well designed and fast stream layer capable of supporting web services with many tens of thousands of connections, and the function 'ext:make-fd-stream already supports character streams in Unicode encodings plus a range of handy extensions such a compressed and encrypted streams. Flexi-streams is not suitable as a de facto standard - the design is just too slow to be practical. If implementations really need to use this then allow them to do so in the usocket backend. Regards Douglas Crosher
participants (3)
-
Douglas Crosher -
Erik Huelsmann -
nick thomas