[flexi-streams-devel] in-memory streams have no way to set external-format?
Hi all, I would like to teach cl-irc about the goodness of external-formats, and because irc doesn't let users specify which external format to use, I need something as flexible as flexi-streams (-: So, once I've read an irc command (as a "line" of latin-1 characters), I'd like to convert it to text that looks the least insane, which is why my code tries several external-formats in a row and returns as soon as it found one that doesn't throw an error: (defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (flexi-streams:with-input-from-sequence (in line) (setf (flexi-streams:flexi-stream-external-format in) external-format) (read-line in)))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded))) Try it with: (try-decode-line "foo" '((:UTF-8 :EOL-STYLE :LF) (:LATIN-1 :EOL-STYLE :LF))) (alternatively, (map 'vector #'char-code "foo") and with (list (make-external-format...)) calls) And for each of the external formats, I get: There is no applicable method for the generic function #<STANDARD-GENERIC-FUNCTION (SETF FLEXI-STREAMS:FLEXI-STREAM-EXTERNAL-FORMAT) (2)> when called with arguments ((:UTF-8 :EOL-STYLE :LF) #<FLEXI-STREAMS::VECTOR-INPUT-STREAM {ABEFA91}>). Hrmpf! Am I abusing flexi-streams too much or is that a bug? How should one read externally-formatted data from an in-memory stream, anyway? And are string-backed in-memory streams even allowed? How to specify the internal external format for them? (-: Thanks for your time and for developing flexi-streams. In return, I hope to be able to buy you a beverage of your choice in Hamburg (-: Cheers, -- Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs
Hi! On Wed, 08 Mar 2006 00:43:12 +0100, Andreas Fuchs <asf@boinkor.net> wrote:
I would like to teach cl-irc about the goodness of external-formats, and because irc doesn't let users specify which external format to use, I need something as flexible as flexi-streams (-:
So, once I've read an irc command (as a "line" of latin-1 characters), I'd like to convert it to text that looks the least insane, which is why my code tries several external-formats in a row and returns as soon as it found one that doesn't throw an error:
(defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (flexi-streams:with-input-from-sequence (in line) (setf (flexi-streams:flexi-stream-external-format in) external-format) (read-line in)))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded)))
If LINE is a string you want this: (defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (with-input-from-string (in line) (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) (read-line flexi))))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded))) But actually I think you want LINE to be a sequence of octets, so this is what you want: (defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (flexi-streams:with-input-from-sequence (in line) (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) (read-line flexi))))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded))) Result: CL-USER 11 > (try-decode-line '(228 246 252) '((:utf-8 :eol-style :lf) (:latin-1 :eol-style :lf))) tried (:UTF-8 :EOL-STYLE :LF): NIL error: Unexpected value #xF6 in UTF-8 sequence. tried (:LATIN-1 :EOL-STYLE :LF): "äöü" error: T "äöü" But you'll need version 0.5.3 to see that because there was a typo in the code which generated the error messages.
Hrmpf! Am I abusing flexi-streams too much or is that a bug? How should one read externally-formatted data from an in-memory stream, anyway?
You forgot that you have to create a flexi stream first - in-memory streams happen to be provided by the same library but they're something different. Use MAKE-FLEXI-STREAM to turn them into flexi streams.
And are string-backed in-memory streams even allowed?
No. CL already has WITH-INPUT-FROM-STRING... :)
Thanks for your time and for developing flexi-streams. In return, I hope to be able to buy you a beverage of your choice in Hamburg (-:
Nice. Looking forward to seeing you there! Cheers, Edi.
Today, Edi Weitz <edi@agharta.de> wrote:
Hi!
Hi there!
On Wed, 08 Mar 2006 00:43:12 +0100, Andreas Fuchs <asf@boinkor.net> wrote:
So, once I've read an irc command (as a "line" of latin-1 characters), I'd like to convert it to text that looks the least insane, which is why my code tries several external-formats in a row and returns as soon as it found one that doesn't throw an error:
If LINE is a string you want this:
(defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (with-input-from-string (in line) (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) (read-line flexi))))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded)))
This didn't exactly work; I get errors stating that the in-memory stream isn't a binary stream.
But actually I think you want LINE to be a sequence of octets, so this is what you want:
And this does work perfectly. Thanks so much! (: My current scheme is (horrible and) as follows: I read a line in the latin-1 external format, then create a vector from the char-codes, and decode that with try-decode-line. Why latin-1, you ask? Because it has a 1:1 code point mapping to bytes, as far as irc is concerned, and I don't have to implement my own buffering, which would really suck - IRC operates on lines, whereas all binary streams need fixed-width buffers. 54It seems like the easiest way to interoperate with broken protocols is to break a little inside, just like their authors once did. (:
But you'll need version 0.5.3 to see that because there was a typo in the code which generated the error messages.
Heh. I just upgraded.
Hrmpf! Am I abusing flexi-streams too much or is that a bug? How should one read externally-formatted data from an in-memory stream, anyway?
You forgot that you have to create a flexi stream first - in-memory streams happen to be provided by the same library but they're something different. Use MAKE-FLEXI-STREAM to turn them into flexi streams.
Oh. I misread the phrase 'These streams can obviously be used as the underlying streams for flexi streams.' as 'They are the base class of flexi-streams, and they work just like flexi-streams do.' -- I blame late-night hacking. (:
And are string-backed in-memory streams even allowed?
No. CL already has WITH-INPUT-FROM-STRING... :)
Thanks for your time and for developing flexi-streams. In return, I hope to be able to buy you a beverage of your choice in Hamburg (-:
Nice. Looking forward to seeing you there!
Me too (: Thanks again, -- Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs
On Wed, 08 Mar 2006 15:01:39 +0100, Andreas Fuchs <asf@boinkor.net> wrote:
This didn't exactly work; I get errors stating that the in-memory stream isn't a binary stream.
OK. I only tested on LispWorks. But as we know now this wasn't what you wanted anyway... :)
My current scheme is (horrible and) as follows: I read a line in the latin-1 external format, then create a vector from the char-codes, and decode that with try-decode-line.
Why latin-1, you ask? Because it has a 1:1 code point mapping to bytes, as far as irc is concerned, and I don't have to implement my own buffering, which would really suck - IRC operates on lines, whereas all binary streams need fixed-width buffers.
Seems reasonable, although not cute... :)
participants (2)
-
Andreas Fuchs -
Edi Weitz