Hi all,
I would like to teach cl-irc about the goodness of external-formats, and because irc doesn't let users specify which external format to use, I need something as flexible as flexi-streams (-:
So, once I've read an irc command (as a "line" of latin-1 characters), I'd like to convert it to text that looks the least insane, which is why my code tries several external-formats in a row and returns as soon as it found one that doesn't throw an error:
(defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (flexi-streams:with-input-from-sequence (in line) (setf (flexi-streams:flexi-stream-external-format in) external-format) (read-line in)))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded)))
Try it with:
(try-decode-line "foo" '((:UTF-8 :EOL-STYLE :LF) (:LATIN-1 :EOL-STYLE :LF)))
(alternatively, (map 'vector #'char-code "foo") and with (list (make-external-format...)) calls) And for each of the external formats, I get:
There is no applicable method for the generic function #<STANDARD-GENERIC-FUNCTION (SETF FLEXI-STREAMS:FLEXI-STREAM-EXTERNAL-FORMAT) (2)> when called with arguments ((:UTF-8 :EOL-STYLE :LF) #<FLEXI-STREAMS::VECTOR-INPUT-STREAM {ABEFA91}>).
Hrmpf! Am I abusing flexi-streams too much or is that a bug? How should one read externally-formatted data from an in-memory stream, anyway?
And are string-backed in-memory streams even allowed? How to specify the internal external format for them? (-:
Thanks for your time and for developing flexi-streams. In return, I hope to be able to buy you a beverage of your choice in Hamburg (-:
Cheers,
Hi!
On Wed, 08 Mar 2006 00:43:12 +0100, Andreas Fuchs asf@boinkor.net wrote:
I would like to teach cl-irc about the goodness of external-formats, and because irc doesn't let users specify which external format to use, I need something as flexible as flexi-streams (-:
So, once I've read an irc command (as a "line" of latin-1 characters), I'd like to convert it to text that looks the least insane, which is why my code tries several external-formats in a row and returns as soon as it found one that doesn't throw an error:
(defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (flexi-streams:with-input-from-sequence (in line) (setf (flexi-streams:flexi-stream-external-format in) external-format) (read-line in)))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded)))
If LINE is a string you want this:
(defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (with-input-from-string (in line) (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) (read-line flexi))))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded)))
But actually I think you want LINE to be a sequence of octets, so this is what you want:
(defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (flexi-streams:with-input-from-sequence (in line) (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) (read-line flexi))))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded)))
Result:
CL-USER 11 > (try-decode-line '(228 246 252) '((:utf-8 :eol-style :lf) (:latin-1 :eol-style :lf))) tried (:UTF-8 :EOL-STYLE :LF): NIL error: Unexpected value #xF6 in UTF-8 sequence. tried (:LATIN-1 :EOL-STYLE :LF): "äöü" error: T "äöü"
But you'll need version 0.5.3 to see that because there was a typo in the code which generated the error messages.
Hrmpf! Am I abusing flexi-streams too much or is that a bug? How should one read externally-formatted data from an in-memory stream, anyway?
You forgot that you have to create a flexi stream first - in-memory streams happen to be provided by the same library but they're something different. Use MAKE-FLEXI-STREAM to turn them into flexi streams.
And are string-backed in-memory streams even allowed?
No. CL already has WITH-INPUT-FROM-STRING... :)
Thanks for your time and for developing flexi-streams. In return, I hope to be able to buy you a beverage of your choice in Hamburg (-:
Nice. Looking forward to seeing you there!
Cheers, Edi.
Today, Edi Weitz edi@agharta.de wrote:
Hi!
Hi there!
On Wed, 08 Mar 2006 00:43:12 +0100, Andreas Fuchs asf@boinkor.net wrote:
So, once I've read an irc command (as a "line" of latin-1 characters), I'd like to convert it to text that looks the least insane, which is why my code tries several external-formats in a row and returns as soon as it found one that doesn't throw an error:
If LINE is a string you want this:
(defun try-decode-line (line external-formats) (loop for external-format in external-formats for decoded = nil for error = nil do (multiple-value-setq (decoded error) (ignore-errors (with-input-from-string (in line) (let ((flexi (flexi-streams:make-flexi-stream in :external-format external-format))) (read-line flexi))))) do (format t "~&tried ~s: ~S~% error: ~A~%" external-format decoded error) if decoded do (return decoded)))
This didn't exactly work; I get errors stating that the in-memory stream isn't a binary stream.
But actually I think you want LINE to be a sequence of octets, so this is what you want:
And this does work perfectly. Thanks so much! (:
My current scheme is (horrible and) as follows: I read a line in the latin-1 external format, then create a vector from the char-codes, and decode that with try-decode-line.
Why latin-1, you ask? Because it has a 1:1 code point mapping to bytes, as far as irc is concerned, and I don't have to implement my own buffering, which would really suck - IRC operates on lines, whereas all binary streams need fixed-width buffers.
54It seems like the easiest way to interoperate with broken protocols is to break a little inside, just like their authors once did. (:
But you'll need version 0.5.3 to see that because there was a typo in the code which generated the error messages.
Heh. I just upgraded.
Hrmpf! Am I abusing flexi-streams too much or is that a bug? How should one read externally-formatted data from an in-memory stream, anyway?
You forgot that you have to create a flexi stream first - in-memory streams happen to be provided by the same library but they're something different. Use MAKE-FLEXI-STREAM to turn them into flexi streams.
Oh. I misread the phrase 'These streams can obviously be used as the underlying streams for flexi streams.' as 'They are the base class of flexi-streams, and they work just like flexi-streams do.' -- I blame late-night hacking. (:
And are string-backed in-memory streams even allowed?
No. CL already has WITH-INPUT-FROM-STRING... :)
Thanks for your time and for developing flexi-streams. In return, I hope to be able to buy you a beverage of your choice in Hamburg (-:
Nice. Looking forward to seeing you there!
Me too (:
Thanks again,
On Wed, 08 Mar 2006 15:01:39 +0100, Andreas Fuchs asf@boinkor.net wrote:
This didn't exactly work; I get errors stating that the in-memory stream isn't a binary stream.
OK. I only tested on LispWorks. But as we know now this wasn't what you wanted anyway... :)
My current scheme is (horrible and) as follows: I read a line in the latin-1 external format, then create a vector from the char-codes, and decode that with try-decode-line.
Why latin-1, you ask? Because it has a 1:1 code point mapping to bytes, as far as irc is concerned, and I don't have to implement my own buffering, which would really suck - IRC operates on lines, whereas all binary streams need fixed-width buffers.
Seems reasonable, although not cute... :)
flexi-streams-devel@common-lisp.net
-
Andreas Fuchs -
Edi Weitz