[hunchentoot-devel] utf-8 encoding problems

I have problems with some of the utf-8 characters in sbcl 1.0.18 and hunchentoot-0.15.7 (Ubuntu) When I create a handler like this: (defun test-handler () (no-cache) (recompute-request-parameters :external-format (flex:make-external-format :utf-8 :eol-style :lf)) (setf (hunchentoot:content-type) "text/html; charset=utf-8") (with-html-output-to-string (*standard-output* nil :prologue nil :indent nil) (:html (:head (:title "Test page")) (:body (:p ""quotation""))))) Where I have those "right and left quotation marks" I get an error: #\LEFT_DOUBLE_QUOTATION_MARK (code 8220) is not a LATIN-1 character. [Condition of type FLEXI-STREAMS:EXTERNAL-FORMAT-ENCODING-ERROR] If, on the other hand, I set global variable *hunchentoot-default-external-format* to utf-8 (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) then everything works. I just want one handler to handle the uft-8 encoding, how can I fix it? Thank you, Andrew

The email system replaced the quotation marks to the regular ones. What I meant was the quotation marks described here<http://en.wikipedia.org/wiki/Smart_quotes#Quotation_marks_in_electronic_documents> . On Tue, Sep 2, 2008 at 2:03 PM, Andrei Stebakov <lispercat@gmail.com> wrote:
I have problems with some of the utf-8 characters in sbcl 1.0.18 and hunchentoot-0.15.7 (Ubuntu) When I create a handler like this:
(defun test-handler () (no-cache) (recompute-request-parameters :external-format (flex:make-external-format :utf-8 :eol-style :lf)) (setf (hunchentoot:content-type) "text/html; charset=utf-8") (with-html-output-to-string (*standard-output* nil :prologue nil :indent nil) (:html (:head (:title "Test page")) (:body (:p
""quotation"")))))
Where I have those "right and left quotation marks" I get an error: #\LEFT_DOUBLE_QUOTATION_MARK (code 8220) is not a LATIN-1 character. [Condition of type FLEXI-STREAMS:EXTERNAL-FORMAT-ENCODING-ERROR]
If, on the other hand, I set global variable *hunchentoot-default-external-format* to utf-8 (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) then everything works. I just want one handler to handle the uft-8 encoding, how can I fix it?
Thank you, Andrew

No luck emailing the quotation marks. Let's try a German character 'ß' (defun test-handler () (no-cache) (recompute-request-parameters :external-format (flex:make-external-format :utf-8 :eol-style :lf)) (setf (hunchentoot:content-type) "text/html; charset=utf-8") (with-html-output-to-string (*standard-output* nil :prologue nil :indent nil) (:html (:head (:title "Test page")) (:body (:p "weiß"))))) Will produce text like "wei�" If I set the global variable: (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) the problem gets fixed. Andrew On Tue, Sep 2, 2008 at 2:03 PM, Andrei Stebakov <lispercat@gmail.com> wrote:
I have problems with some of the utf-8 characters in sbcl 1.0.18 and hunchentoot-0.15.7 (Ubuntu) When I create a handler like this:
(defun test-handler () (no-cache) (recompute-request-parameters :external-format (flex:make-external-format :utf-8 :eol-style :lf)) (setf (hunchentoot:content-type) "text/html; charset=utf-8") (with-html-output-to-string (*standard-output* nil :prologue nil :indent nil) (:html (:head (:title "Test page")) (:body (:p
""quotation"")))))
Where I have those "right and left quotation marks" I get an error: #\LEFT_DOUBLE_QUOTATION_MARK (code 8220) is not a LATIN-1 character. [Condition of type FLEXI-STREAMS:EXTERNAL-FORMAT-ENCODING-ERROR]
If, on the other hand, I set global variable *hunchentoot-default-external-format* to utf-8 (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) then everything works. I just want one handler to handle the uft-8 encoding, how can I fix it?
Thank you, Andrew

2008/9/2 Andrei Stebakov <lispercat@gmail.com>:
No luck emailing the quotation marks. Let's try a German character 'ß'
(defun test-handler () (no-cache) (recompute-request-parameters :external-format (flex:make-external-format :utf-8 :eol-style :lf)) (setf (hunchentoot:content-type) "text/html; charset=utf-8") (with-html-output-to-string (*standard-output* nil :prologue nil :indent nil) (:html (:head (:title "Test page")) (:body (:p "weiß")))))
Will produce text like "wei�" If I set the global variable: (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) the problem gets fixed.
Andrew
As a matter of interest, why do you have a problem using UTF8 everywhere? Just about everything supports it these days, and the lower seven bits look like ASCII? Anyway, (setf (reply-external-format &optional reply) new-value) with the make-external-format thing as 'new-value' should do the trick. Rob

Thanks, Robert, it works! The reason I don't use it globally is because in the system I have right now I pass a lot of textual data in the GET requests and for some reason it didn't work off the bat when I set that global variable (I'll look into it anyway). Andrew On Tue, Sep 2, 2008 at 4:38 PM, Robert Synnott <rsynnott@gmail.com> wrote:
2008/9/2 Andrei Stebakov <lispercat@gmail.com>:
No luck emailing the quotation marks. Let's try a German character 'ß'
(defun test-handler () (no-cache) (recompute-request-parameters :external-format (flex:make-external-format :utf-8 :eol-style :lf)) (setf (hunchentoot:content-type) "text/html; charset=utf-8") (with-html-output-to-string (*standard-output* nil :prologue nil :indent nil) (:html (:head (:title "Test page")) (:body (:p "weiß")))))
Will produce text like "wei�" If I set the global variable: (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) the problem gets fixed.
Andrew
As a matter of interest, why do you have a problem using UTF8 everywhere? Just about everything supports it these days, and the lower seven bits look like ASCII?
Anyway, (setf (reply-external-format &optional reply) new-value) with the make-external-format thing as 'new-value' should do the trick. Rob
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel

As a matter of interest, why do you have a problem using UTF8
everywhere? Just about everything supports it these days, and the
lower seven bits look like ASCII? If I set global (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) Then all my handlers which generate images will produce empty outputs. Andrew On Tue, Sep 2, 2008 at 4:38 PM, Robert Synnott <rsynnott@gmail.com> wrote:
2008/9/2 Andrei Stebakov <lispercat@gmail.com>:
No luck emailing the quotation marks. Let's try a German character 'ß'
(defun test-handler () (no-cache) (recompute-request-parameters :external-format (flex:make-external-format :utf-8 :eol-style :lf)) (setf (hunchentoot:content-type) "text/html; charset=utf-8") (with-html-output-to-string (*standard-output* nil :prologue nil :indent nil) (:html (:head (:title "Test page")) (:body (:p "weiß")))))
Will produce text like "wei�" If I set the global variable: (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) the problem gets fixed.
Andrew
As a matter of interest, why do you have a problem using UTF8 everywhere? Just about everything supports it these days, and the lower seven bits look like ASCII?
Anyway, (setf (reply-external-format &optional reply) new-value) with the make-external-format thing as 'new-value' should do the trick. Rob
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel

On Tue, 2 Sep 2008 17:48:00 -0400, "Andrei Stebakov" <lispercat@gmail.com> wrote:
If I set global (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) Then all my handlers which generate images will produce empty outputs.
How do your image handlers look like? They should return binary data and shouldn't be affected by the external format.

My image handlers look approximately like: (defun image-handler () (no-cache) (setf (hunchentoot:content-type) "image/jpeg") (let ((stream (send-headers))) (with-image (new 100 100) (do-something-with-img new) (write-jpeg-to-stream stream :image new)))) Andrew On Wed, Sep 3, 2008 at 2:42 AM, Edi Weitz <edi@agharta.de> wrote:
On Tue, 2 Sep 2008 17:48:00 -0400, "Andrei Stebakov" <lispercat@gmail.com> wrote:
If I set global (setf *hunchentoot-default-external-format* (flex:make-external-format :utf-8 :eol-style :lf)) Then all my handlers which generate images will produce empty outputs.
How do your image handlers look like? They should return binary data and shouldn't be affected by the external format. _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel

On Wed, 3 Sep 2008 10:10:32 -0400, "Andrei Stebakov" <lispercat@gmail.com> wrote:
My image handlers look approximately like:
(defun image-handler () (no-cache) (setf (hunchentoot:content-type) "image/jpeg") (let ((stream (send-headers))) (with-image (new 100 100) (do-something-with-img new) (write-jpeg-to-stream stream :image new))))
You want to write binary data to this stream. See the sentence about "faithful output" here: http://weitz.de/cl-gd/#write-jpeg-to-stream Edi.

Does it mean that I need to get the underlying binary stream from the flexi stream returned by (send-headers) and use it with write-jpeg-to-stream? Or use some other function (not write-jpeg-to-stream) so it returns an array of octets instead of writing to the stream? Andrew On Wed, Sep 3, 2008 at 10:34 AM, Edi Weitz <edi@agharta.de> wrote:
On Wed, 3 Sep 2008 10:10:32 -0400, "Andrei Stebakov" <lispercat@gmail.com> wrote:
My image handlers look approximately like:
(defun image-handler () (no-cache) (setf (hunchentoot:content-type) "image/jpeg") (let ((stream (send-headers))) (with-image (new 100 100) (do-something-with-img new) (write-jpeg-to-stream stream :image new))))
You want to write binary data to this stream. See the sentence about "faithful output" here:
http://weitz.de/cl-gd/#write-jpeg-to-stream
Edi. _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel

On Wed, 3 Sep 2008 12:05:03 -0400, "Andrei Stebakov" <lispercat@gmail.com> wrote:
Does it mean that I need to get the underlying binary stream from the flexi stream returned by (send-headers) and use it with write-jpeg-to-stream? Or use some other function (not write-jpeg-to-stream) so it returns an array of octets instead of writing to the stream?
All of these should work (I think): - get the underlying stream - set the element type of the (flexi) stream to 'flex:octet - set the external format of the (flexi) stream to latin-1, or - return an array of octets. If you're not on CLISP, you should also be able to write binary data directly to the stream, but CL-GD won't do that (as it predates flexi-streams).
participants (3)
-
Andrei Stebakov
-
Edi Weitz
-
Robert Synnott