Edi Weitz edi@agharta.de writes:

On Fri, 18 May 2007 10:01:06 -0400, Dimitre Liotev dl@znain.net wrote:

...

when I have UTF characters in a template I get this error:

#? is not a LATIN-1 character. [Condition of type FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR]

What external format did you specify? How about showing us the code you tried and providing a backtrace?

Cheers, Edi.

Hi Edi,

I am using the "Simple example" from your page http://weitz.de/html-template/, specifying :external-format "UTF-8" in the call to create-template-printer:

(defparameter *template-dispatcher* (hunchentoot:create-prefix-dispatcher "/template-test/" (lambda () (let* ((html-template:*string-modifier* #'identity) (rows (loop for i below 49 by 7 collect (list :cols (loop for j from i below (+ i 7) for string = (format nil "~R" j) collect (list :content string :colorful-style (oddp j)))))) (values (list :rows rows))) (with-output-to-string (html-template:*default-template-output*) (html-template:fill-and-print-template (html-template:create-template-printer #p"foo.tmpl" :external-format "UTF-8") values))))))

(push *template-dispatcher* hunchentoot:*dispatch-table*)

The template:

<table border=1> <!-- TMPL_LOOP rows --> <tr> <!-- TMPL_LOOP cols --> <!-- TMPL_IF colorful-style --> <td align="right" bgcolor="blue">some cyrullic chars: ДЖГБ<!-- TMPL_VAR content --></td> <!-- TMPL_ELSE --> <td align="right" ><!-- TMPL_VAR content --></td> <!-- /TMPL_IF --> <!-- /TMPL_LOOP --> </tr> <!-- /TMPL_LOOP --> </table>

Now if I point the browser to http://localhost:8082/template-test/ it works fine if the UTF chars are not there, but if I put any cyrillic UTF encoded character in the template I get the error:

#? is not a LATIN-1 character. [Condition of type FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR]

Restarts: 0: [ABORT] Abort entirely from this (lisp) process.

Backtrace: 0: ((:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0)) Locals: CONDITION = :UNKNOWN 1: ((:INTERNAL (:TOP-LEVEL-FORM "swank-backend.lisp" 20112) 0) #<Function SWANK-DEBUGGER-HOOK> #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>) Locals: SWANK-BACKEND::HOOK = #<Function SWANK-DEBUGGER-HOOK> SWANK-BACKEND::FUN = #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a> SWANK-BACKEND::FUN = #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a> 2: (SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK #<Function SWANK-DEBUGGER-HOOK> #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>) Locals: SWANK-BACKEND::HOOK = #<Function SWANK-DEBUGGER-HOOK> SWANK-BACKEND::FUN = #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a> SWANK-BACKEND::HOOK = #<Function SWANK-DEBUGGER-HOOK> SWANK-BACKEND::FUN = #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a> 3: ((:INTERNAL SWANK::CALL-WITH-CONNECTION 0)) Locals: SWANK::FUN = #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a> 4: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION @ #x207161da> #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @ #x2127a6aa>) Locals: SWANK::CONNECTION = #<SWANK::CONNECTION @ #x207161da> FUNCTION = #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @ #x2127a6aa> FUNCTION = #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @ #x2127a6aa> SWANK::IO = #<TWO-WAY-STREAM "" @ #x20726182> SWANK::IN = #<SWANK-BACKEND::SLIME-INPUT-STREAM @ #x20726192> SWANK::CONNECTION = #<SWANK::CONNECTION @ #x207161da> SWANK::OUT = #<SWANK-BACKEND::SLIME-OUTPUT-STREAM @ #x207161ca> 5: (SWANK::MAYBE-CALL-WITH-IO-REDIRECTION #<SWANK::CONNECTION @ #x207161da> #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @ #x2127a6aa>) Locals: SWANK::CONNECTION = #<SWANK::CONNECTION @ #x207161da> SWANK::FUN = #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @ #x2127a6aa> SWANK::CONNECTION = #<SWANK::CONNECTION @ #x207161da> SWANK::FUN = #(#<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>) 6: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION @ #x207161da> #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>) Locals: SWANK::CONNECTION = #<SWANK::CONNECTION @ #x207161da> SWANK::FUN = #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a> SWANK::CONNECTION = #<SWANK::CONNECTION @ #x207161da> SYSTEM::.LAMBDA-LEXICAL-ENV. = #(#<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>) SWANK::FUN = #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a> 7: (SWANK:SWANK-DEBUGGER-HOOK #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> #<Function SWANK-DEBUGGER-HOOK>) Locals: CONDITION = #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> SWANK::HOOK = #<Function SWANK-DEBUGGER-HOOK> SYSTEM::.LAMBDA-LEXICAL-ENV. = #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a> CONDITION = :UNKNOWN 8: (INVOKE-DEBUGGER #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602>) Locals: CONDITION = #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> EXCL::LOCAL-0 = #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> EXCL::LOCAL-1 = #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> EXCL::LOCAL-2 = T 9: (SIGNAL #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602>) Locals: EXCL::DATUM = #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> EXCL::ARGUMENTS = NIL EXCL::LOCAL-0 = NIL EXCL::LOCAL-1 = #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> EXCL::LOCAL-2 = ((ERROR . #<Function MAYBE-INVOKE-DEBUGGER>)) EXCL::LOCAL-3 = (ERROR . #<Function MAYBE-INVOKE-DEBUGGER>) EXCL::LOCAL-4 = #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> EXCL::LOCAL-5 = #\null EXCL::LOCAL-6 = #<Function MAYBE-INVOKE-DEBUGGER> 10: (ERROR FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR :FORMAT-CONTROL "~S is not a LATIN-1 character." :FORMAT-ARGUMENTS (#?) :STREAM #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>) Locals: EXCL::DATUM = FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR EXCL::ARGUMENTS = (:FORMAT-CONTROL "~S is not a LATIN-1 character." :FORMAT-ARGUMENTS (#?) :STREAM #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>) EXCL::LOCAL-0 = (:FORMAT-CONTROL "~S is not a LATIN-1 character." :FORMAT-ARGUMENTS (#?) :STREAM #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>) EXCL::LOCAL-1 = FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR EXCL::LOCAL-2 = #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602> EXCL::LOCAL-3 = 97 EXCL::LOCAL-4 = 1 EXCL::LOCAL-5 = #<non-lisp object @ #xfffffffd> EXCL::LOCAL-6 = #<non-lisp object @ #x63897f0f> :UNKNOWN = NIL :UNKNOWN = NIL :UNKNOWN = NIL :UNKNOWN = NIL :UNKNOWN = NIL 11: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> "~S is not a LATIN-1 character." #?) Locals: FLEXI-STREAMS:FLEXI-STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> FLEXI-STREAMS::FORMAT-CONTROL = "~S is not a LATIN-1 character." FLEXI-STREAMS::FORMAT-ARGS = (#?) FLEXI-STREAMS::FORMAT-ARGS = (#?) FLEXI-STREAMS:FLEXI-STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> FLEXI-STREAMS::FORMAT-CONTROL = "~S is not a LATIN-1 character." 12: ((METHOD STREAM-WRITE-CHAR (FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM T)) #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> #?) Locals: STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> CHAR = #? CHAR = #? FLEXI-STREAMS:OCTET = 1044 STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> 13: ((:INTERNAL (:EFFECTIVE-METHOD 2 NIL NIL NIL NIL) 0) #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> #?) Locals: EXCL::METHARG0 = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> EXCL::METHARG1 = #? EXCL::LOCAL-0 = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> EXCL::LOCAL-1 = #? EXCL::LOCAL-2 = #<Closure (:INTERNAL (:EFFECTIVE-METHOD 2 NIL NIL NIL NIL) 0) @ #x20c1f10a> EXCL::LOCAL-3 = #<non-lisp object @ #x2> EXCL::LOCAL-4 = NIL EXCL::LOCAL-5 = 3 EXCL::LOCAL-6 = 137387322 EXCL::LOCAL-7 = #<Function (METHOD STREAM-WRITE-CHAR (FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM T))> EXCL::LOCAL-8 = 2 14: ((METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (FLEXI-STREAMS:FLEXI-OUTPUT-STREAM T T T)) #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> "<table border=1>

<tr>

<td align="right" >zero</td> ..) Locals: FLEXI-STREAMS:FLEXI-OUTPUT-STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> SEQUENCE = "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> .. FLEXI-STREAMS::START = 173 FLEXI-STREAMS::END = 4537 SEQUENCE = "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> .. FLEXI-STREAMS:FLEXI-OUTPUT-STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> #:|g1798| = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> 15: ((METHOD STREAM-WRITE-STRING (TRIVIAL-GRAY-STREAMS:TRIVIAL-GRAY-STREAM-MIXIN T)) #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> "<table border=1>

<tr>

<td align="right" >zero</td> ..) Locals: STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> TRIVIAL-GRAY-STREAMS::SEQ = "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> .. TRIVIAL-GRAY-STREAMS::START = 0 TRIVIAL-GRAY-STREAMS::END = NIL TRIVIAL-GRAY-STREAMS::START = 0 TRIVIAL-GRAY-STREAMS::END = NIL STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> 16: (FLEXI-STREAMS:STRING-TO-OCTETS "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> ..) Locals: STRING = "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> .. FLEXI-STREAMS::EXTERNAL-FORMAT = #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) @ #x20760e02> FLEXI-STREAMS::START = :UNSUPPLIED FLEXI-STREAMS::END = :UNSUPPLIED FLEXI-STREAMS::OUT = #<FLEXI-STREAMS::VECTOR-OUTPUT-STREAM @ #x21279d72> NIL = 0 STRING = "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> .. FLEXI-STREAMS::START = 0 FLEXI-STREAMS::END = NIL FLEXI-STREAMS::EXTERNAL-FORMAT = #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) @ #x20760e02> 17: (HUNCHENTOOT::COMPUTE-LENGTH "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> ..) Locals: NIL = "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> .. EXCL::LOCAL-0 = "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> .. EXCL::LOCAL-1 = #<HUNCHENTOOT::REPLY @ #x2126e772> EXCL::LOCAL-2 = #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) @ #x20760e02> 18: (HUNCHENTOOT::START-OUTPUT "<table border=1>

<tr>

<td align="right" >zero</td>

<td align="right" bgcolor="blue">some cyrillic chars: ???? one</td> ..) 19: (HUNCHENTOOT::PROCESS-REQUEST ((:HOST . "localhost:8082") (:USER-AGENT . "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)") (:ACCEPT . "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5") (:ACCEPT-LANGUAGE . "en-us,en;q=0.5") (:ACCEPT-ENCODING . "gzip,deflate") (:ACCEPT-CHARSET . "ISO-8859-1,utf-8;q=0.7,*;q=0.7") (:KEEP-ALIVE . "300") (:CONNECTION . "keep-alive") ..)) 20: (HUNCHENTOOT::PROCESS-CONNECTION #<HUNCHENTOOT::SERVER @ #x20c14f9a> #<MULTIVALENT #1=stream socket #1#connected from #2=localhost/8082 to #2#/4142 @ #x211e0dba>)

Edi Weitz edi@agharta.de writes:

On Fri, 18 May 2007 11:24:15 -0400, Dimitre Liotev dl@znain.net wrote:

...

I am using the "Simple example" from your page http://weitz.de/html-template/, specifying :external-format "UTF-8" in the call to create-template-printer:

Is the string "UTF-8" a valid external format specifier on AllegroCL?

I thought so, but it seems that the format must be specified as a keyword:

CL-USER> (describe (find-if (lambda (x) (member :UTF-8 (excl:ef-nicknames x))) (EXCL:ALL-EXTERNAL-FORMATS)))

#<EXTERNAL-FORMAT :UTF8 ['(:E-CRLF :UTF8-BASE)] @ #x20116f6a> is a structure of type EXTERNAL-FORMAT. It has these slots: NAME (:E-CRLF :UTF8-BASE) NICKNAMES (:UTF8 :UTF-8) ...

I thought the string "UTF-8" would also work because when I try WITH-OPEN-FILE with an invalid external format like "UTF-9", an error is thrown, while "UTF-8" is accepted and no error is raised.

...

(defparameter *template-dispatcher* (hunchentoot:create-prefix-dispatcher "/template-test/" (lambda () (let* ((html-template:*string-modifier* #'identity) (rows (loop for i below 49 by 7 collect (list :cols (loop for j from i below (+ i 7) for string = (format nil "~R" j) collect (list :content string :colorful-style (oddp j)))))) (values (list :rows rows))) (with-output-to-string (html-template:*default-template-output*) (html-template:fill-and-print-template (html-template:create-template-printer #p"foo.tmpl" :external-format "UTF-8") values))))))

[...]

12: ((METHOD STREAM-WRITE-CHAR (FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM T)) #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> #?) Locals: STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> CHAR = #? CHAR = #? FLEXI-STREAMS:OCTET = 1044 STREAM = #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>

The condition is signalled when Hunchentoot is trying to write the page to the client. You have to make sure that Hunchentoot knows that it's supposed to output UTF-8. See the examples that come with Hunchentoot for how this can be done. (And you should also set the headers accordingly, so the browser knows what to expect.)

Thanks!

For the sake of those who might encounter the same problem: I added this to my handler function:

(setf (hunchentoot:reply-external-format) :UTF-8)

This will set the reply-external-format only for the given handler. To set it globaly use this:

(setf hunchentoot:*hunchentoot-default-external-format* :UTF-8)