From dl@znain.net Fri May 18 11:24:22 2007
From: Dimitre Liotev <dl@znain.net>
To: html-template-devel@common-lisp.net
Subject: Re: [html-template-devel] problem with UTF characters in template
Date: Fri, 18 May 2007 11:24:15 -0400
Message-ID: <814pmagnog.fsf@oh177.znain.com>
In-Reply-To: <ud50y6x70.fsf@agharta.de>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============5287617933509540636=="

--===============5287617933509540636==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Edi Weitz <edi(a)agharta.de> writes:

> On Fri, 18 May 2007 10:01:06 -0400, Dimitre Liotev <dl(a)znain.net> wrote:
>
>> when I have UTF characters in a template I get this error:
>>
>> #\? is not a LATIN-1 character.
>>    [Condition of type FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR]
>
> What external format did you specify?  How about showing us the code
> you tried and providing a backtrace?
>
> Cheers,
> Edi.


Hi Edi,

I am using the "Simple example" from your page
http://weitz.de/html-template/, specifying :external-format "UTF-8" in
the call to create-template-printer:

(defparameter *template-dispatcher*
  (hunchentoot:create-prefix-dispatcher "/template-test/"
   (lambda ()
     (let* ((html-template:*string-modifier* #'identity)
           (rows (loop for i below 49 by 7
                   collect
                     (list :cols
                           (loop for j from i below (+ i 7)
                              for string =3D (format nil "~R" j)
                              collect (list :content string=20
                                            :colorful-style (oddp j))))))
          (values (list :rows rows)))
       (with-output-to-string (html-template:*default-template-output*)
         (html-template:fill-and-print-template
          (html-template:create-template-printer #p"foo.tmpl" :external-forma=
t "UTF-8")
          values))))))                          =20


(push *template-dispatcher* hunchentoot:*dispatch-table*)


The template:

<table border=3D1>
  <!-- TMPL_LOOP rows -->
    <tr>
      <!-- TMPL_LOOP cols -->
        <!-- TMPL_IF colorful-style -->
          <td align=3D"right" bgcolor=3D"blue">some cyrullic chars: =D0=94=D0=
=96=D0=93=D0=91<!-- TMPL_VAR content --></td>
        <!-- TMPL_ELSE -->
          <td align=3D"right" ><!-- TMPL_VAR content --></td>
        <!-- /TMPL_IF -->
      <!-- /TMPL_LOOP -->
    </tr>
  <!-- /TMPL_LOOP -->
</table>


Now if I point the browser to http://localhost:8082/template-test/ it
works fine if the UTF chars are not there, but if I put any cyrillic UTF
encoded character in the template I get the error:


#\? is not a LATIN-1 character.
   [Condition of type FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR]

Restarts:
 0: [ABORT] Abort entirely from this (lisp) process.

Backtrace:
  0: ((:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0))
      Locals:
        CONDITION =3D :UNKNOWN
  1: ((:INTERNAL (:TOP-LEVEL-FORM "swank-backend.lisp" 20112) 0)
      #<Function SWANK-DEBUGGER-HOOK>
      #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>)
      Locals:
        SWANK-BACKEND::HOOK =3D #<Function SWANK-DEBUGGER-HOOK>
        SWANK-BACKEND::FUN =3D #<Closure (:INTERNAL
  SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>
        SWANK-BACKEND::FUN =3D #<Closure (:INTERNAL
  SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>
  2: (SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK
       #<Function SWANK-DEBUGGER-HOOK>
       #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>)
      Locals:
        SWANK-BACKEND::HOOK =3D #<Function SWANK-DEBUGGER-HOOK>
        SWANK-BACKEND::FUN =3D #<Closure (:INTERNAL
  SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>
        SWANK-BACKEND::HOOK =3D #<Function SWANK-DEBUGGER-HOOK>
        SWANK-BACKEND::FUN =3D #<Closure (:INTERNAL
  SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>
  3: ((:INTERNAL SWANK::CALL-WITH-CONNECTION 0))
      Locals:
        SWANK::FUN =3D #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @
  #x2127a67a>
  4: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION @ #x207161da>
       #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @
  #x2127a6aa>)
      Locals:
        SWANK::CONNECTION =3D #<SWANK::CONNECTION @ #x207161da>
        FUNCTION =3D #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @
  #x2127a6aa>
        FUNCTION =3D #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @
  #x2127a6aa>
        SWANK::IO =3D #<TWO-WAY-STREAM "" @ #x20726182>
        SWANK::IN =3D #<SWANK-BACKEND::SLIME-INPUT-STREAM @ #x20726192>
        SWANK::CONNECTION =3D #<SWANK::CONNECTION @ #x207161da>
        SWANK::OUT =3D #<SWANK-BACKEND::SLIME-OUTPUT-STREAM @ #x207161ca>
  5: (SWANK::MAYBE-CALL-WITH-IO-REDIRECTION
       #<SWANK::CONNECTION @ #x207161da>
       #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0) @
  #x2127a6aa>)
      Locals:
        SWANK::CONNECTION =3D #<SWANK::CONNECTION @ #x207161da>
        SWANK::FUN =3D #<Closure (:INTERNAL SWANK::CALL-WITH-CONNECTION 0)
  @ #x2127a6aa>
        SWANK::CONNECTION =3D #<SWANK::CONNECTION @ #x207161da>
        SWANK::FUN =3D #(#<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0)
  @ #x2127a67a>)
  6: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION @ #x207161da>
       #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>)
      Locals:
        SWANK::CONNECTION =3D #<SWANK::CONNECTION @ #x207161da>
        SWANK::FUN =3D #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @
  #x2127a67a>
        SWANK::CONNECTION =3D #<SWANK::CONNECTION @ #x207161da>
        SYSTEM::.LAMBDA-LEXICAL-ENV. =3D #(#<Closure (:INTERNAL
  SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>)
        SWANK::FUN =3D #<Closure (:INTERNAL SWANK:SWANK-DEBUGGER-HOOK 0) @
  #x2127a67a>
  7: (SWANK:SWANK-DEBUGGER-HOOK
       #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602>
       #<Function SWANK-DEBUGGER-HOOK>)
      Locals:
        CONDITION =3D #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
  #x2127a602>
        SWANK::HOOK =3D #<Function SWANK-DEBUGGER-HOOK>
        SYSTEM::.LAMBDA-LEXICAL-ENV. =3D #<Closure (:INTERNAL
  SWANK:SWANK-DEBUGGER-HOOK 0) @ #x2127a67a>
        CONDITION =3D :UNKNOWN
  8: (INVOKE-DEBUGGER #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
                        #x2127a602>)
      Locals:
        CONDITION =3D #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
  #x2127a602>
        EXCL::LOCAL-0 =3D #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
  #x2127a602>
        EXCL::LOCAL-1 =3D #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
  #x2127a602>
        EXCL::LOCAL-2 =3D T
  9: (SIGNAL #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @ #x2127a602>)
      Locals:
        EXCL::DATUM =3D #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
  #x2127a602>
        EXCL::ARGUMENTS =3D NIL
        EXCL::LOCAL-0 =3D NIL
        EXCL::LOCAL-1 =3D #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
  #x2127a602>
        EXCL::LOCAL-2 =3D ((ERROR . #<Function MAYBE-INVOKE-DEBUGGER>))
        EXCL::LOCAL-3 =3D (ERROR . #<Function MAYBE-INVOKE-DEBUGGER>)
        EXCL::LOCAL-4 =3D #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
  #x2127a602>
        EXCL::LOCAL-5 =3D #\null
        EXCL::LOCAL-6 =3D #<Function MAYBE-INVOKE-DEBUGGER>
 10: (ERROR FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR :FORMAT-CONTROL
            "~S is not a LATIN-1 character." :FORMAT-ARGUMENTS (#\?)
            :STREAM
            #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>)
      Locals:
        EXCL::DATUM =3D FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR
        EXCL::ARGUMENTS =3D (:FORMAT-CONTROL "~S is not a LATIN-1
  character." :FORMAT-ARGUMENTS
                           (#\?) :STREAM
                           #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM
  @ #x21279e5a>)
        EXCL::LOCAL-0 =3D (:FORMAT-CONTROL "~S is not a LATIN-1
  character." :FORMAT-ARGUMENTS
                         (#\?) :STREAM
                         #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @
  #x21279e5a>)
        EXCL::LOCAL-1 =3D FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR
        EXCL::LOCAL-2 =3D #<FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR @
  #x2127a602>
        EXCL::LOCAL-3 =3D 97
        EXCL::LOCAL-4 =3D 1
        EXCL::LOCAL-5 =3D #<non-lisp object @ #xfffffffd>
        EXCL::LOCAL-6 =3D #<non-lisp object @ #x63897f0f>
        :UNKNOWN =3D NIL
        :UNKNOWN =3D NIL
        :UNKNOWN =3D NIL
        :UNKNOWN =3D NIL
        :UNKNOWN =3D NIL
 11: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR
       #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>
       "~S is not a LATIN-1 character." #\?)
      Locals:
        FLEXI-STREAMS:FLEXI-STREAM =3D
  #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>
        FLEXI-STREAMS::FORMAT-CONTROL =3D "~S is not a LATIN-1 character."
        FLEXI-STREAMS::FORMAT-ARGS =3D (#\?)
        FLEXI-STREAMS::FORMAT-ARGS =3D (#\?)
        FLEXI-STREAMS:FLEXI-STREAM =3D
  #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>
        FLEXI-STREAMS::FORMAT-CONTROL =3D "~S is not a LATIN-1 character."
 12: ((METHOD STREAM-WRITE-CHAR
       (FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM T))
      #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> #\?)
      Locals:
        STREAM =3D #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @
  #x21279e5a>
        CHAR =3D #\?
        CHAR =3D #\?
        FLEXI-STREAMS:OCTET =3D 1044
        STREAM =3D #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @
  #x21279e5a>
 13: ((:INTERNAL (:EFFECTIVE-METHOD 2 NIL NIL NIL NIL) 0)
      #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a> #\?)
      Locals:
        EXCL::METHARG0 =3D #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @
  #x21279e5a>
        EXCL::METHARG1 =3D #\?
        EXCL::LOCAL-0 =3D #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @
  #x21279e5a>
        EXCL::LOCAL-1 =3D #\?
        EXCL::LOCAL-2 =3D #<Closure (:INTERNAL (:EFFECTIVE-METHOD 2 NIL
  NIL NIL NIL) 0) @
                          #x20c1f10a>
        EXCL::LOCAL-3 =3D #<non-lisp object @ #x2>
        EXCL::LOCAL-4 =3D NIL
        EXCL::LOCAL-5 =3D 3
        EXCL::LOCAL-6 =3D 137387322
        EXCL::LOCAL-7 =3D #<Function (METHOD STREAM-WRITE-CHAR
                                    (FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STRE=
AM
  T))>
        EXCL::LOCAL-8 =3D 2
 14: ((METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE
       (FLEXI-STREAMS:FLEXI-OUTPUT-STREAM T T T))
      #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>
      "<table border=3D1>
      =20
         <tr>
          =20
            =20
               <td align=3D\"right\" >zero</td>
              ..)
      Locals:
        FLEXI-STREAMS:FLEXI-OUTPUT-STREAM =3D
              #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>
        SEQUENCE =3D "<table border=3D1>
                    =20
                       <tr>
                        =20
                          =20
                             <td align=3D\"right\" >zero</td>
                          =20
                        =20
                          =20
                             <td align=3D\"right\" bgcolor=3D\"blue\">some
                             cyrillic chars: ???? one</td> ..
        FLEXI-STREAMS::START =3D 173
        FLEXI-STREAMS::END =3D 4537
        SEQUENCE =3D "<table border=3D1>
                    =20
                       <tr>
                        =20
                          =20
                             <td align=3D\"right\" >zero</td>
                          =20
                        =20
                          =20
                             <td align=3D\"right\" bgcolor=3D\"blue\">some
                             cyrillic chars: ???? one</td> ..
        FLEXI-STREAMS:FLEXI-OUTPUT-STREAM =3D
                             #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM
                             @ #x21279e5a>
        #:|g1798| =3D #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @
                             #x21279e5a>
 15: ((METHOD STREAM-WRITE-STRING
       (TRIVIAL-GRAY-STREAMS:TRIVIAL-GRAY-STREAM-MIXIN T))
      #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @ #x21279e5a>
      "<table border=3D1>
      =20
         <tr>
          =20
            =20
               <td align=3D\"right\" >zero</td>
              ..)
      Locals:
        STREAM =3D #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @
              #x21279e5a>
        TRIVIAL-GRAY-STREAMS::SEQ =3D "<table border=3D1>
                                     =20
                                        <tr>
                                         =20
                                           =20
                                              <td align=3D\"right\"
                                              >zero</td>
                                           =20
                                         =20
                                           =20
                                              <td align=3D\"right\"
                                              bgcolor=3D\"blue\">some
                                              cyrillic chars: ????
                                              one</td> ..
        TRIVIAL-GRAY-STREAMS::START =3D 0
        TRIVIAL-GRAY-STREAMS::END =3D NIL
        TRIVIAL-GRAY-STREAMS::START =3D 0
        TRIVIAL-GRAY-STREAMS::END =3D NIL
        STREAM =3D #<FLEXI-STREAMS::FLEXI-LATIN-1-OUTPUT-STREAM @
                                              #x21279e5a>
 16: (FLEXI-STREAMS:STRING-TO-OCTETS "<table border=3D1>
      =20
         <tr>
          =20
            =20
               <td align=3D\"right\" >zero</td>
            =20
          =20
            =20
               <td align=3D\"right\" bgcolor=3D\"blue\">some cyrillic chars:
               ???? one</td> ..)
      Locals:
        STRING =3D "<table border=3D1>
                  =20
                     <tr>
                      =20
                        =20
                           <td align=3D\"right\" >zero</td>
                        =20
                      =20
                        =20
                           <td align=3D\"right\" bgcolor=3D\"blue\">some
                           cyrillic chars: ???? one</td> ..
        FLEXI-STREAMS::EXTERNAL-FORMAT =3D
                           #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1
                           :EOL-STYLE :LF) @
                                           #x20760e02>
        FLEXI-STREAMS::START =3D :UNSUPPLIED
        FLEXI-STREAMS::END =3D :UNSUPPLIED
        FLEXI-STREAMS::OUT =3D #<FLEXI-STREAMS::VECTOR-OUTPUT-STREAM @
                           #x21279d72>
        NIL =3D 0
        STRING =3D "<table border=3D1>
                  =20
                     <tr>
                      =20
                        =20
                           <td align=3D\"right\" >zero</td>
                        =20
                      =20
                        =20
                           <td align=3D\"right\" bgcolor=3D\"blue\">some
                           cyrillic chars: ???? one</td> ..
        FLEXI-STREAMS::START =3D 0
        FLEXI-STREAMS::END =3D NIL
        FLEXI-STREAMS::EXTERNAL-FORMAT =3D
                           #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1
                           :EOL-STYLE :LF) @
                                           #x20760e02>
 17: (HUNCHENTOOT::COMPUTE-LENGTH "<table border=3D1>
      =20
         <tr>
          =20
            =20
               <td align=3D\"right\" >zero</td>
            =20
          =20
            =20
               <td align=3D\"right\" bgcolor=3D\"blue\">some cyrillic chars:
               ???? one</td> ..)
      Locals:
        NIL =3D "<table border=3D1>
               =20
                  <tr>
                   =20
                     =20
                        <td align=3D\"right\" >zero</td>
                     =20
                   =20
                     =20
                        <td align=3D\"right\" bgcolor=3D\"blue\">some
                        cyrillic chars: ???? one</td> ..
        EXCL::LOCAL-0 =3D "<table border=3D1>
                         =20
                            <tr>
                             =20
                               =20
                                  <td align=3D\"right\" >zero</td>
                               =20
                             =20
                               =20
                                  <td align=3D\"right\"
                                  bgcolor=3D\"blue\">some cyrillic chars:
                                  ???? one</td> ..
        EXCL::LOCAL-1 =3D #<HUNCHENTOOT::REPLY @ #x2126e772>
        EXCL::LOCAL-2 =3D #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1
                                  :EOL-STYLE :LF) @
                          #x20760e02>
 18: (HUNCHENTOOT::START-OUTPUT "<table border=3D1>
      =20
         <tr>
          =20
            =20
               <td align=3D\"right\" >zero</td>
            =20
          =20
            =20
               <td align=3D\"right\" bgcolor=3D\"blue\">some cyrillic chars:
               ???? one</td> ..)
 19: (HUNCHENTOOT::PROCESS-REQUEST
       ((:HOST . "localhost:8082")
        (:USER-AGENT
         . "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)")
        (:ACCEPT
         . "text/xml,application/xml,application/xhtml+xml,text/html;q=3D0.9,=
text/plain;q=3D0.8,image/png,*/*;q=3D0.5")
        (:ACCEPT-LANGUAGE . "en-us,en;q=3D0.5")
        (:ACCEPT-ENCODING . "gzip,deflate")
        (:ACCEPT-CHARSET . "ISO-8859-1,utf-8;q=3D0.7,*;q=3D0.7")
        (:KEEP-ALIVE . "300") (:CONNECTION . "keep-alive") ..))
 20: (HUNCHENTOOT::PROCESS-CONNECTION #<HUNCHENTOOT::SERVER @
               #x20c14f9a>
       #<MULTIVALENT #1=3Dstream socket #1#connected from
               #2=3Dlocalhost/8082
         to #2#/4142 @ #x211e0dba>)

--===============5287617933509540636==--