[tbnl-devel] Strange encoding problems with SBCL and TBNL
Hi list, I'm running SBCL 0.9.13.52 (same error with 0.9.13 release) and TBNL 0.9.10. I've got a simple page set up that I need to simply print the return value of an external function. That function returns a list containing UTF-8 characters. I've set *show-lisp-errors-p* to T, here is the error: encoding error on stream #<SB-SYS:FD-STREAM for "standard output" {A6877B9}> (:EXTERNAL-FORMAT :ASCII): the character with code 20170 cannot be encoded. I had done some digging on google and found someone in a similar situation with TBNL, and he tried changing his locale LC_ALL to a utf-8 one. The above (:EXTERNAL-FORMAT :ASCII part was :ASCII only after I set my locale to UTF-8, beforehand it was :LATIN-1, which I do not understand. Here is the code, if it helps: (in-package :tbnl-foo) (setq *dispatch-table* (list (create-prefix-dispatcher "/tbnl/foo" 'foo) 'default-dispatcher)) (defun foo () (let ((foo (jalat::search-hash "hello"))) (with-html-output (*standard-output*) (:h2 (str foo))))) (setq *dispatch-table* (nconc (mapcar (lambda (args) (apply #'create-prefix-dispatcher args)) '(("/tbnl/test/" foo))) (list #'default-dispatcher))) The external function simply returns a list of plists containing UTF-8 characters. I also tried setting *tbnl-default-external-format* to no avail. Thanks guys, -- Dan Beauchesne dbeauchesne -at- gmail -dot- com
On 26 Jun 2006 14:30:18 +0900, Dan Beauchesne <dbeauchesne@gmail.com> wrote:
I'm running SBCL 0.9.13.52 (same error with 0.9.13 release) and TBNL 0.9.10.
I've got a simple page set up that I need to simply print the return value of an external function. That function returns a list containing UTF-8 characters. I've set *show-lisp-errors-p* to T, here is the error:
encoding error on stream #<SB-SYS:FD-STREAM for "standard output" {A6877B9}> (:EXTERNAL-FORMAT :ASCII): the character with code 20170 cannot be encoded.
I had done some digging on google and found someone in a similar situation with TBNL, and he tried changing his locale LC_ALL to a utf-8 one. The above (:EXTERNAL-FORMAT :ASCII part was :ASCII only after I set my locale to UTF-8, beforehand it was :LATIN-1, which I do not understand.
Maybe you used a locale which wasn't recognized and therefore the system defaulted to ASCII?
Here is the code, if it helps:
(in-package :tbnl-foo)
(setq *dispatch-table* (list (create-prefix-dispatcher "/tbnl/foo" 'foo) 'default-dispatcher))
(defun foo () (let ((foo (jalat::search-hash "hello"))) (with-html-output (*standard-output*) (:h2 (str foo)))))
Note that even if everything else works the "Content-Length" header will probably be wrong.
(setq *dispatch-table* (nconc (mapcar (lambda (args) (apply #'create-prefix-dispatcher args)) '(("/tbnl/test/" foo))) (list #'default-dispatcher)))
The external function simply returns a list of plists containing UTF-8 characters. I also tried setting *tbnl-default-external-format* to no avail.
The main problem seems to be to find out how one has to set up the stream to be able to send arbitrary Unicode characters. This is not really related to TBNL. If you don't get help here, I'd propose to ask on the SBCL mailing list. Cheers, Edi.
Hi! Thanks for the reply,
I had done some digging on google and found someone in a similar situation with TBNL, and he tried changing his locale LC_ALL to a utf-8 one. The above (:EXTERNAL-FORMAT :ASCII part was :ASCII only after I set my locale to UTF-8, beforehand it was :LATIN-1, which I do not understand.
Maybe you used a locale which wasn't recognized and therefore the system defaulted to ASCII?
Yes, it did just turn out to be bad configuration on my end. Sorry about that. I've been having a lot of fun mucking around with TBNL, but I've run into a bit of a wall here: in regards to my tiny page in my previous post, I've got a list of plists which I would like to use to generate a page of clickable links (one link for each plist) which, when clicked, would evaluate a lisp expression (actually, I just want to push that plist entry onto another list). I thought about using create-prefix-dispatcher and somehow keeping a separate *dispatch-table* for each session, but I thought there may be an easier way without having to modify the base code. Could you give me any advice on some solution? Thank you, -- Dan Beauchesne dbeauchesne -at- gmail -dot- com
Dan Beauchesne wrote:
I've got a list of plists which I would like to use to generate a page of clickable links (one link for each plist) which, when clicked, would evaluate a lisp expression (actually, I just want to push that plist entry onto another list).
This isn't a Lisp or TBNL question, as much as a web programming one. If the main list of plists isn't mutating, you can tag each link with an index into the main list (/put?entry=23), so that the receiving function (the one handling "/put") knows what to do: (push (nth (get-parameter "entry") *main-list*) *other-list*) Otherwise, if the main list is mutating all the time, one way to do it is: 1. make a (shallow? deep?) copy of the main list; 2. use the copy to generate the links; 3. store the copy in a session variable; 4. use the session variable in the receiving handler. Remember to use locks (such as KMRCL's make-lock and with-lock-held) if the users can modify any global variable, otherwise chaos WILL ensue. (TBNL gurus correct me if I'm wrong) Toby -- Signed/encrypted mail welcome. GPG/PGP Key-Id: 0x15C5C2EA
On Sat, 1 Jul 2006 13:12:57 +0200, Toby <tobia.conforto@linux.it> wrote:
Otherwise, if the main list is mutating all the time, one way to do it is: 1. make a (shallow? deep?) copy of the main list; 2. use the copy to generate the links; 3. store the copy in a session variable; 4. use the session variable in the receiving handler.
Doing this with sessions is a good idea IMHO. I've done something similar myself several times. You can even create something like "private" (encoded) links which translate to arbitrary Lisp code and keep a table which translates from URLs to the corresponding actions in your session.
On 7/3/06, Edi Weitz <edi@agharta.de> wrote:
On Sat, 1 Jul 2006 13:12:57 +0200, Toby <tobia.conforto@linux.it> wrote:
Otherwise, if the main list is mutating all the time, one way to do it is: 1. make a (shallow? deep?) copy of the main list; 2. use the copy to generate the links; 3. store the copy in a session variable; 4. use the session variable in the receiving handler.
Doing this with sessions is a good idea IMHO. I've done something similar myself several times. You can even create something like "private" (encoded) links which translate to arbitrary Lisp code and keep a table which translates from URLs to the corresponding actions in your session.
Cool, thanks for the great ideas. Also, in regards to the UTF-8 + SBCL + content-length problem, I'm curious if Dr. Weitz's Lisp (Allegro?) does not have this problem? I've been looking for a good excuse to grab the student edition, maybe this is it :) Thanks again, -- Dan Beauchesne dbeauchesne -at- gmail -dot- com
On 05 Jul 2006 12:28:10 +0900, Dan Beauchesne <dbeauchesne@gmail.com> wrote:
Also, in regards to the UTF-8 + SBCL + content-length problem, I'm curious if Dr. Weitz's Lisp (Allegro?) does not have this problem? I've been looking for a good excuse to grab the student edition, maybe this is it :)
I'm not using AllegroCL, I use LispWorks. The problem is actually always the same - the HTTP standard demands that the content length header counts the number of octets, but in Common Lisp, LENGTH applied to a string counts the number of characters which is not the same for UTF-8. The test/ directory of TBNL contains two example pages that deliver UTF-8 content. One is LW-only but it should be easy to convert it to SBCL using SBCL's internal STRING-TO-OCTETS function (see util.lisp), the other one uses TBNL:SEND-HEADERS and thus avoids to send a content length header at all. If your front-end is able to do that (like Apache/mod_lisp or Hunchentoot), then this should result in chunked encoding. HTH, Edi.
On Wed, 05 Jul 2006 15:59:46 +0200, Edi Weitz <edi@agharta.de> wrote:
using SBCL's internal STRING-TO-OCTETS function
That should read "TBNL's internal ..." - I think the name of the function is the same in SBCL but it's external there.
participants (3)
-
Dan Beauchesne -
Edi Weitz -
Toby