babel-devel

babel-devel@common-lisp.net

51 discussions

Re: [babel-devel] [cffi-devel] utf-16
by Luís Oliveira 31 May '10

31 May '10

On Mon, May 31, 2010 at 1:32 PM, Helmut Eller <heller(a)common-lisp.net> wrote: > Is this assertion > > (assert > (equal > (cffi:with-foreign-string (data "foobar" :encoding :utf-16) > (cffi:foreign-string-to-lisp data :count 6 :encoding :utf-16)) > "foobar")) > > expected to fail? FOREIGN-STRING-ALLOC currently doesn't insert BOM markers and the :UTF-16 encoder uses native endianness. The :UTF-16 decoder defaults to big endian when a BOM marker is not present, as recommended by the Unicode specification. That's why the assertion fails. One solution is to use :UTF-16BE or :UTF-16LE explicitly, which unfortunately aren't implemented in Babel yet. -- Luís Oliveira http://r42.eu/~luis/

1 0

[babel-devel] japanese encoding support.
by SANO Masatoshi 29 Dec '09

29 Dec '09

Hi. The attached file is japanese encoding (euc-jp and cp932) support for babel. Could you merge it? -- 佐野匡俊(SANO Masatoshi) snmsts(a)gmail.com

2 1

[babel-devel] Version 0.3.1 is requried
by Andrey Moskvitin 28 Dec '09

28 Dec '09

Since the last release had a few changes, in particular, important for Russia support cp1251. But these changes are now available only in darcs-version. What prevents release version 0.3.1? This would simplify the distribution of some packages. Andrey

2 2

[babel-devel] STREAM-LINE-COLUMN for VECTOR-OUTPUT-STREAMs
by Ariel Badichi 05 Dec '09

05 Dec '09

Hello, The vector output stream implementation is missing an applicable method for STREAM-LINE-COLUMN, which is required for character output support. Patch follows. Ariel diff -rN -u old-babel/src/streams.lisp new-babel/src/streams.lisp --- old-babel/src/streams.lisp 2009-11-29 11:31:24.000000000 +0200 +++ new-babel/src/streams.lisp 2009-11-29 11:31:25.000000000 +0200 @@ -323,6 +323,11 @@ (extend-vector-output-stream-buffer sequence stream :start start :end end))) sequence) +(defmethod stream-line-column ((stream vector-output-stream)) + "Dummy line-column method that always returns NIL, needed for +character output streams." + nil) + (defmethod stream-file-position ((stream vector-stream)) "Simply returns the index into the underlying vector." (declare (optimize speed))

2 1

[babel-devel] Unibyte decoders/encoders return wrong values
by Ariel Badichi 29 Nov '09

29 Nov '09

Hello, Unibyte decoders/encoders return (- START END) rather than (- END START). Patch follows. Ariel diff -rN -u old-babel/src/encodings.lisp new-babel/src/encodings.lisp --- old-babel/src/encodings.lisp 2009-11-29 03:01:45.000000000 +0200 +++ new-babel/src/encodings.lisp 2009-11-29 03:01:45.000000000 +0200 @@ -375,7 +375,7 @@ (declare (type code-point ,',code)) (block ,',encoding ,@',body))) ,',dest ,',di) - finally (return (the fixnum (- ,',d-start ,',di)))))))) + finally (return (the fixnum (- ,',di ,',d-start)))))))) ;;; The decoder version of the above macro. (defmacro define-unibyte-decoder (encoding (octet) &body body) @@ -400,7 +400,7 @@ (declare (type ub8 ,',octet)) (block ,',encoding ,@',body))) ,',dest ,',di) - finally (return (the fixnum (- ,',d-start ,',di)))))))) + finally (return (the fixnum (- ,',di ,',d-start)))))))) ;;;; Error Conditions ;;;

1 0

[babel-devel] XCVB patch for Babel
by Faré 23 Oct '09

23 Oct '09

Here is the patch I use to build Babel with XCVB (assuming alexandria and trivial-features are similarly patched). Is it OK to include this patch upstream? [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] [...] there is what I call the "roundtrip fallacy": it is a mistake to use, as journalists and some economists do, statistics without logic but the reverse does not hold: It is not a mistake to use logic without statistics. -- N. C. Taleb, Fooled by Randomness, 2004.

2 2

[babel-devel] [ANNOUNCE] Fedora RPM for babel available
by Robert Uhl 23 Jun '09

23 Jun '09

I have a babel RPM available for Fedora at: http://yum.octopodial-chrome.com/packages/cl-babel It's probably easiest to install the octopodial-chrome package first, then use 'yum install cl-babel' to grab it and cl-alexandria. -- Robert Uhl <http://public.xdi.org/=ruhl> `We _must_ implement multi-processor object-oriented Java-based client-server technologies immediately!' `You know, FORTRAN and slide rules put men on the moon and got them back safely multiple times.' --Matt Roberds

1 0

[babel-devel] [PATCH] elide note when compiling BOM-VECTOR
by Tobias C. Rittweiler 02 Jun '09

02 Jun '09

Compiling BOM-VECTOR emits a "type assertion to complex to check note." The reason is that (THE FOO (QUUX)) is actually of type (VALUES FOO &REST T) which SBCL, at least for the moment, cannot cope well with. The below patch changes it to (THE FOO (VALUES (QUUX))). -T. diff -rN -u old-babel/src/strings.lisp new-babel/src/strings.lisp --- old-babel/src/strings.lisp 2009-06-01 14:58:56.000000000 +0200 +++ new-babel/src/strings.lisp 2009-06-01 14:58:56.000000000 +0200 @@ -237,15 +237,16 @@ (defun bom-vector (encoding use-bom) (check-type use-bom (member :default t nil)) (the simple-vector - (if (null use-bom) - #() - (let ((enc (typecase encoding - (external-format (external-format-encoding encoding)) - (t (get-character-encoding encoding))))) - (if (or (eq use-bom t) - (and (eq use-bom :default) (enc-use-bom enc))) - (enc-bom-encoding enc) - #()))))) + (values + (if (null use-bom) + #() + (let ((enc (typecase encoding + (external-format (external-format-encoding encoding)) + (t (get-character-encoding encoding))))) + (if (or (eq use-bom t) + (and (eq use-bom :default) (enc-use-bom enc))) + (enc-bom-encoding enc) + #())))))) (defun string-to-octets (string &key (encoding *default-character-encoding*) (start 0) end (use-bom :default)

2 1

[babel-devel] Changes
by Dan Weinreb 25 Apr '09

25 Apr '09

There are the changes I had to make in tests.lisp in order to get the tests to pass, in the latest ITA version of Clozure Common Lisp (formerly known as OpenMCL). CCL does not support having a character with code #\udcf0. The reader signals a condition if it sees this. Unfortunately, using #-ccl does not seem to solve the problem, presumably since the #- macro is working by calling "read" and it is not suppressing unhandled conditions, or something like that. It might be hard to fix that in a robust way. In order to make progress, I had to just comment these out. I do not suggest merging that into the official sources, but it would be very nice if we could find a way to write tests.lisp in such a way that these tests would apply when the characters are supported, and not when they are not. The (or (code-char ..) ...) change, on the other hand, I think should be made in the official sources. The Hyperspec says clearly that code-char is allowed to return nil. What do you think? -- Dan Index: trunk/qres/lisp/libs/babel/tests/tests.lisp =================================================================== --- trunk/qres/lisp/libs/babel/tests/tests.lisp (revision 249746) +++ trunk/qres/lisp/libs/babel/tests/tests.lisp (revision 262389) @@ -259,22 +259,25 @@ #(97 98 99)) -(defstest utf-8b.1 - (string-to-octets (coerce #(#\a #\b #\udcf0) 'unicode-string) - :encoding :utf-8b) - #(97 98 #xf0)) - -(defstest utf-8b.2 - (octets-to-string (ub8v 97 98 #xcd) :encoding :utf-8b) - #(#\a #\b #\udccd)) - -(defstest utf-8b.3 - (octets-to-string (ub8v 97 #xf0 #xf1 #xff #x01) :encoding :utf-8b) - #(#\a #\udcf0 #\udcf1 #\udcff #\udc01)) - -(deftest utf-8b.4 () - (let* ((octets (coerce (loop repeat 8192 collect (random (+ #x82))) - '(array (unsigned-byte 8) (*)))) - (string (octets-to-string octets :encoding :utf-8b))) - (is (equalp octets (string-to-octets string :encoding :utf-8b))))) +;; CCL does not suppport Unicode characters between d800 and e000. +;(defstest utf-8b.1 +; (string-to-octets (coerce #(#\a #\b #\udcf0) 'unicode-string) +; :encoding :utf-8b) +; #(97 98 #xf0)) + +;; CCL does not suppport Unicode characters between d800 and e000. +;(defstest utf-8b.2 +; (octets-to-string (ub8v 97 98 #xcd) :encoding :utf-8b) +; #(#\a #\b #\udccd)) + +;; CCL does not suppport Unicode characters between d800 and e000. +;(defstest utf-8b.3 +; (octets-to-string (ub8v 97 #xf0 #xf1 #xff #x01) :encoding :utf-8b) +; #(#\a #\udcf0 #\udcf1 #\udcff #\udc01)) + +;(deftest utf-8b.4 () +; (let* ((octets (coerce (loop repeat 8192 collect (random (+ #x82))) +; '(array (unsigned-byte 8) (*)))) +; (string (octets-to-string octets :encoding :utf-8b))) +; (is (equalp octets (string-to-octets string :encoding :utf-8b))))) ;;; The following tests have been adapted from SBCL's @@ -338,5 +341,6 @@ (let ((string (make-string unicode-char-code-limit))) (dotimes (i unicode-char-code-limit) - (setf (char string i) (code-char i))) + ;; CCL does not suppport Unicode characters between d800 and e000. + (setf (char string i) (or (code-char i) #\a))) (let ((string2 (octets-to-string (string-to-octets string :encoding enc

4 11

Re: [babel-devel] Changes
by Luís Oliveira 14 Apr '09

14 Apr '09

On Tue, Apr 14, 2009 at 10:17 AM, David Lichteblau <david(a)lichteblau.com> wrote: > In Allegro CL and LispWorks, the situation is very different. They use > UTF-16 to represent Lisp strings in memory, so surrogates aren't just > forbidden in Lisp strings, user code actually needs to work with > surrogates to be able to use all of Unicode. My understanding was that they use UCS-2, i.e., they are limited to the BMP. AFAICT, their external formats don't produce surrogates in Lisp strings. (ACL doesn't. Didn't test Lispworks but its documentation specifically mentions the BMP and UCS-2.) They don't seem to have functions to deal with surrogates either. > SBCL has 21 bit characters like CCL and currently has characters for the > surrogate code points. But I am not aware of any consensus that this is > the right thing to do. Personally I think it's a bug and SBCL should be > changed to do it like CCL. I don't feel that strongly about either option. (I feel it's more important that the various Lisps agree on one of them.) That said, so far I haven't heard any compelling arguments in favor of having CODE-CHAR return NIL for such code points. > As far as I understand, the only Lisp with 21 bit characters whose > author thinks that SBCL's behaviour is correct is ECL, but I failed to > understand the reasoning behind that when it was being discussed > on comp.lang.lisp. [Assuming you're referring to the "Unicode and Common Lisp" thread. <ab100561-286c-4570-aabc-72fd877f22ae(a)v18g2000pro.googlegroups.com> <http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/97ff103a…>] I couldn't find where Juanjo argued for or against CODE-CHAR returning NIL for surrogates. His (somewhat unrelated) main point, IIUC, is that you shouldn't try to support full Unicode when all you have is 16-bit characters and instead restrict Unicode handling to the BMP. (U+0000 through U+FFFF.) > (As a side note, I find it a huge hassle to write code portable between > the Lisp implementations with Unicode support. For CXML, I needed read > time conditionals checking for UTF-16 Lisps. And it still doesn't > actually work, because most of the other free libraries like Babel, > CL-Unicode, and in turn, CL-PPCRE, expect 21 bit characters and are > effectively broken on Allegro and LispWorks.) I would argue that you're trying too hard. Would you support UTF-8 on CMUCL too? (Yes, I am aware that Unicode support for CMUCL is eminent. Using actual UTF-16... With surrogates... *sigh*) Just ignore/replace code points equal to or greater than CHAR-CODE-LIMIT. Does that mean CXML won't pass the test suites for Allegro and Lispworks? So be it. If enough Allegro or Lispworks costumers have the need to deal with characters outside the BMP, they'll complain and it'll be fixed. Duane Rettig hints in that c.l.l thread that Allegro might support full 21-bit Unicode characters in the future. (But perhaps I'm being too optimistic. Perhaps you really really need to use CXML+Lispworks/Allegro along with characters outside the BMP? Then ignore what I said above, I guess.) My plan for Babel was not to assume 21-bit characters, but to punt on characters above the CHAR-CODE-LIMIT. (It doesn't do that properly at the moment. That's definitely a bug that will have to be fixed.) Would you argue that it'd be better for Babel to instead use UTF-16 on Lispworks/Allegro? (Not a rhetorical question; if that turns out to be a good idea I'd change Babel in that direction.) What about UTF-8 for Lisps with 8-bit characters? I suspect that restricting oneself to a subset of Unicode is more robust and more manageable for portable programs. > While I have no ideas regarding UTF-8b, I think it worth pointing out > that for the important use case of file names, there is a different way of > achieving a round-trip involving file names in "might be UTF-8" format. > > The idea is to interpret invalid UTF-8 bytes in Latin 1, but prefix them > with the code point 0. > > On encoding back to a file name, such null characters would be stripped > again. That sounds like worse a hack than UTF-8B because if you convert such a string into another encoding you'll get bogus characters with no indication of error instead of, say, replacement characters. (That seems to be a big advantage of representing invalid bytes as invalid characters. Doesn't that make sense?) -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

babel-devel