[cl-rdbms-devel] hu.dwim.rdbms.oracle-utf-problems-8
12:20 0 RDBMS DRIBBLE Convert from 5, size is 90, content: 66 00 6F 00 6F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
this data is utf-16, babel can deal with that, too.
i'm looking into installing oracle now, so that i can test it myself...
Hm, the connection encoding is UTF-16. SBCL doesn't support UTF-16. Therefore I have to convert the UTF-16 string with babel to UTF-8 if babel does such a thing.
reading this makes me think that you don't have a clear model of lisp strings/unicode/encodings.
I probably don't, yes.
i don't remember reading this article, but a quick glance looks promising: http://www.joelonsoftware.com/articles/Unicode.html
Interesting read, I saw this once but didn't read it. But I did now.
in short: you need to convert between lisp strings and byte arrays, sending/receiving byte arrays to/from the database. the encoding needed for OCI, oracle's C interface, is utf-16.
Is getting strings in utf-16 at this point a feature or a bug? Asked another way: Am I supposed to convert utf-16 strings to lisp strings or should that already be in there? If it were not a bug I would get a byte array instead of a utf-16 string? Error messages don't show up correctly either e.g. for violations of unique constraints? On the other hand, this being a feature would makes sense when thinking about where the data is supposed to be seen, the major web browsers. Assuming the major web browsers support utf-16. But then, how do I go about manipulation utf-16 string data with sbcl. Do I have to take a harder look at babel or flexi-streams?
utf-8 is nowhere in the picture (if not the encoding emacs/slime uses to communicate with the cl process).
Isn't sbcl able to use utf-8 to represent its lisp strings? Babel isn't able to convert between utf-16 and utf-8, yet? Confusion reigns ... ORACLE> (let ((octet-array (make-array 6 :element-type '(unsigned-byte 8) :initial-contents (vector #X66 #X00 #X6F #X00 #X6F #X00)))) (babel:octets-to-string octet-array :encoding :utf-8)) "f@^o@^o^@" ORACLE> Regards, chris
-- attila
!DSPAM:4cd1af6448581812115720!
in short: you need to convert between lisp strings and byte arrays, sending/receiving byte arrays to/from the database. the encoding needed for OCI, oracle's C interface, is utf-16.
Is getting strings in utf-16 at this point a feature or a bug?
you don't get utf-16 strings. you get lisp strings (which consists of characters, and whose encoding is implementation detail), which have been constructed in a bogus way. so, yes, it's a bug (or misconfiguration). but setting up oracle is still a pita, my efforts have failed so far. will play with it a bit more tomorrow.
Asked another way: Am I supposed to convert utf-16 strings to lisp strings or should that already be in there?
the byte array -> lisp strings conversion should have been done properly. (it's been done in a bogus way).
On the other hand, this being a feature would makes sense when thinking about where the data is supposed to be seen, the major web browsers. Assuming the major web browsers support utf-16.
again, the API of hu.dwim.rfbms should return lisp strings, which have no encoding (besides the implementation detail you can't see without looking at the sources of your lisp vm).
But then, how do I go about manipulation utf-16 string data with sbcl. Do I have to take a harder look at babel or flexi-streams?
hu.dwim.rdbms should have decoded utf-16 into lisp strings properly. you, as a user of it, have nothing to do. as a developer, it's a bug and/or misconfiguration to be found.
utf-8 is nowhere in the picture (if not the encoding emacs/slime uses to communicate with the cl process).
Isn't sbcl able to use utf-8 to represent its lisp strings?
it is able to *export* its strings into byte arrays using various encodings. but it's a very different thing from using utf-8 to represent lisp strings.
Babel isn't able to convert between utf-16 and utf-8, yet? Confusion reigns ... ORACLE> (let ((octet-array (make-array 6 :element-type '(unsigned-byte 8) :initial-contents (vector #X66 #X00 #X6F #X00 #X6F #X00)))) (babel:octets-to-string octet-array :encoding :utf-8)) "f@^o@^o^@"
babel can do these, and only these conversions: byte vector -> lisp string lisp string -> byte vector in the above you feed in an utf-16 byte vector and tell babel to decode it as utf-8. -- attila
oh, and one more thing: there's a pending patch to babel which introduces utf-16 encoding with explicit byte order... but it's not in the repo yet. i've dropped a mail asking what's the situation with it. reading your other mail, i think it's a byte order issue. -- attila
participants (2)
-
Attila Lendvai -
Marshall McLuhan