Ivan Boldyrev <lispnik@gmail.com> writes:

On Mon, Feb 14, 2011 at 1:49 AM, Nicolas Martyanoff < khaelin@gmail.com> wrote:

I have a query which inserts text containing the '’' character (unicode U+2019). Postmodern seems to generate a correct SQL query, but I get a postgresql error:  

Database error 08P01: invalid message format

Is your Lisp implementation Unicode-aware?  Does it work with other Unicode chars?

I use sbcl, which has full unicode support. I have a different error message with some characters, for example with 'é', I have: Database error 08P01: insufficient data left in message Regards, -- Nicolas Martyanoff http://codemore.org khaelin@gmail.com

Marijn Haverbeke <marijnh@gmail.com> writes:

After loading the system, check the value of cl-postgres-system::*unicode* . Is it T?

Yes it is. -- Nicolas Martyanoff http://codemore.org khaelin@gmail.com

Marijn Haverbeke <marijnh@gmail.com> writes:

Hm. Then, this would suggest a bug somewhere. Do simpler non-ascii things work? (For example, walk through the quickstart.) Can you give me your Postgres version, and the exact query that goes wrong?

Yep, no problem with ascii. I attached the test file I used: The query is: (postmodern:execute (:update 'posts :set 'title title 'content content 'content-html content-html 'creation-date (date-format-iso-8601 creation-date) 'visible visible-p :where (:= 'id id))) Where content-html contains the string read from the html file. I then get the following error: Database error 08P01: invalid message format Query: UPDATE posts SET title = E'Test', content = E'- Some ascii. - Some UTF-8 (exists in latin 1): é - Some UTF-8 (does not exist in latin 1): ’', content_html = E'<ul

<li Some ascii.</li <li Some UTF–8 (exists in latin 1): é</li <li Some UTF–8 (does not exist in latin 1): ’</li </ul ', creation_date = E'2011-02-03 08:13:43+01:00', visible = true WHERE (id = E'1') [Condition of type CL-POSTGRES:DATABASE-ERROR]

Regards, -- Nicolas Martyanoff http://codemore.org khaelin@gmail.com

Marijn Haverbeke <marijnh@gmail.com> writes:

...

Yep, no problem with ascii.

Sure, but that's not what I asked. Does this query from the quickstart work? (query "select 22, 'Folie et déraison', 4.5") This one works.

Also, again, which version of Postgres are you using? Ooops :) I use the git version, 5fe54bd.

Regards, -- Nicolas Martyanoff http://codemore.org khaelin@gmail.com

Marijn Haverbeke <marijnh@gmail.com> writes:

...

...

Also, again, which version of Postgres are you using?

...

Ooops :) I use the git version, 5fe54bd.

I meant PosgreSQL, not Postmodern.

Postgresql 9.0.3 -- Nicolas Martyanoff http://codemore.org khaelin@gmail.com

Marijn Haverbeke <marijnh@gmail.com> writes:

Hi Nicolas,

Still no luck reproducing this. Does this simple query fail too? (query "select '’'")

All right, I finally figured out the problem. The string came from a file, which was loaded with: (defun file-read (filename) (with-open-file (stream filename) (let ((content (make-string (file-length stream)))) (read-sequence content stream) content))) This didn't correctly handle multibyte characters, and yielded a string containg null bytes at the end. This triggered the error. Using the following works perfectly: (defun file-read (filename) (with-open-file (stream filename :element-type '(unsigned-byte 8)) (let ((content (make-array (file-length stream) :element-type '(unsigned-byte 8)))) (read-sequence content stream) (babel:octets-to-string content)))) Sorry for the noise, and thank you for trying to help me. Regards, -- Nicolas Martyanoff http://codemore.org khaelin@gmail.com