Ivan Boldyrev lispnik@gmail.com writes:

On Mon, Feb 14, 2011 at 1:49 AM, Nicolas Martyanoff < khaelin@gmail.com> wrote:

I have a query which inserts text containing the '’' character
(unicode
U+2019). Postmodern seems to generate a correct SQL query, but I
get a
postgresql error:
 

Database error 08P01: invalid message format

Is your Lisp implementation Unicode-aware?  Does it work with other Unicode chars?

I use sbcl, which has full unicode support. I have a different error message with some characters, for example with 'é', I have:

Database error 08P01: insufficient data left in message

Regards,

Marijn Haverbeke marijnh@gmail.com writes:

After loading the system, check the value of cl-postgres-system::*unicode* . Is it T?

Yes it is.

Marijn Haverbeke marijnh@gmail.com writes:

Hm. Then, this would suggest a bug somewhere. Do simpler non-ascii things work? (For example, walk through the quickstart.) Can you give me your Postgres version, and the exact query that goes wrong?

Yep, no problem with ascii.

I attached the test file I used:

The query is:

(postmodern:execute (:update 'posts :set 'title title 'content content 'content-html content-html 'creation-date (date-format-iso-8601 creation-date) 'visible visible-p :where (:= 'id id)))

Where content-html contains the string read from the html file.

I then get the following error:

Database error 08P01: invalid message format Query: UPDATE posts SET title = E'Test', content = E'- Some ascii. - Some UTF-8 (exists in latin 1): é - Some UTF-8 (does not exist in latin 1): ’', content_html = E'<ul

<li Some ascii.</li <li Some UTF–8 (exists in latin 1): é</li <li Some UTF–8 (does not exist in latin 1): ’</li </ul ', creation_date = E'2011-02-03 08:13:43+01:00', visible = true WHERE (id = E'1')

[Condition of type CL-POSTGRES:DATABASE-ERROR]

Regards,

Marijn Haverbeke marijnh@gmail.com writes:

...

Yep, no problem with ascii.

Sure, but that's not what I asked. Does this query from the quickstart work? (query "select 22, 'Folie et déraison', 4.5")

This one works.

Also, again, which version of Postgres are you using?

Ooops :) I use the git version, 5fe54bd.

Regards,

Marijn Haverbeke marijnh@gmail.com writes:

...

...

Also, again, which version of Postgres are you using?

...

Ooops :) I use the git version, 5fe54bd.

I meant PosgreSQL, not Postmodern.

Postgresql 9.0.3

Marijn Haverbeke marijnh@gmail.com writes:

Hi Nicolas,

Still no luck reproducing this. Does this simple query fail too? (query "select '’'")

All right, I finally figured out the problem. The string came from a file, which was loaded with:

(defun file-read (filename) (with-open-file (stream filename) (let ((content (make-string (file-length stream)))) (read-sequence content stream) content)))

This didn't correctly handle multibyte characters, and yielded a string containg null bytes at the end. This triggered the error.

Using the following works perfectly:

(defun file-read (filename) (with-open-file (stream filename :element-type '(unsigned-byte 8)) (let ((content (make-array (file-length stream) :element-type '(unsigned-byte 8)))) (read-sequence content stream) (babel:octets-to-string content))))

Sorry for the noise, and thank you for trying to help me.

Regards,