#7: Bug in read-vector --------------------------+------------------------------------------------- Reporter: jcunningham | Owner: somebody Type: defect | Status: new Priority: major | Milestone: Component: Core | Version: 19c Resolution: | Keywords: read-vector --------------------------+------------------------------------------------- Old description:

I have been trying to read large binary files of floating point data using CMUCL (19c). I thought I would have to do it using some form of FFI and went to comp.lang.lisp for help getting that working. I succeeded. But Duane Rettig at Allegro suggested it would be easier to use 'read-vector. So I tried that as follows:

(let ((vec (make-array 10 :element-type 'double-float))) (with-open-file (os "d10.bin") (read-vector vec os) (print vec)))

where "d10.bin" is a double-float binary file containing 10 elements. When I try to read the file it produces the following error:

Type-error in KERNEL::OBJECT-NOT-DOUBLE-FLOAT-ERROR-HANDLER: #\Null is not of type DOUBLE-FLOAT

Here is the C-code code I used to produce the file:

#include <iostream> #include <fstream> #include <complex> using namespace std; int main() { int n=10; double *d = new double[n]; for (int i=0; i<n; i++) d[i] = i; FILE *of = fopen("d10.bin", "wb"); fwrite(f,8,n,of); fclose(of); return 0; }

Here is Duane's sample code that he says works in Allegro along with comments: (copied from http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/67876101085aee82/05a20cfcd11f8fbe?hl=en#05a20cfcd11f8fbe) ................................................................... "You must be using a simple-streams implementation from another lisp. Allegro CL doesn't have a KERNEL package.

What you're seeing above is a bug; you should report it to that lisp's support team.

It works fine in Allegro CL (for which you can download the Express Edition for free):

[edited formating for readability on Trac - jkc] (defvar *x* (make-array 10 :element-type 'double-float :initial-contents (loop for i from 0.0d0 to 9.0d0 collect i)))

#(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)

(with-open-file (s "z.dat" :direction :io :if-exists :overwrite :if-does-not-exist :create) (write-vector *x* s)) 80 (defvar *y* (make-array 10 :element-type 'double-float :initial-element 10.0d0))

(with-open-file (s "z.dat") (read-vector *y* s))

*y* #(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)

-- Duane Rettig"

Further correspondence with Raymond Toy corroborates this is a bug:

"Yes, this does appear to be a bug in the implementation of read-vector.

You can, however, achieve what you want by opening the file with an element-type of, say, (unsigned-byte 8), instead of the default 'character.

I'll have to read some more to understand how read-vector interacts with the stream element type. It seems, though, that the element-type of the vector overrides the element-type of the stream, more or less. Currently, a stream element type of character basically causes the code to read in characters.

Ray"

If you need more information, let me know and I'll see what I can do.

-jeff

New description:

I have been trying to read large binary files of floating point data using CMUCL (19c). I thought I would have to do it using some form of FFI and went to comp.lang.lisp for help getting that working. I succeeded. But Duane Rettig at Allegro suggested it would be easier to use 'read-vector. So I tried that as follows: {{{ (let ((vec (make-array 10 :element-type 'double-float))) (with-open-file (os "d10.bin") (read-vector vec os) (print vec))) }}} where "d10.bin" is a double-float binary file containing 10 elements. When I try to read the file it produces the following error:

Type-error in KERNEL::OBJECT-NOT-DOUBLE-FLOAT-ERROR-HANDLER: #\Null is not of type DOUBLE-FLOAT

Here is the C-code code I used to produce the file: {{{ #include <iostream> #include <fstream> #include <complex> using namespace std; int main() { int n=10; double *d = new double[n]; for (int i=0; i<n; i++) d[i] = i; FILE *of = fopen("d10.bin", "wb"); fwrite(f,8,n,of); fclose(of); return 0; } }}}

Here is Duane's sample code that he says works in Allegro along with comments: (copied from http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/67876101085aee82/05a20cfcd11f8fbe?hl=en#05a20cfcd11f8fbe) ................................................................... "You must be using a simple-streams implementation from another lisp. Allegro CL doesn't have a KERNEL package.

What you're seeing above is a bug; you should report it to that lisp's support team.

It works fine in Allegro CL (for which you can download the Express Edition for free):

[edited formating for readability on Trac - jkc] {{{ (defvar *x* (make-array 10 :element-type 'double-float :initial-contents (loop for i from 0.0d0 to 9.0d0 collect i)))

#(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)

(with-open-file (s "z.dat" :direction :io :if-exists :overwrite :if-does-not-exist :create) (write-vector *x* s)) 80 (defvar *y* (make-array 10 :element-type 'double-float :initial-element 10.0d0))

(with-open-file (s "z.dat") (read-vector *y* s))

*y* #(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0) }}} -- Duane Rettig"

Further correspondence with Raymond Toy corroborates this is a bug:

"Yes, this does appear to be a bug in the implementation of read-vector.

You can, however, achieve what you want by opening the file with an element-type of, say, (unsigned-byte 8), instead of the default 'character.

I'll have to read some more to understand how read-vector interacts with the stream element type. It seems, though, that the element-type of the vector overrides the element-type of the stream, more or less. Currently, a stream element type of character basically causes the code to read in characters.

Ray"

If you need more information, let me know and I'll see what I can do.

-jeff

Comment (by rtoy):

First I added a bit of markup so that the code bits are formatted better.

Now for the bug report, there are a couple of issues here. CMUCL has an extension, READ-VECTOR, which is similar to the SIMPLE-STREAM's READ- VECTOR, but not quite. In particular, it produces the bug that you see. However, if you {{{(require 'simple-streams)}}}, READ-VECTOR will be replaced with the simple-stream version. Then, the following snip of code works as Duane mentions:

{{{ (let ((vec (make-array 10 :element-type 'double-float))) (with-open-file (os "d10.bin" :class 'stream:file-simple- stream) (stream:read-vector vec os) (print vec))) }}}

The important part is that you open the file with the appropriate class.

Not sure what to do. We can change READ-VECTOR to match simple-stream READ-VECTOR. (BTW, READ-VECTOR is much faster than STREAM:READ-VECTOR.)