#7: Bug in read-vector --------------------------+------------------------------------------------- Reporter: jcunningham | Owner: somebody Type: defect | Status: new Priority: major | Milestone: Component: Core | Version: 19c Resolution: | Keywords: read-vector --------------------------+------------------------------------------------- Old description:
I have been trying to read large binary files of floating point data using CMUCL (19c). I thought I would have to do it using some form of FFI and went to comp.lang.lisp for help getting that working. I succeeded. But Duane Rettig at Allegro suggested it would be easier to use 'read-vector. So I tried that as follows:
(let ((vec (make-array 10 :element-type 'double-float))) (with-open-file (os "d10.bin") (read-vector vec os) (print vec)))
where "d10.bin" is a double-float binary file containing 10 elements. When I try to read the file it produces the following error:
Type-error in KERNEL::OBJECT-NOT-DOUBLE-FLOAT-ERROR-HANDLER: #\Null is not of type DOUBLE-FLOAT
Here is the C-code code I used to produce the file:
#include <iostream> #include <fstream> #include <complex> using namespace std; int main() { int n=10; double *d = new double[n]; for (int i=0; i<n; i++) d[i] = i; FILE *of = fopen("d10.bin", "wb"); fwrite(f,8,n,of); fclose(of); return 0; }
Here is Duane's sample code that he says works in Allegro along with comments: (copied from http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/67876101085aee82/05a20cfcd11f8fbe?hl=en#05a20cfcd11f8fbe) ................................................................... "You must be using a simple-streams implementation from another lisp. Allegro CL doesn't have a KERNEL package.
What you're seeing above is a bug; you should report it to that lisp's support team.
It works fine in Allegro CL (for which you can download the Express Edition for free):
[edited formating for readability on Trac - jkc] (defvar *x* (make-array 10 :element-type 'double-float :initial-contents (loop for i from 0.0d0 to 9.0d0 collect i)))
#(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)
(with-open-file (s "z.dat" :direction :io :if-exists :overwrite :if-does-not-exist :create) (write-vector *x* s)) 80 (defvar *y* (make-array 10 :element-type 'double-float :initial-element 10.0d0))
(with-open-file (s "z.dat") (read-vector *y* s))
*y* #(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)
-- Duane Rettig"
Further correspondence with Raymond Toy corroborates this is a bug:
"Yes, this does appear to be a bug in the implementation of read-vector.
You can, however, achieve what you want by opening the file with an element-type of, say, (unsigned-byte 8), instead of the default 'character.
I'll have to read some more to understand how read-vector interacts with the stream element type. It seems, though, that the element-type of the vector overrides the element-type of the stream, more or less. Currently, a stream element type of character basically causes the code to read in characters.
Ray"
If you need more information, let me know and I'll see what I can do.
-jeff
New description:
I have been trying to read large binary files of floating point data using CMUCL (19c). I thought I would have to do it using some form of FFI and went to comp.lang.lisp for help getting that working. I succeeded. But Duane Rettig at Allegro suggested it would be easier to use 'read-vector. So I tried that as follows: {{{ (let ((vec (make-array 10 :element-type 'double-float))) (with-open-file (os "d10.bin") (read-vector vec os) (print vec))) }}} where "d10.bin" is a double-float binary file containing 10 elements. When I try to read the file it produces the following error:
Type-error in KERNEL::OBJECT-NOT-DOUBLE-FLOAT-ERROR-HANDLER: #\Null is not of type DOUBLE-FLOAT
Here is the C-code code I used to produce the file: {{{ #include <iostream> #include <fstream> #include <complex> using namespace std; int main() { int n=10; double *d = new double[n]; for (int i=0; i<n; i++) d[i] = i; FILE *of = fopen("d10.bin", "wb"); fwrite(f,8,n,of); fclose(of); return 0; } }}}
Here is Duane's sample code that he says works in Allegro along with comments: (copied from http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/67876101085aee82/05a20cfcd11f8fbe?hl=en#05a20cfcd11f8fbe) ................................................................... "You must be using a simple-streams implementation from another lisp. Allegro CL doesn't have a KERNEL package.
What you're seeing above is a bug; you should report it to that lisp's support team.
It works fine in Allegro CL (for which you can download the Express Edition for free):
[edited formating for readability on Trac - jkc] {{{ (defvar *x* (make-array 10 :element-type 'double-float :initial-contents (loop for i from 0.0d0 to 9.0d0 collect i)))
#(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)
(with-open-file (s "z.dat" :direction :io :if-exists :overwrite :if-does-not-exist :create) (write-vector *x* s)) 80 (defvar *y* (make-array 10 :element-type 'double-float :initial-element 10.0d0))
(with-open-file (s "z.dat") (read-vector *y* s))
*y* #(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0) }}} -- Duane Rettig"
Further correspondence with Raymond Toy corroborates this is a bug:
"Yes, this does appear to be a bug in the implementation of read-vector.
You can, however, achieve what you want by opening the file with an element-type of, say, (unsigned-byte 8), instead of the default 'character.
I'll have to read some more to understand how read-vector interacts with the stream element type. It seems, though, that the element-type of the vector overrides the element-type of the stream, more or less. Currently, a stream element type of character basically causes the code to read in characters.
Ray"
If you need more information, let me know and I'll see what I can do.
-jeff
Comment (by rtoy):
First I added a bit of markup so that the code bits are formatted better.
Now for the bug report, there are a couple of issues here. CMUCL has an extension, READ-VECTOR, which is similar to the SIMPLE-STREAM's READ- VECTOR, but not quite. In particular, it produces the bug that you see. However, if you {{{(require 'simple-streams)}}}, READ-VECTOR will be replaced with the simple-stream version. Then, the following snip of code works as Duane mentions:
{{{ (let ((vec (make-array 10 :element-type 'double-float))) (with-open-file (os "d10.bin" :class 'stream:file-simple- stream) (stream:read-vector vec os) (print vec))) }}}
The important part is that you open the file with the appropriate class.
Not sure what to do. We can change READ-VECTOR to match simple-stream READ-VECTOR. (BTW, READ-VECTOR is much faster than STREAM:READ-VECTOR.)