[Antik-devel] Reader macro for sharp-m returns literal foreign-array (patch included)
Hi, I have encountered the following problem a couple of times before, although not as reproducibly as the memory corruption issue from my previous message: If I use the #m reader macro in a function somewhere, dump an SBCL core image, load the image, and call the function, I get memory corruption. I have mostly dealt with it up until now by not using the #m() notation in anything that I plan to dump to a binary. I believe that the cause is that the #m macro returns a foreign-array object directly (when foreign-array is the default grid type). If it returned a `make-grid' form instead, it would operate identically on e.g. the REPL. It would have the advantage, however, of being usable in a dumped image, since the array would be created at run-time instead of at read-time, and hence no foreign-pointers would be saved into the image. I've attached a patch that changes the #m reader macro to return a creation form instead of a literal. I hope you'll consider adding it. I've pasted a repl session below that demonstrates the difference. Thanks, James ;;; Before the patch GRID> (read-from-string "#m(1d0 2d0)") #m(1.000000000000000d0 2.000000000000000d0) 11 GRID> #m(1d0 2d0) #m(1.000000000000000d0 2.000000000000000d0) ;;; After the patch GRID> (read-from-string "#m(1d0 2d0)") (MAKE-GRID '((FOREIGN-ARRAY) DOUBLE-FLOAT) :INITIAL-CONTENTS '(1.0 2.0)) 11 GRID> #m(1d0 2d0) #m(1.000000000000000d0 2.000000000000000d0)
On Fri, Dec 9, 2011 at 4:04 PM, James Wright <james@chumsley.org> wrote:
Hi,
I have encountered the following problem a couple of times before, although not as reproducibly as the memory corruption issue from my previous message: If I use the #m reader macro in a function somewhere, dump an SBCL core image, load the image, and call the function, I get memory corruption. I have mostly dealt with it up until now by not using the #m() notation in anything that I plan to dump to a binary.
This is odd. There is a make-load-form for foreign-arrays, so I assumed this would work. Example?
I believe that the cause is that the #m macro returns a foreign-array object directly (when foreign-array is the default grid type). If it returned a `make-grid' form instead, it would operate identically on e.g. the REPL. It would have the advantage, however, of being usable in a dumped image, since the array would be created at run-time instead of at read-time, and hence no foreign-pointers would be saved into the image.
My intent is that it be analogous to #, e.g. #(1.0d0 2.0d0) expands to a (CL) array. I used to have it the way you want but I changed it as of 3e610bd4d5261 to be analogous to the regular array macro (in fact, #m should make a CL array if grid:*default-grid-type* is set accordingly; I'm still not sure about the wisdom of doing that). I had a good reason for changing it, but I can't recall exactly what it was. Something was not working right is all I recall. By the way, do you know about grid:grid? Maybe that's what you want. It's a function, so it evaluates its arguments (grid:grid pi pi pi) #m(3.141592653589793d0 3.141592653589793d0 3.141592653589793d0)
I've attached a patch that changes the #m reader macro to return a creation form instead of a literal. I hope you'll consider adding it. I've pasted a repl session below that demonstrates the difference.
Thanks,
James
I think I'd like to figure out why #m can't be saved before I consider changing this back. Liam
Hi Liam,
I have encountered the following problem a couple of times before, although not as reproducibly as the memory corruption issue from my previous message: If I use the #m reader macro in a function somewhere, dump an SBCL core image, load the image, and call the function, I get memory corruption. I have mostly dealt with it up until now by not using the #m() notation in anything that I plan to dump to a binary.
This is odd. There is a make-load-form for foreign-arrays, so I assumed this would work. Example?
I'm working on trying to reproduce this, but I haven't had any luck so far. (This is why memory issues are such a joy). However, I don't think that SBCL uses make-load-form when dumping an image; my understanding is that a dumped image is really just a copy of the raw memory bytes. I've attached a script with output that demonstrates that SBCL seems to want a make-load-form method for writing a FASL but not for writing a core image.
My intent is that it be analogous to #, e.g. #(1.0d0 2.0d0) expands to a (CL) array. I used to have it the way you want but I changed it as of 3e610bd4d5261 to be analogous to the regular array macro
Fair enough. My version does not evaluate its arguments, so in that sense it is the same as #( or #2a(, but on the other hand it does return a form instead of a literal value, so in that sense it is less like #(.
(in fact, #m should make a CL array if grid:*default-grid-type* is set accordingly; I'm still not sure about the wisdom of doing that).
It is nice to be able to switch seamlessly between CL and foreign arrays. One funny situation the current version leads to is that evaluating the string printed by the REPL for a foreign array can give you a CL array back: ANTIK-USER> (setf *default-grid-type* 'array) ARRAY ANTIK-USER> (make-foreign-array 'double-float :initial-contents '(1d0 2d0)) #m(1.000000000000000d0 2.000000000000000d0) ANTIK-USER> #m(1.000000000000000d0 2.000000000000000d0) #(1d0 2d0)
By the way, do you know about grid:grid? Maybe that's what you want. It's a function, so it evaluates its arguments (grid:grid pi pi pi) #m(3.141592653589793d0 3.141592653589793d0 3.141592653589793d0)
I didn't know about that; it looks like it will simplify my code. Thanks!
I think I'd like to figure out why #m can't be saved before I consider changing this back.
I think it's analogous to the problem with foreign pointers in global vars: if you have an actual literal foreign array in your code, then that is what the image will contain when you dump core, so it'll come back as an uninitialized foreign pointer; but if you have a form for creating a foreign array in your code, then the creation form (or rather the code that it compiles down to, of course) is what will get dumped. I'll spend a bit more time next week and see if I can cough up a failing example. At any rate, it's an easy problem for end users to avoid, so I don't want to come across as badgering. But I do think it's a bit of a stability landmine to be aware of for people that need to dump binaries instead of being able to run from fasls; needing a binary is my typical gsll use case, since I use gsll on a torque cluster. Thanks! James
Hi Liam,
If I use the #m reader macro in a function somewhere, dump an SBCL core image, load the image, and call the function, I get memory corruption. I have mostly dealt with it up until now by not using the #m() notation in anything that I plan to dump to a binary.
This is odd. There is a make-load-form for foreign-arrays, so I assumed this would work. Example?
I've managed to come up with a script that I believe demonstrates the problem with #m returning a literal foreign-array. I've attached the script to this message, as well as a text file containing the output on my machine. The script loads a program containing a #m-defined foreign array in a let form in the `main' function, runs the function (which just prints the array), and dumps core. It then loads the core and runs the function again. In the second run, the the array contents are garbage, because the contents of the foreign memory were not dumped or restored, and the foreign pointer of the array, although it has the same numeric value as before, no longer points to memory that it owns. The script then loads the core again and runs a different function, `main2'. That function also contains a #m array in a let form; it freshly allocates a bunch of foreign memory until it gets back the same pointer that the #m array thinks it owns, and demonstrates that the two values use the same memory. I haven't managed to force a crash, but it seems clear that creating wild pointers opens up the possibility. Thanks, James
On Sun, Dec 18, 2011 at 1:13 AM, James Wright <james@chumsley.org> wrote:
Hi Liam,
If I use the #m reader macro in a function somewhere, dump an SBCL core image, load the image, and call the function, I get memory corruption. I have mostly dealt with it up until now by not using the #m() notation in anything that I plan to dump to a binary.
This is odd. There is a make-load-form for foreign-arrays, so I assumed this would work. Example?
I've managed to come up with a script that I believe demonstrates the problem with #m returning a literal foreign-array. I've attached the script to this message, as well as a text file containing the output on my machine.
The script loads a program containing a #m-defined foreign array in a let form in the `main' function, runs the function (which just prints the array), and dumps core. It then loads the core and runs the function again. In the second run, the the array contents are garbage, because the contents of the foreign memory were not dumped or restored, and the foreign pointer of the array, although it has the same numeric value as before, no longer points to memory that it owns.
The script then loads the core again and runs a different function, `main2'. That function also contains a #m array in a let form; it freshly allocates a bunch of foreign memory until it gets back the same pointer that the #m array thinks it owns, and demonstrates that the two values use the same memory.
I haven't managed to force a crash, but it seems clear that creating wild pointers opens up the possibility.
Thanks, James
James, After researching your previous issue, I think I have some insight on this one too. Unfortunately, contrary to my previous presumption, the save-lisp-and-die function does not in any way use make-load-form. So it's going to save any actual image with a foreign pointer that points to random nonsense. So, what to do about it. Your suggestion to expand to a form instead of the array itself should work. However, I would like to keep it as is; I changed it from form to object because I was having some kind of trouble using it analogously to # and #2A (unfortunately, I can't remember what that was). May I suggest that you use #'grid:grid instead of #m in these contexts so that you don't have problems. It is intended to be analogous to #'list and #'vector, i.e., it is code to build the object rather than the object itself. I wrote this function to be the evaluating equivalent of #m. That should work just fine in your usage. What do you think? Liam
Hi Liam,
After researching your previous issue, I think I have some insight on this one too. Unfortunately, contrary to my previous presumption, the save-lisp-and-die function does not in any way use make-load-form. So it's going to save any actual image with a foreign pointer that points to random nonsense.
That certainly fits with the behavior I was seeing.
However, I would like to keep it as is; I changed it from form to object because I was having some kind of trouble using it analogously to # and #2A (unfortunately, I can't remember what that was). May I suggest that you use #'grid:grid instead of #m in these contexts so that you don't have problems.
Sure, sounds pretty reasonable to me. Thanks! James
participants (2)
-
James Wright -
Liam Healy