[Asdf-devel] Need to do uiop:setup-temporary-directory in image built with uiop
Dear ASDF maintainer(s), I just noticed that it is necessary to do (uiop:setup-temporary-directory) in an image which has UIOP built into it, when that image is run on a different machine or under a different user account (at least on Windows). Otherwise, the function (uiop:temporary-directory) returns whatever was the temporary directory as computed in the build environment (in my case it's in the "dcooper8" home directory. So am I correct that it's pretty much a requirement to call (uiop:setup-temporary-directory) upon startup of an image which contains uiop built into it? Are there any other functions which should be called to initialize uiop? Is there a way to achieve this at image dump time instead of startup time (e.g. clearing the temporary directory entirely so it will be forced to reset when next demanded in the built image)? If I'm missing something from the documentation on this please feel free to slap me in that direction. Regards Dave Cooper
Dave Cooper wrote:
Dear ASDF maintainer(s),
I just noticed that it is necessary to do
(uiop:setup-temporary-directory)
in an image which has UIOP built into it, when that image is run on a different machine or under a different user account (at least on Windows). Otherwise, the function
(uiop:temporary-directory)
returns whatever was the temporary directory as computed in the build environment (in my case it's in the "dcooper8" home directory.
So am I correct that it's pretty much a requirement to call
(uiop:setup-temporary-directory)
upon startup of an image which contains uiop built into it?
Are there any other functions which should be called to initialize uiop?
Is there a way to achieve this at image dump time instead of startup time (e.g. clearing the temporary directory entirely so it will be forced to reset when next demanded in the built image)?
If I'm missing something from the documentation on this please feel free to slap me in that direction.
I am looking at uiop/image.lisp and it looks like it should do what you want out of the box. At the very bottom of the file I see a call to REGISTER-IMAGE-RESTORE-HOOK, and one of the elements of that list is setup-temporary-directory. So it seems like either this call is not happening properly or there's a bug in setup-temporary-directory. Is there any chance you could have overwritten UIOP's restore hook? Would it be possible for you to trace RESTORE-IMAGE and SETUP-TEMPORARY-DIRECTORY to see what's happening? I confess to not knowing whether TRACE will persist over image dumping.... Thanks, r
I think my issue is that I am not using the UIOP image dump facility. This is because in Allegro CL we use thei built-in excl functions like excl:generate-application --- it's not a matter of a generic dumplisp. So I'm pretty sure the solution is to call the register-image-restore-hook myself as part of e.g. the post-load-form of e.g. excl:generate-application. Longer term, maybe I can look into extending the UIOP image dump mechanism itself, to be able to use the different mechanisms in Allegro in addition to the generic excl:dumplisp. I will study the code and that the current documentation some more, and assuming the current code is sufficient for these situations, I'll see what I can do at least to contribute to the documentation to cover these kinds of situations. On Tuesday, June 17, 2014, Robert P. Goldman <rpgoldman@sift.info> wrote:
Dave Cooper wrote:
Dear ASDF maintainer(s),
I just noticed that it is necessary to do
(uiop:setup-temporary-directory)
in an image which has UIOP built into it, when that image is run on a different machine or under a different user account (at least on Windows). Otherwise, the function
(uiop:temporary-directory)
returns whatever was the temporary directory as computed in the build environment (in my case it's in the "dcooper8" home directory.
So am I correct that it's pretty much a requirement to call
(uiop:setup-temporary-directory)
upon startup of an image which contains uiop built into it?
Are there any other functions which should be called to initialize uiop?
Is there a way to achieve this at image dump time instead of startup time (e.g. clearing the temporary directory entirely so it will be forced to reset when next demanded in the built image)?
If I'm missing something from the documentation on this please feel free to slap me in that direction.
I am looking at uiop/image.lisp and it looks like it should do what you want out of the box. At the very bottom of the file I see a call to REGISTER-IMAGE-RESTORE-HOOK, and one of the elements of that list is setup-temporary-directory.
So it seems like either this call is not happening properly or there's a bug in setup-temporary-directory. Is there any chance you could have overwritten UIOP's restore hook?
Would it be possible for you to trace RESTORE-IMAGE and SETUP-TEMPORARY-DIRECTORY to see what's happening? I confess to not knowing whether TRACE will persist over image dumping....
Thanks, r
-- My Best, Dave Cooper, Genworks Support david.cooper@genworks.com, dave.genworks.com(skype) USA: 248-327-3253(o), 1-248-330-2979(mobile) UK: 0191 645 1699
Dave Cooper wrote:
I think my issue is that I am not using the UIOP image dump facility. This is because in Allegro CL we use thei built-in excl functions like excl:generate-application --- it's not a matter of a generic dumplisp.
So I'm pretty sure the solution is to call the register-image-restore-hook myself as part of e.g. the post-load-form of e.g. excl:generate-application. Longer term, maybe I can look into extending the UIOP image dump mechanism itself, to be able to use the different mechanisms in Allegro in addition to the generic excl:dumplisp.
You might also want to contact the Franz folks and warn them that ASDF may corrupt images that they dump, and have them fix their application-building and image-dumping facilities to avoid these problems. Maybe Franz should extend their copy of the ASDF library to infiltrate the image-handling process when they bundle it into ACL. I'd encourage you to do that, and CC me when you submit the bugreport to Franz. Best, R
On Tue, Jun 17, 2014 at 2:51 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
Dave Cooper wrote:
I think my issue is that I am not using the UIOP image dump facility. This is because in Allegro CL we use thei built-in excl functions like excl:generate-application --- it's not a matter of a generic dumplisp.
So I'm pretty sure the solution is to call the register-image-restore-hook myself as part of e.g. the post-load-form of e.g. excl:generate-application. Longer term, maybe I can look into extending the UIOP image dump mechanism itself, to be able to use the different mechanisms in Allegro in addition to the generic excl:dumplisp.
You might also want to contact the Franz folks and warn them that ASDF may corrupt images that they dump, and have them fix their application-building and image-dumping facilities to avoid these problems.
Maybe Franz should extend their copy of the ASDF library to infiltrate the image-handling process when they bundle it into ACL.
Corrupt is a big word. Fail to implicitly call ASDF-defined image hooks. It is probably a bug that there isn't a uiop/stream:clear-temporary-directory that gets registered via uiop/configuration:register-clear-configuration-hook in uiop/configuration. I can do it, though I'd rather someone else do it. That said, even with that hook function defined and registered, you still need to explicitly (call-image-dump-hook) before you dump an image using implementation-specific tools, and you need to (call-image-restore-hook) when you restart.
I'd encourage you to do that, and CC me when you submit the bugreport to Franz.
I don't believe there's any Franz bug. It's a bug in how UIOP is not used properly. These hooks probably need to be documented better, too. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org He was a great patriot, a humanitarian, a loyal friend; provided, of course, he really is dead.
Faré wrote:
On Tue, Jun 17, 2014 at 2:51 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
Dave Cooper wrote:
I think my issue is that I am not using the UIOP image dump facility. This is because in Allegro CL we use thei built-in excl functions like excl:generate-application --- it's not a matter of a generic dumplisp.
So I'm pretty sure the solution is to call the register-image-restore-hook myself as part of e.g. the post-load-form of e.g. excl:generate-application. Longer term, maybe I can look into extending the UIOP image dump mechanism itself, to be able to use the different mechanisms in Allegro in addition to the generic excl:dumplisp. You might also want to contact the Franz folks and warn them that ASDF may corrupt images that they dump, and have them fix their application-building and image-dumping facilities to avoid these problems.
Maybe Franz should extend their copy of the ASDF library to infiltrate the image-handling process when they bundle it into ACL.
Corrupt is a big word. Fail to implicitly call ASDF-defined image hooks.
It is probably a bug that there isn't a uiop/stream:clear-temporary-directory that gets registered via uiop/configuration:register-clear-configuration-hook in uiop/configuration. I can do it, though I'd rather someone else do it.
I can do this, but I need some help understanding what's going on here. The following is a little circular: (defun clear-configuration () "Call the functions in *CLEAR-CONFIGURATION-HOOK*" What is the configuration? ASDF has at least two configurations: the configuration of source locations and the configuration of output-file translations. Is this one of those, or something else? I'm a bit confused because uiop/configuration.lisp looks like it provides a *general* utility for configuring things. And TEMPORARY-FILE is something that is yet another aspect of configuration, outside both of these. If you can provide me a bit of a directive, I'll try to fix this. I assume that when I understand this, I'll be able to understand how to test it.
That said, even with that hook function defined and registered, you still need to explicitly (call-image-dump-hook) before you dump an image using implementation-specific tools, and you need to (call-image-restore-hook) when you restart.
I'd encourage you to do that, and CC me when you submit the bugreport to Franz.
I don't believe there's any Franz bug. It's a bug in how UIOP is not used properly. These hooks probably need to be documented better, too.
I don't agree. If a lisp implementation ships with ASDF, and a lisp image dumping facility, then to the maximum extent possible the lisp image dumping facility should not require additional work on the user's part. Also, it's likely to make the implementation maintainers happier if they don't have to get complaints from users about cryptic errors from the combination of these two features. Suggestion: if there are bits of configuration that can leak out of ASDF and UIOP across image dumpings, let's try to document them, and make a note for implementers that they may wish to appropriately modify their image-dumping facilities to behave properly with their ASDF without requiring additional user actions. If worst comes to worst, and the implementors don't feel like handling this, the documentation will be useful to the ASDF user who has to handle it him or herself. Follow-up question: is it getting to be time to add a separate manual for UIOP? Cheers, r
On Tue, Jun 17, 2014 at 4:33 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
It is probably a bug that there isn't a uiop/stream:clear-temporary-directory that gets registered via uiop/configuration:register-clear-configuration-hook in uiop/configuration. I can do it, though I'd rather someone else do it.
I can do this, but I need some help understanding what's going on here. The following is a little circular:
(defun clear-configuration () "Call the functions in *CLEAR-CONFIGURATION-HOOK*"
What is the configuration? ASDF has at least two configurations: the configuration of source locations and the configuration of output-file translations. Is this one of those, or something else? I'm a bit confused because uiop/configuration.lisp looks like it provides a *general* utility for configuring things. And TEMPORARY-FILE is something that is yet another aspect of configuration, outside both of these.
clear-configuration is the function you call when configuration files may have been modified and you want to invalidate any current configuration loaded from them. All it does is call the functions in the *clear-configuration-hook*, so you can extend what "the configuration" means. UIOP itself has none (so far), but ASDF has configuration for source-registry and output-translations. clear-configuration is itself registered as an image-dump-hook, so when you save an image, you don't get configuration from whichever user, process, machine dumped the image. I'm not completely sure whether you should (register-configuration-hook 'clear-temporary-directory) or more directly (register-image-dump-hook 'clear-temporary-directory) Since default-temporary-directory (at least for now) doesn't consult any configuration file, only an environment variable, that is unlikely to change during execution, an image-dump-hook is probably all you need. It's a subtle difference in the intent of future possible modifications.
I assume that when I understand this, I'll be able to understand how to test it. Testing this functionality involves dumping an image while TMPDIR is something and running it while it is something else. Sigh. Could be done in test-program.script, I suppose.
That said, even with that hook function defined and registered, you still need to explicitly (call-image-dump-hook) before you dump an image using implementation-specific tools, and you need to (call-image-restore-hook) when you restart.
I'd encourage you to do that, and CC me when you submit the bugreport to Franz.
I don't believe there's any Franz bug. It's a bug in how UIOP is not used properly. These hooks probably need to be documented better, too.
I don't agree. If a lisp implementation ships with ASDF, and a lisp image dumping facility, then to the maximum extent possible the lisp image dumping facility should not require additional work on the user's part. Also, it's likely to make the implementation maintainers happier if they don't have to get complaints from users about cryptic errors from the combination of these two features.
Well, so far UIOP is defined as a layer on top of what the implementation provides and not the other way around. That said, it might be great if the documentation for these implementations would remind the users that if they use these functions while building their software with ASDF (or similar software), they may want to explicitly (uiop:call-image-dump-hook) and (uiop:call-image-restore-hook) (or the equivalent for their software).
Suggestion: if there are bits of configuration that can leak out of ASDF and UIOP across image dumpings, let's try to document them, and make a note for implementers that they may wish to appropriately modify their image-dumping facilities to behave properly with their ASDF without requiring additional user actions.
If there's a leak, it's probably a bug, to be fixed with an appropriate image-dump-hook.
If worst comes to worst, and the implementors don't feel like handling this, the documentation will be useful to the ASDF user who has to handle it him or herself.
Follow-up question: is it getting to be time to add a separate manual for UIOP?
That would be great. I was kind of hoping for one of the many automatic documentation generators to be used based on the many docstrings. There are just too many functions to hope to maintain the documentation outside of said docstrings and other source comments. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "Transported to a surreal landscape, a young girl kills the first woman she meets and then teams up with three complete strangers to kill again." — TV listing for the Wizard of Oz in the Marin Independent Journal
Faré wrote:
On Tue, Jun 17, 2014 at 4:33 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
It is probably a bug that there isn't a uiop/stream:clear-temporary-directory that gets registered via uiop/configuration:register-clear-configuration-hook in uiop/configuration. I can do it, though I'd rather someone else do it. I can do this, but I need some help understanding what's going on here. The following is a little circular:
(defun clear-configuration () "Call the functions in *CLEAR-CONFIGURATION-HOOK*"
What is the configuration? ASDF has at least two configurations: the configuration of source locations and the configuration of output-file translations. Is this one of those, or something else? I'm a bit confused because uiop/configuration.lisp looks like it provides a *general* utility for configuring things. And TEMPORARY-FILE is something that is yet another aspect of configuration, outside both of these.
clear-configuration is the function you call when configuration files may have been modified and you want to invalidate any current configuration loaded from them. All it does is call the functions in the *clear-configuration-hook*, so you can extend what "the configuration" means. UIOP itself has none (so far),
I'm confused by this given what David Cooper says. Now, it's maybe not a *user-managed* configuration, but isn't the cached temp directory setting a configuration? but ASDF has configuration for
source-registry and output-translations.
Right, and that's what confuses me. For the source-registry, we have CLEAR-SOURCE-REGISTRY, and for output-translations we have CLEAR-OUTPUT-TRANSLATIONS, so what's a one-sentence description of what UIOP:CLEAR-CONFIGURATION is supposed to do? Does it subsume those two, so that calling UIOP:CLEAR-CONFIGURATION effectively clears both source-registry and output-translations? Do you see what I mean? Without a crisp definition of "configuration," I don't know what "clear-configuration" should do...
clear-configuration is itself registered as an image-dump-hook, so when you save an image, you don't get configuration from whichever user, process, machine dumped the image.
I'm not completely sure whether you should (register-configuration-hook 'clear-temporary-directory) or more directly (register-image-dump-hook 'clear-temporary-directory)
Since default-temporary-directory (at least for now) doesn't consult any configuration file, only an environment variable, that is unlikely to change during execution, an image-dump-hook is probably all you need. It's a subtle difference in the intent of future possible modifications.
I assume that when I understand this, I'll be able to understand how to test it. Testing this functionality involves dumping an image while TMPDIR is something and running it while it is something else. Sigh. Could be done in test-program.script, I suppose.
That said, even with that hook function defined and registered, you still need to explicitly (call-image-dump-hook) before you dump an image using implementation-specific tools, and you need to (call-image-restore-hook) when you restart.
I'd encourage you to do that, and CC me when you submit the bugreport to Franz.
I don't believe there's any Franz bug. It's a bug in how UIOP is not used properly. These hooks probably need to be documented better, too. I don't agree. If a lisp implementation ships with ASDF, and a lisp image dumping facility, then to the maximum extent possible the lisp image dumping facility should not require additional work on the user's part. Also, it's likely to make the implementation maintainers happier if they don't have to get complaints from users about cryptic errors from the combination of these two features.
Well, so far UIOP is defined as a layer on top of what the implementation provides and not the other way around. That said, it might be great if the documentation for these implementations would remind the users that if they use these functions while building their software with ASDF (or similar software), they may want to explicitly (uiop:call-image-dump-hook) and (uiop:call-image-restore-hook) (or the equivalent for their software).
Well, maybe UIOP is a layer on top, but it's a layer on top that seeps down (in a way that seems very much analogous to the way that the readtable seeps down from the REPL into the compiler...).
Suggestion: if there are bits of configuration that can leak out of ASDF and UIOP across image dumpings, let's try to document them, and make a note for implementers that they may wish to appropriately modify their image-dumping facilities to behave properly with their ASDF without requiring additional user actions.
If there's a leak, it's probably a bug, to be fixed with an appropriate image-dump-hook.
If worst comes to worst, and the implementors don't feel like handling this, the documentation will be useful to the ASDF user who has to handle it him or herself.
Follow-up question: is it getting to be time to add a separate manual for UIOP?
That would be great. I was kind of hoping for one of the many automatic documentation generators to be used based on the many docstrings. There are just too many functions to hope to maintain the documentation outside of said docstrings and other source comments.
Maybe I'll see what I can manage with Didier's library. I think it allows us to wrap structure around docstrings (like the way the SBCL doc gen does). I like documentation generators for exactly the reason you do -- it keeps the documentation for functions close to their definitions so they don't stray. But I *dislike* them when, like doxygen and javadoc, they just give you a big heap of unstructured mess. I'd like to structure the docstrings into a manual-shaped skeleton. Wish I was going to ILC to talk about this with Didier.... :-( Best, r
On Tue, Jun 17, 2014 at 11:47 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
clear-configuration is the function you call when configuration files may have been modified and you want to invalidate any current configuration loaded from them. All it does is call the functions in the *clear-configuration-hook*, so you can extend what "the configuration" means. UIOP itself has none (so far),
I'm confused by this given what David Cooper says. Now, it's maybe not a *user-managed* configuration, but isn't the cached temp directory setting a configuration?
That's one way of seeing it. But currently, there's no clear-temporary-directory, and it's not registered in either the clear-configuration-hook or the image-dump-hook.
but ASDF has configuration for source-registry and output-translations.
Right, and that's what confuses me. For the source-registry, we have CLEAR-SOURCE-REGISTRY, and for output-translations we have CLEAR-OUTPUT-TRANSLATIONS, so what's a one-sentence description of what UIOP:CLEAR-CONFIGURATION is supposed to do? Does it subsume those two, so that calling UIOP:CLEAR-CONFIGURATION effectively clears both source-registry and output-translations?
Yes, these two functions are both registered in the clear-configuration-hook, so that UIOP:CLEAR-CONFIGURATION will call them both. Actually, nothing else currently uses the *clear-configuration-hook*, but other systems may, and quite possibly should.
Do you see what I mean? Without a crisp definition of "configuration," I don't know what "clear-configuration" should do...
It's an open question. The intent is that other libraries with similar configuration needs would register more stuff in the configuration hooks.
clear-configuration is itself registered as an image-dump-hook, so when you save an image, you don't get configuration from whichever user, process, machine dumped the image.
I'm not completely sure whether you should (register-configuration-hook 'clear-temporary-directory) or more directly (register-image-dump-hook 'clear-temporary-directory)
Since default-temporary-directory (at least for now) doesn't consult any configuration file, only an environment variable, that is unlikely to change during execution, an image-dump-hook is probably all you need. It's a subtle difference in the intent of future possible modifications.
Well, so far UIOP is defined as a layer on top of what the implementation provides and not the other way around. That said, it might be great if the documentation for these implementations would remind the users that if they use these functions while building their software with ASDF (or similar software), they may want to explicitly (uiop:call-image-dump-hook) and (uiop:call-image-restore-hook) (or the equivalent for their software).
Well, maybe UIOP is a layer on top, but it's a layer on top that seeps down (in a way that seems very much analogous to the way that the readtable seeps down from the REPL into the compiler...).
Well, yes, sometimes you want to call UIOP, and sometimes, you want UIOP to call you, using its hooks. Yes, it does require users to do proper configuration — but just imagine how hard it is if they try to do it without UIOP: now every system must invent its own API and implement it on every implementation, and every user must learn the API of every system. Ouch. And if you don't use UIOP, it's not integrated with the system that builds binaries, so you have to write your own build scripts, too.
Follow-up question: is it getting to be time to add a separate manual for UIOP?
That would be great. I was kind of hoping for one of the many automatic documentation generators to be used based on the many docstrings. There are just too many functions to hope to maintain the documentation outside of said docstrings and other source comments.
Maybe I'll see what I can manage with Didier's library. I think it allows us to wrap structure around docstrings (like the way the SBCL doc gen does). I like documentation generators for exactly the reason you do -- it keeps the documentation for functions close to their definitions so they don't stray. But I *dislike* them when, like doxygen and javadoc, they just give you a big heap of unstructured mess. I'd like to structure the docstrings into a manual-shaped skeleton.
Wish I was going to ILC to talk about this with Didier.... :-(
You can fire him a mail, and maybe hangout — if he has time! —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org There are three types of people in the world; those who can count, and those who can't.
participants (3)
-
Dave Cooper -
Faré -
Robert P. Goldman