[asdf-devel] Hoist by my own petard: problem with asdf-binary-locations compatibility.
One of my own systems got blown up by ASDF-OUTPUT-TRANSLATIONS recently, and the case may be an interesting one: I am working on a system that involves taking an ontology from the Protege tool, and interpreting it as CL. So we have a special component type which is "pont" (from Protege ontology). A pont file is /almost/ readable CL, but not quite. So before we can do the compile-op on the pont file, we first do a pont-to-lisp operation on it, which involves calling a perl script to generate a lisp file from the pont file. So..... We have a pont-file which ISA cl-source-file. But there is no .lisp file to begin with. Instead, we have the following chain of operations: pont-file ==== pont-to-lisp ====> lisp file ==== compile-op ====> fasl This works fine in ASDF Classic, even with asdf-binary-locations, because asdf-binary-locations ONLY rewrites the output-files of COMPILE-OP. So, the output of the pont-to-lisp file, the lisp file, gets written into the source directory where the compile-op can find it. Unfortunately, this is not what happens with ASDF-OUTPUT-LOCATIONS, because A-O-L translates /all/ output-files, not just those of compile-op. This means that our lisp file gets dropped somewhere into /var/cache/common-lisp-controller/501/.... [Is this a misconfiguration of A-O-L, btw? My colleague is compiling this code as an ordinary user. I guess I'm surprised that the default is to drop stuff into /var instead of into a userdir. Seems like this is going to create a lot of dark matter in the filesystem, since one will have to look pretty hard to clean up zombie fasls... I would have thought something in the user's homedir would be more likely as a default...] I'm afraid this suggests that we should tweak the asdf-binary-locations compatibility code yet again. I will see if I can suggest a patch. BTW, I discovered this because some ubuntu version seems to be shipping with common-lisp-controller with an ASDF 2 from git's womb untimely ripp'd... So Ubuntu people are unwittingly getting their SBCL served to them with ASDF2 rolled into it. Slightly alarming. Best, r
To prevent Lisp files from being redirected, you can add a rule such as ((:root #p"**/*.lisp") t) in your translations. OR, you can make sure your output-file methods return T as a second value to prevent relocation. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] To seek God is to find him. To seek happiness is to miss it. On 14 April 2010 21:04, Robert Goldman <rpgoldman@sift.info> wrote:
One of my own systems got blown up by ASDF-OUTPUT-TRANSLATIONS recently, and the case may be an interesting one:
I am working on a system that involves taking an ontology from the Protege tool, and interpreting it as CL.
So we have a special component type which is "pont" (from Protege ontology).
A pont file is /almost/ readable CL, but not quite. So before we can do the compile-op on the pont file, we first do a pont-to-lisp operation on it, which involves calling a perl script to generate a lisp file from the pont file.
So.....
We have a pont-file which ISA cl-source-file.
But there is no .lisp file to begin with. Instead, we have the following chain of operations:
pont-file ==== pont-to-lisp ====> lisp file ==== compile-op ====> fasl
This works fine in ASDF Classic, even with asdf-binary-locations, because asdf-binary-locations ONLY rewrites the output-files of COMPILE-OP.
So, the output of the pont-to-lisp file, the lisp file, gets written into the source directory where the compile-op can find it.
Unfortunately, this is not what happens with ASDF-OUTPUT-LOCATIONS, because A-O-L translates /all/ output-files, not just those of compile-op. This means that our lisp file gets dropped somewhere into /var/cache/common-lisp-controller/501/....
[Is this a misconfiguration of A-O-L, btw? My colleague is compiling this code as an ordinary user. I guess I'm surprised that the default is to drop stuff into /var instead of into a userdir. Seems like this is going to create a lot of dark matter in the filesystem, since one will have to look pretty hard to clean up zombie fasls... I would have thought something in the user's homedir would be more likely as a default...]
I'm afraid this suggests that we should tweak the asdf-binary-locations compatibility code yet again.
I will see if I can suggest a patch.
BTW, I discovered this because some ubuntu version seems to be shipping with common-lisp-controller with an ASDF 2 from git's womb untimely ripp'd... So Ubuntu people are unwittingly getting their SBCL served to them with ASDF2 rolled into it. Slightly alarming.
Best, r
On 4/14/10 Apr 14 -8:41 PM, Faré wrote:
To prevent Lisp files from being redirected, you can add a rule such as ((:root #p"**/*.lisp") t) in your translations.
OR, you can make sure your output-file methods return T as a second value to prevent relocation.
Right. I understand that, and I can make ASDF2 do the right thing for this case. But this shows an imperfection in our backward compatibility. I will see if I can propose a fix for this. I was pretty shocked, actually, that a git version of ASDF2 was shipped in ubuntu... best, r
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] To seek God is to find him. To seek happiness is to miss it.
On 14 April 2010 21:04, Robert Goldman <rpgoldman@sift.info> wrote:
One of my own systems got blown up by ASDF-OUTPUT-TRANSLATIONS recently, and the case may be an interesting one:
I am working on a system that involves taking an ontology from the Protege tool, and interpreting it as CL.
So we have a special component type which is "pont" (from Protege ontology).
A pont file is /almost/ readable CL, but not quite. So before we can do the compile-op on the pont file, we first do a pont-to-lisp operation on it, which involves calling a perl script to generate a lisp file from the pont file.
So.....
We have a pont-file which ISA cl-source-file.
But there is no .lisp file to begin with. Instead, we have the following chain of operations:
pont-file ==== pont-to-lisp ====> lisp file ==== compile-op ====> fasl
This works fine in ASDF Classic, even with asdf-binary-locations, because asdf-binary-locations ONLY rewrites the output-files of COMPILE-OP.
So, the output of the pont-to-lisp file, the lisp file, gets written into the source directory where the compile-op can find it.
Unfortunately, this is not what happens with ASDF-OUTPUT-LOCATIONS, because A-O-L translates /all/ output-files, not just those of compile-op. This means that our lisp file gets dropped somewhere into /var/cache/common-lisp-controller/501/....
[Is this a misconfiguration of A-O-L, btw? My colleague is compiling this code as an ordinary user. I guess I'm surprised that the default is to drop stuff into /var instead of into a userdir. Seems like this is going to create a lot of dark matter in the filesystem, since one will have to look pretty hard to clean up zombie fasls... I would have thought something in the user's homedir would be more likely as a default...]
I'm afraid this suggests that we should tweak the asdf-binary-locations compatibility code yet again.
I will see if I can suggest a patch.
BTW, I discovered this because some ubuntu version seems to be shipping with common-lisp-controller with an ASDF 2 from git's womb untimely ripp'd... So Ubuntu people are unwittingly getting their SBCL served to them with ASDF2 rolled into it. Slightly alarming.
Best, r
On 4/14/10 Apr 14 -8:52 PM, Robert Goldman wrote:
On 4/14/10 Apr 14 -8:41 PM, Faré wrote:
To prevent Lisp files from being redirected, you can add a rule such as ((:root #p"**/*.lisp") t) in your translations.
OR, you can make sure your output-file methods return T as a second value to prevent relocation.
Right. I understand that, and I can make ASDF2 do the right thing for this case. But this shows an imperfection in our backward compatibility. I will see if I can propose a fix for this.
Actually, this seems more difficult than I had hoped. Here's the problem as I see it: 1. asdf-binary-locations compatbility does not set any state variable to indicate that ASDF is in compatbility mode. 2. The most straightforward way to restore a-b-l compatibility wrt this case would be, in the :around method for output-files, to check to see if we are in compatibliity mode, and then only rewrite output-files when the operation is compile-op. 3. Unfortunately, there is no way for output-files to detect that ASDF is in compatibility mode. I am reluctant to add a global variable into your lovely code, but I don't see an alternative. Yes, I see that I can punch a hole in the rewriting so that lisp files are not rewritten, but that doesn't solve the compatibility issue. Whether or not files are rewritten by A-B-L is a matter of both the operation and the component types. Neither of these is checked in the new code, which is only a function of pathnames. There is no set of filename patterns that I can assemble that can reliably reflect the criteria that Gary used in his code, because those criteria are not based on pathnames exclusively, but on ASDF operations and components, as well. I'm going to push a branch with a proposed fix, but will not push the fix onto the master branch, hoping that someone can come up with something that does not involve a global variable. OTOH, the sets of output-translation rules are effectively a global variable, too, so I should probably just get over this. best, r
I was pretty shocked, actually, that a git version of ASDF2 was shipped in ubuntu...
best, r
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] To seek God is to find him. To seek happiness is to miss it.
On 14 April 2010 21:04, Robert Goldman <rpgoldman@sift.info> wrote:
One of my own systems got blown up by ASDF-OUTPUT-TRANSLATIONS recently, and the case may be an interesting one:
I am working on a system that involves taking an ontology from the Protege tool, and interpreting it as CL.
So we have a special component type which is "pont" (from Protege ontology).
A pont file is /almost/ readable CL, but not quite. So before we can do the compile-op on the pont file, we first do a pont-to-lisp operation on it, which involves calling a perl script to generate a lisp file from the pont file.
So.....
We have a pont-file which ISA cl-source-file.
But there is no .lisp file to begin with. Instead, we have the following chain of operations:
pont-file ==== pont-to-lisp ====> lisp file ==== compile-op ====> fasl
This works fine in ASDF Classic, even with asdf-binary-locations, because asdf-binary-locations ONLY rewrites the output-files of COMPILE-OP.
So, the output of the pont-to-lisp file, the lisp file, gets written into the source directory where the compile-op can find it.
Unfortunately, this is not what happens with ASDF-OUTPUT-LOCATIONS, because A-O-L translates /all/ output-files, not just those of compile-op. This means that our lisp file gets dropped somewhere into /var/cache/common-lisp-controller/501/....
[Is this a misconfiguration of A-O-L, btw? My colleague is compiling this code as an ordinary user. I guess I'm surprised that the default is to drop stuff into /var instead of into a userdir. Seems like this is going to create a lot of dark matter in the filesystem, since one will have to look pretty hard to clean up zombie fasls... I would have thought something in the user's homedir would be more likely as a default...]
I'm afraid this suggests that we should tweak the asdf-binary-locations compatibility code yet again.
I will see if I can suggest a patch.
BTW, I discovered this because some ubuntu version seems to be shipping with common-lisp-controller with an ASDF 2 from git's womb untimely ripp'd... So Ubuntu people are unwittingly getting their SBCL served to them with ASDF2 rolled into it. Slightly alarming.
Best, r
_______________________________________________ asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
False alarm. A-B-L compatibility /is/ working here. Now I need to figure out /why/ it is working for compatibility, but fails when the compatibility is turned off (and the .lisp file gets written in the wrong place). Anyway, I clearly shouldn't go messing about with this until I'm sure I know what I'm doing... Best, r
good evening; On 2010-04-15, at 03:04 , Robert Goldman wrote:
One of my own systems got blown up by ASDF-OUTPUT-TRANSLATIONS recently, and the case may be an interesting one:
[ ... ] We have a pont-file which ISA cl-source-file.
But there is no .lisp file to begin with. Instead, we have the following chain of operations:
pont-file ==== pont-to-lisp ====> lisp file ==== compile-op ====> fasl
This works fine in ASDF Classic, even with asdf-binary-locations, because asdf-binary-locations ONLY rewrites the output-files of COMPILE-OP.
So, the output of the pont-to-lisp file, the lisp file, gets written into the source directory where the compile-op can find it.
Unfortunately, this is not what happens with ASDF-OUTPUT-LOCATIONS, because A-O-L translates /all/ output-files, not just those of compile-op. This means that our lisp file gets dropped somewhere into /var/cache/common-lisp-controller/501/....
when my former construction crew partner used to characterize the "sawzall"[1] as a "tool of the devil", he was right. it was rightly respected as an infernal device - exactly for its single-minded simplicity. and rightly in demand. a-o-t finds itself on the same page, not just for its dsl qualities, but just as well for the potential the undo the unsuspecting. in its case, however, more likely for the surprising consequences of unappreciated interacting factors. if only there were a simpler alternative... it has been suggested, that logical pathnames are an alternative. to the response, that logical pathnames themselves are not compatible with asdf realpolitik - real-world system component names simply do not conform to logical pathname restrictions. on the other hand, given that a-o-l devolves to translate-pathname, how far off can logical pathnames be from supporting the real asdf use cases? as an experiment on this topic it seemed worth while to see if it would be possible to arrange asdf to separate the component naming from the component location mapping. it turns out to be quite straight-forward. enclosed is a diff to asdf 1.502 which exhibits the following: - system definition proceeds in two phases. the first phase is the standard call to parse-component. this is followed immediately by an additional operation, bind-component-pathname, which walks the system top-down and computes an absolute pathname for each component from a combination of the parent absolute pathname, the component name, the component relative pathname, and the component type. - if the absolute pathname is a logical pathname, it is arranged that all components be valid and, where necessary, a map is constructed for components which must be "canonicalized" to retain the respective original term. - component-pathname, input-files, and output-files are modified to refer to the component's absolute pathname only. the first two resolve the absolute to a physical pathname and then map back components as appropriate. output-files first performs the input-
output translation.
the modified version passes all but four of the pathname tests. the fours test which expect the component named "module2/typed-file.type" to designate the file "typed-file.lisp" fail, as the absolute pathname construction uses the expressed type rather than overriding it with the component file type. which, upon reflection, really seems correct in any case. as is evident in the diff, the mechanism to manage the pathname component mapping requires about two dozen lines of code - the extra slot per component to hold the mapping, the component-translate- pathname operator, and the logical operator in bind-component- pathname. although the top-down apriori construction makes it easier to understand, that code is independent of the mapping. --- [1] : http://en.wikipedia.org/wiki/Sawzall ---
participants (3)
-
Faré -
james anderson -
Robert Goldman