Attached are patches in 'abcl-asdf.patch' to get asdf-1.641 to work against ABCL for the run-tests.sh.
Unfortunately, you will need to build [ABCL from trunk][1] using at least svn r12550, because I had to patch ABCL to work with ASDF. And you'll need to apply the 'abcl-translate-pathname.patch'.
[1]: svn://common-lisp.net/project/armedbear/svn/trunk/abcl
Notes:
1. The fugliness of the conditional around ASDF:GET-UID works around a bug in the ABCL compiler. We are in the process of [analyzing the error][2].
[2]: http://trac.common-lisp.net/armedbear/ticket/89
2. There is some undiagnosed problem in translating the binary location for the new ASDF2 "~/.cache" conventions, as ABCL seems to "collapses" everything into a single directory. Somehow, the default *output-translations* have a "/**/**/*.*" where there should be "/**/*.*". I'm not sure if this is a problem in "abcl-translate-pathname.patch", which is why I didn't apply this to ABCL trunk.
3. ASDF's run-tests.sh seems to ignore the "flags" setting, which seems to be broken on the ASDF side.
4. The changes for ASDF look quite interesting. However I would advocate that you should make ASDF2 work *exactly* like ASDF1 out of the box, meaning you shouldn't do any of the subsumed ASDF-BINARY-LOCATIONS stuff without being asked to by configuration.
How do you feel about integrating asdf-binary-locations into ABCL as well. I would make one change to what's there now, which is to add :java-1.4, :java-1.5, :java-1.6 to the list of architectures it has.
-Alan
On Tue, Mar 16, 2010 at 11:59 AM, Mark Evenson evenson@panix.com wrote:
Attached are patches in 'abcl-asdf.patch' to get asdf-1.641 to work against ABCL for the run-tests.sh.
Unfortunately, you will need to build [ABCL from trunk][1] using at least svn r12550, because I had to patch ABCL to work with ASDF. And you'll need to apply the 'abcl-translate-pathname.patch'.
Notes:
- The fugliness of the conditional around ASDF:GET-UID works around a bug
in the ABCL compiler. We are in the process of [analyzing the error][2].
- There is some undiagnosed problem in translating the binary location for
the new ASDF2 "~/.cache" conventions, as ABCL seems to "collapses" everything into a single directory. Somehow, the default *output-translations* have a "/**/**/*.*" where there should be "/**/*.*". I'm not sure if this is a problem in "abcl-translate-pathname.patch", which is why I didn't apply this to ABCL trunk.
- ASDF's run-tests.sh seems to ignore the "flags" setting, which seems to
be broken on the ASDF side.
- The changes for ASDF look quite interesting. However I would advocate
that you should make ASDF2 work *exactly* like ASDF1 out of the box, meaning you shouldn't do any of the subsumed ASDF-BINARY-LOCATIONS stuff without being asked to by configuration.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
Alan,
ASDF-Binary-Locations has been obsoleted and replaced by the more easily configurable ASDF-Output-Translations, which is now built into ASDF itself.
ASDF 1.660 also includes some compatibility mode with ABL, if for some reason you love the old ABL way.
I hope ABCL has fixed its merge-pathnames woes.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] My understanding of macros is expanding — Matthias Felleisen wrt the PLT Scheme module system
On 16 March 2010 12:58, Alan Ruttenberg alanruttenberg@gmail.com wrote:
How do you feel about integrating asdf-binary-locations into ABCL as well. I would make one change to what's there now, which is to add :java-1.4, :java-1.5, :java-1.6 to the list of architectures it has.
-Alan
On Tue, Mar 16, 2010 at 11:59 AM, Mark Evenson evenson@panix.com wrote:
Attached are patches in 'abcl-asdf.patch' to get asdf-1.641 to work against ABCL for the run-tests.sh.
Unfortunately, you will need to build [ABCL from trunk][1] using at least svn r12550, because I had to patch ABCL to work with ASDF. And you'll need to apply the 'abcl-translate-pathname.patch'.
Notes:
- The fugliness of the conditional around ASDF:GET-UID works around a bug
in the ABCL compiler. We are in the process of [analyzing the error][2].
- There is some undiagnosed problem in translating the binary location for
the new ASDF2 "~/.cache" conventions, as ABCL seems to "collapses" everything into a single directory. Somehow, the default *output-translations* have a "/**/**/*.*" where there should be "/**/*.*". I'm not sure if this is a problem in "abcl-translate-pathname.patch", which is why I didn't apply this to ABCL trunk.
- ASDF's run-tests.sh seems to ignore the "flags" setting, which seems to
be broken on the ASDF side.
- The changes for ASDF look quite interesting. However I would advocate
you shouldn't do any of the subsumed ASDF-BINARY-LOCATIONS stuff without being asked to by configuration.
On 3/23/10 4:53 PM, Faré wrote: […]
I hope ABCL has fixed its merge-pathnames woes.
[…]
As I reported last week, ASDF2 works with ABCL from trunk and the upcoming abcl-0.19 release, for which I have fixed a couple issues with our pathname support.
Are there current failures with ABCL's MERGE-PATHNAMES that you are referring to, or is this referring to the state of ABCL a couple weeks ago? If there are known problems, please help me identify them so we can fix them or at least record the information in our issue tracker.
Sorry, I haven't tried recompiling the latest ABCL from source and don't know if any pathname issue remains.
Have you tried downloading the latest ASDF and running its test suite? Do you pass all tests? I ought to include more tests about the output-translations configuration and the source-registry configuration...
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Majority, n.: That quality that distinguishes a crime from a law.
On 24 March 2010 04:14, Mark Evenson evenson@panix.com wrote:
On 3/23/10 4:53 PM, Faré wrote: […]
I hope ABCL has fixed its merge-pathnames woes.
[…]
As I reported last week, ASDF2 works with ABCL from trunk and the upcoming abcl-0.19 release, for which I have fixed a couple issues with our pathname support.
Are there current failures with ABCL's MERGE-PATHNAMES that you are referring to, or is this referring to the state of ABCL a couple weeks ago? If there are known problems, please help me identify them so we can fix them or at least record the information in our issue tracker.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 3/24/10 2:52 PM, Faré wrote:
Sorry, I haven't tried recompiling the latest ABCL from source and don't know if any pathname issue remains.
Have you tried downloading the latest ASDF and running its test suite? Do you pass all tests? I ought to include more tests about the output-translations configuration and the source-registry configuration...
Yes, and I just retried with ABCL trunk r12570 with ADSF 1.661 with no test failures.
Is ASDF2 what gets loaded when you require 'asdf in trunk, or some other action required to use it? -Alan
On Wed, Mar 24, 2010 at 7:21 AM, Mark Evenson evenson@panix.com wrote:
On 3/24/10 2:52 PM, Faré wrote:
Sorry, I haven't tried recompiling the latest ABCL from source and don't know if any pathname issue remains.
Have you tried downloading the latest ASDF and running its test suite? Do you pass all tests? I ought to include more tests about the output-translations configuration and the source-registry configuration...
Yes, and I just retried with ABCL trunk r12570 with ADSF 1.661 with no test failures.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Hi Alan,
On Wed, Mar 24, 2010 at 3:36 PM, Alan Ruttenberg alanruttenberg@gmail.com wrote:
Is ASDF2 what gets loaded when you require 'asdf in trunk, or some other action required to use it? -Alan
I think we will need to update our ASDF now that the new ASDF2 is available. What you get when you use REQUIRE is a very old version of ASDF1.
Bye,
Erik.
On 3/24/10 4:54 PM, Erik Huelsmann wrote:
Hi Alan,
On Wed, Mar 24, 2010 at 3:36 PM, Alan Ruttenberg alanruttenberg@gmail.com wrote:
Is ASDF2 what gets loaded when you require 'asdf in trunk, or some other action required to use it? -Alan
I think we will need to update our ASDF now that the new ASDF2 is available. What you get when you use REQUIRE is a very old version of ASDF1.
Right: we still include ASDF 1.3 with ABCL.
I'm looking at upgrading ABCL trunk to the new (aka ASDF2) version, but right now ASDF2 cannot load systems from jar files so I haven't put it on trunk just yet.
One can load ASDF2 from ASDF quite easily, for those wishing to experiment along with me.
On 24 March 2010 12:59, Mark Evenson evenson@panix.com wrote:
Right: we still include ASDF 1.3 with ABCL.
I'm looking at upgrading ABCL trunk to the new (aka ASDF2) version, but right now ASDF2 cannot load systems from jar files so I haven't put it on trunk just yet.
One can load ASDF2 from ASDF quite easily, for those wishing to experiment along with me.
Can ASDF1 do this loading from jar files? I certainly didn't remove anything from it. If you have a local patch to ASDF1 to do it, I'll be please to include it in either ASDF2 or a contrib to it.
If there are any regressions from ASDF1 to ASDF2, I want them fixed. Otherwise, there shouldn't be a problem upgrading ABCL from ASDF1 to ASDF2.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The secret of survival is: Always expect the unexpected. — Dr. Who
On 3/24/10 7:08 PM, Faré wrote: […]
Can ASDF1 do this loading from jar files? I certainly didn't remove anything from it. If you have a local patch to ASDF1 to do it, I'll be please to include it in either ASDF2 or a contrib to it.
Yes, ABCL ADSF1 can load from jar files and no, nothing has been removed from ASDF2 as it's more of a case of what has been added, namely that it now OPENs the .asd file instead of LOADing it, and ASDF-BINARY-LOCATIONS are baked-in.
That a Pathname for jar locations in ABCL are not currently capable of being OPENed turns out to be easily implemented (see attached patch).
That a jar Pathname is not a writable location turns out to cause problems for the binary locations part of ASDF2. It was sufficient to [patch the behavior][1] of ASDF1 to have OPERATION-DONE-P return T if all the members of the output files of COMPILE-OP on a CL-SOURCE-FILE were determined to be within a jar. But this turns out only to have worked with ABCL because there was a bug in how *LOAD-TRUENAME* was computed in loading from a jar Pathname that I would like to fix going forward (it's fixed in the patch).
[1]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/src/org/armedbear/l...
It would be perhaps cleaner to have the binary locations machinery of ASDF2 react to not being able to write to the Pathname derived from the location of the ".asd" file in an extensible manner. This might be useful for users of Lisps other than ABCL who don't have permission to write to the system ASDF location for instance. My current problem is that the :BEFORE for PERFORM specialized on COMPILE-OP SOURCE-FILE contains an ENSURE-DIRECTORIES-EXIST which by default is derived from the ".asd" Pathname. Is there a way to per-system customization of the output location?
Thinking quickly for an ASDF2 algorithm:
1. Is the directory containing the ".asd" writable? If so, continue normally.
2. Otherwise, can we establish an alternative location to write the output of COMPILE-OP as configured by the user? If so, continue normally.
3. Otherwise, skip the COMPILE-OP of this system. For LOAD-OP, search for FASLs by some user-extensible mechanism, falling back to looking in the same directory as the ".lisp" files. If no FASLs can be found, load the source files.
If there are any regressions from ASDF1 to ASDF2, I want them fixed. Otherwise, there shouldn't be a problem upgrading ABCL from ASDF1 to ASDF2.
No regressions have been noted; I will report them if found. As it stands, I can't upgrade ABCL to ASDF2 without breaking the existing functionality of loading ADSF definitions from jars. The responsibility for fixing this situation certainly lies with ABCL's extension of Pathnames to cover jar files, which we are working to bring in line with new features offered by ASDF2.
Dear Mark,
Yes, ABCL ADSF1 can load from jar files and no, nothing has been removed from ASDF2 as it's more of a case of what has been added, namely that it now OPENs the .asd file instead of LOADing it, and ASDF-BINARY-LOCATIONS are baked-in.
OK.
That a Pathname for jar locations in ABCL are not currently capable of being OPENed turns out to be easily implemented (see attached patch).
OK.
That a jar Pathname is not a writable location turns out to cause problems for the binary locations part of ASDF2. It was sufficient to [patch the behavior][1] of ASDF1 to have OPERATION-DONE-P return T if all the members of the output files of COMPILE-OP on a CL-SOURCE-FILE were determined to be within a jar. But this turns out only to have worked with ABCL because there was a bug in how *LOAD-TRUENAME* was computed in loading from a jar Pathname that I would like to fix going forward (it's fixed in the patch).
OK. But doesn't asdf-output-translations (the replacement for asdf-binary-locations) by default redirect the output to a user cache? Or did you disable that?
Do you want me to merge that into upstream ASDF with #+abcl conditions?
Note: If ABCL supports matching and translating such paths, there could be ASDF-Output-Translation rules by default with ABCL in the idea of (#p"/**/*.jar/**/*.*" (:home #p"**/*.jar/**/*.*")
It would be perhaps cleaner to have the binary locations machinery of ASDF2 react to not being able to write to the Pathname derived from the location of the ".asd" file in an extensible manner. This might be useful for users of Lisps other than ABCL who don't have permission to write to the system ASDF location for instance. My current problem is that the :BEFORE for PERFORM specialized on COMPILE-OP SOURCE-FILE contains an ENSURE-DIRECTORIES-EXIST which by default is derived from the ".asd" Pathname. Is there a way to per-system customization of the output location?
How do you propose to do that?
Thinking quickly for an ASDF2 algorithm:
- Is the directory containing the ".asd" writable? If so, continue
normally.
- Otherwise, can we establish an alternative location to write the output
of COMPILE-OP as configured by the user? If so, continue normally.
- Otherwise, skip the COMPILE-OP of this system. For LOAD-OP, search for
FASLs by some user-extensible mechanism, falling back to looking in the same directory as the ".lisp" files. If no FASLs can be found, load the source files.
Juanjo of ECL was thinking of a different delivery mechanism, where compiling a ASDF system would yield another ASDF system of the same interchangeable name, but the content of which would be just one or several binary objects to load.
Perhaps that's what you really want to be using?
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The Philosophy of Liberty, or Libertarianism, is a theory of Law; it is an ethics of Liberty and Responsibility; it is a cybernetics of Human Action; it is the only authentically subversive ideology.
On 3/25/10 7:34 AM, Faré wrote: […]
Do you want me to merge that into upstream ASDF with #+abcl conditions?
Not yet: it doesn't make sense until I figure out what works with ASDF2 (that patch only works with ASDF 1.3).
Note: If ABCL supports matching and translating such paths, there could be ASDF-Output-Translation rules by default with ABCL in the idea of (#p"/**/*.jar/**/*.*" (:home #p"**/*.jar/**/*.*")
An avenue that I will explore. Thanks.
It would be perhaps cleaner to have the binary locations machinery of ASDF2 react to not being able to write to the Pathname derived from the location of the ".asd" file in an extensible manner. This might be useful for users of Lisps other than ABCL who don't have permission to write to the system ASDF location for instance. My current problem is that the :BEFORE for PERFORM specialized on COMPILE-OP SOURCE-FILE contains an ENSURE-DIRECTORIES-EXIST which by default is derived from the ".asd" Pathname. Is there a way to per-system customization of the output location?
How do you propose to do that?
Not sure, which is why I was floating the idea for comments, but if I can run a user-defined function after the .asd has been parsed to programatically implement the following, that might work. I need to read up on what is present in the currently defined mechanism.
Thinking quickly for an ASDF2 algorithm:
- Is the directory containing the ".asd" writable? If so, continue
normally.
[…]
Juanjo of ECL was thinking of a different delivery mechanism, where compiling a ASDF system would yield another ASDF system of the same interchangeable name, but the content of which would be just one or several binary objects to load.
Perhaps that's what you really want to be using?
That's worth investigating as well.
Thanks for the input.
On 3/25/10 Mar 25 -1:49 AM, Mark Evenson wrote:
On 3/25/10 7:34 AM, Faré wrote: […]
....
It would be perhaps cleaner to have the binary locations machinery of ASDF2 react to not being able to write to the Pathname derived from the location of the ".asd" file in an extensible manner. This might be useful for users of Lisps other than ABCL who don't have permission to write to the system ASDF location for instance. My current problem is that the :BEFORE for PERFORM specialized on COMPILE-OP SOURCE-FILE contains an ENSURE-DIRECTORIES-EXIST which by default is derived from the ".asd" Pathname. Is there a way to per-system customization of the output location?
How do you propose to do that?
Not sure, which is why I was floating the idea for comments, but if I can run a user-defined function after the .asd has been parsed to programatically implement the following, that might work. I need to read up on what is present in the currently defined mechanism.
Faré would it be possible/appropriate to take the core of find-system which does the work of loading the system (opening the file, interpreting it, etc.), and package it up as a generic function so that it would be possible for customizers to add :before, :after, and :around methods? Or, alternatively, for "in-house" ASDF extensions like Juanjo's, or a proposed ABCL jar extension, to provide additional "wrapper" methods that would leave :before, :after, and :around free for "outsiders"?
Just a casual thought,
r
[Removing Erik and Alan from the CC who should see it on the armedbear list if they're interested/
On Mar 25, 2010, at 2:00 PM, Robert Goldman wrote:
On 3/25/10 Mar 25 -1:49 AM, Mark Evenson wrote:
On 3/25/10 7:34 AM, Faré wrote: […]
....
It would be perhaps cleaner to have the binary locations machinery of ASDF2 react to not being able to write to the Pathname derived from the location of the ".asd" file in an extensible manner. This might be useful for users of Lisps other than ABCL who don't have permission to write to the system ASDF location for instance. My current problem is that the :BEFORE for PERFORM specialized on COMPILE-OP SOURCE-FILE contains an ENSURE-DIRECTORIES-EXIST which by default is derived from the ".asd" Pathname. Is there a way to per-system customization of the output location?
How do you propose to do that?
Not sure, which is why I was floating the idea for comments, but if I can run a user-defined function after the .asd has been parsed to programatically implement the following, that might work. I need to read up on what is present in the currently defined mechanism.
Faré would it be possible/appropriate to take the core of find-system which does the work of loading the system (opening the file, interpreting it, etc.), and package it up as a generic function so that it would be possible for customizers to add :before, :after, and :around methods? Or, alternatively, for "in-house" ASDF extensions like Juanjo's, or a proposed ABCL jar extension, to provide additional "wrapper" methods that would leave :before, :after, and :around free for "outsiders"?
I've now spent some time understanding the output translations method in ASDF2, and think ABCL would be best served if we could add the ability to specify an arbitrary function as an output translation "destination". The extensions that Robert suggests would be interesting, but I don't have a current use case for them in application to ABCL.
Fare's suggestion that I use an output translation based on the jar pathname doesn't quite work, because in our current implementation, the pathname of the jar is stored in DEVICE, separate from the rest of the jar pathname. I extended PATHNAME-MATCH-P to match jars correctly, but I don't see a possible extension of TRANSLATE-PATHNAME. For purposes of discussion consider that a jar pathname has a namestring composed of "jar:file:DEVICE!/DIRECTORY/NAME.TYPE". This isn't quite the whole story: [See the design document for th gory details if interested][1]. Having the path of the jar file in DEVICE means that I cannot use TRANSLATE-PATHNAME to incorporate the path of the jar in the output translation to a plain file. The best I can do is
(initialize-output-translations '(:output-translations :ignore-inherited-configuration (#p"jar:file:/**/*.jar!/**/*.*" (:home #p"jar-files/**/*.*"))))
which "squashes" all of the output files under one directory irregardless of jar they are specified in , meaning that both "jar:file:/some/thing.jar!/package.lisp" and "jar:file:/a/completely/different.jar!/package.lisp" would have the same output translation.
[1]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/doc/design/pathname...
So I would ask the ASDF developers to consider extending the output translation DSL to allow something like
(initialize-output-translations '(:output-translations :ignore-inherited-configuration (#p"jar:file:/**/*.jar!/**/*.*" :function SYS::ASDF-JAR-OUTPUT-TRANSLATE)))
or
(initialize-output-translations '(:output-translations :ignore-inherited-configuration (#p"jar:file:/**/*.jar!/**/*.*" :function (lambda (wildstring source) (pathname source)))))
Where the argument following :FUNCTION would either name a symbol for a function or a lambda expression that specifies a function that takes two arguments, the WILDCARD matched and the pathname of the SOURCE that matched. This function would return a pathname suitable for the output location.
I'm working on an implementation of this on your git version as a proof of concept, but haven't gotten much further than getting your output configuration to accept the new syntax. I assume that actually applying the function will be easy, but I am not sure.
And, is there a syntax to specify that a pathname matching a given pattern should have no output location, but should just load the source?
--
"A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 3/25/10 2:39 PM, Mark Evenson wrote: […]
I'm working on an implementation of this on your git version as a proof of concept, but haven't gotten much further than getting your output configuration to accept the new syntax. I assume that actually applying the function will be easy, but I am not sure.
[…]
Attached please find a patch to implement the use of an arbitrary output translation function in ASDF2.
An example of the use would be the directive
(#p"jar:file:/**/*.jar!/**/*.*" :function translate-jar-pathname)
which will call the function TRANSLATE-JAR-PATHNAME on a match of the first element of the list. The function will be invoked with two arguments, the first being the matching wildcard, and the second being the pathname to be translated. The function returns the translation.
Hopefully, after review and critique, we can can get something like this in ASDF2.
With the attached patch to abcl trunk, I can successfully use ASDF2 to load ASDF systems in jar files, with the compiled files being output to a translation under :USER-CACHE.
Sorry, haven't had a chance to look closely at asdf2, but wanted to point out that in certain contexts, like running applets, writing to any file system is prohibited. For those cases, it would be nice if it was super-easy to set a flag that says: if there isn't a compiled version, or if it is stale, don't try to compile - just load source.
-Alan
On Thu, Mar 25, 2010 at 8:38 AM, Mark Evenson evenson@panix.com wrote:
On 3/25/10 2:39 PM, Mark Evenson wrote: […]
I'm working on an implementation of this on your git version as a proof of concept, but haven't gotten much further than getting your output configuration to accept the new syntax. I assume that actually applying the function will be easy, but I am not sure.
[…]
Attached please find a patch to implement the use of an arbitrary output translation function in ASDF2.
An example of the use would be the directive
(#p"jar:file:/**/*.jar!/**/*.*" :function translate-jar-pathname)
which will call the function TRANSLATE-JAR-PATHNAME on a match of the first element of the list. The function will be invoked with two arguments, the first being the matching wildcard, and the second being the pathname to be translated. The function returns the translation.
Hopefully, after review and critique, we can can get something like this in ASDF2.
With the attached patch to abcl trunk, I can successfully use ASDF2 to load ASDF systems in jar files, with the compiled files being output to a translation under :USER-CACHE.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
On Thu, 2010-03-25 at 09:17 -0700, Alan Ruttenberg wrote:
Sorry, haven't had a chance to look closely at asdf2, but wanted to point out that in certain contexts, like running applets, writing to any file system is prohibited. For those cases, it would be nice if it was super-easy to set a flag that says: if there isn't a compiled version, or if it is stale, don't try to compile - just load source.
That is already possible: just use LOAD-SOURCE-OP instead of LOAD-OP.
Right: I'll make sure that such an ability exists when we finally include ASDF2 in ABCL.
Tersely written from my iPod
On Mar 25, 2010, at 5:17 PM, Alan Ruttenberg alanruttenberg@gmail.com wrote:
Sorry, haven't had a chance to look closely at asdf2, but wanted to point out that in certain contexts, like running applets, writing to any file system is prohibited. For those cases, it would be nice if it was super-easy to set a flag that says: if there isn't a compiled version, or if it is stale, don't try to compile - just load source.
-Alan
On Thu, Mar 25, 2010 at 8:38 AM, Mark Evenson evenson@panix.com wrote:
On 3/25/10 2:39 PM, Mark Evenson wrote: […]
I'm working on an implementation of this on your git version as a proof of concept, but haven't gotten much further than getting your output configuration to accept the new syntax. I assume that actually applying the function will be easy, but I am not sure.
[…]
Attached please find a patch to implement the use of an arbitrary output translation function in ASDF2.
An example of the use would be the directive
(#p"jar:file:/**/*.jar!/**/*.*" :function translate-jar-pathname)
which will call the function TRANSLATE-JAR-PATHNAME on a match of the first element of the list. The function will be invoked with two arguments, the first being the matching wildcard, and the second being the pathname to be translated. The function returns the translation.
Hopefully, after review and critique, we can can get something like this in ASDF2.
With the attached patch to abcl trunk, I can successfully use ASDF2 to load ASDF systems in jar files, with the compiled files being output to a translation under :USER-CACHE.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
On 3/25/10 Mar 25 -11:17 AM, Alan Ruttenberg wrote:
Sorry, haven't had a chance to look closely at asdf2, but wanted to point out that in certain contexts, like running applets, writing to any file system is prohibited. For those cases, it would be nice if it was super-easy to set a flag that says: if there isn't a compiled version, or if it is stale, don't try to compile - just load source.
I may be seeing the same demon everywhere, but I /think/ I see what we would like to be able to do, and why that's very difficult in the existing ASDF.
* What we would like to be able to do is to check and see if the file system is writeable and if it's unwriteable, we should treat the compile operation on the system as having been done.
So it might /seem/ that the following pseudo-cod would work
(defmethod operation-done-p ((op compile-op) (sys jar-system)) (or (filesystem-not-writeable-p sys) (call-next-method)))
... but it won't because
* operation-done-p on a system does /not/ mean "this system has had the operation done on it," but only "any additional compile-op for this system AFTER ITS COMPONENTS HAVE UNDERGONE THE COMPILE-OP are done."
See my earlier email postings on the problems with the ASDF object model and, in particular, the dual meaning of the SYSTEM class object. This was discussed in great length at the time of my rewrite of the TRAVERSE function.
best, r
Dear Mark,
Fare's suggestion that I use an output translation based on the jar pathname doesn't quite work, because in our current implementation, the pathname of the jar is stored in DEVICE, separate from the rest of the jar pathname. I extended PATHNAME-MATCH-P to match jars correctly, but I don't see a possible extension of TRANSLATE-PATHNAME.
Couldn't you extend the ABCL pathname matching algorithm to allow for wildcards and all in the device component?
So I would ask the ASDF developers to consider extending the output translation DSL to allow something like
(initialize-output-translations '(:output-translations :ignore-inherited-configuration (#p"jar:file:/**/*.jar!/**/*.*" :function SYS::ASDF-JAR-OUTPUT-TRANSLATE)))
I suggest that either you make (:function foo) a valid second value for an output-translations list, or you make said list accept a &key function argument in addition to its two current positional arguments. But having weird non-standard vararg convention for such lists is probably a bad API. The default function would be TRANSLATE-PATHNAME and the calling convention would be the same.
And, is there a syntax to specify that a pathname matching a given pattern should have no output location, but should just load the source?
You mean, something to force LOAD-SOURCE-OP instead of LOAD-OP? Nope. But I suppose some component class could be defined to do this.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The Keyboard is mightier than the Death Ray...
On 3/29/10 8:25 AM, Faré wrote:
Dear Mark,
Fare's suggestion that I use an output translation based on the jar pathname doesn't quite work, because in our current implementation, the pathname of the jar is stored in DEVICE, separate from the rest of the jar pathname. I extended PATHNAME-MATCH-P to match jars correctly, but I don't see a possible extension of TRANSLATE-PATHNAME.
Couldn't you extend the ABCL pathname matching algorithm to allow for wildcards and all in the device component?
Wildcards do work in DEVICE but you can't currently use unused results of the wildcard matches on the DEVICE components to "carry over" to the translation of the DIRECTORY component. I suppose that's what you were suggesting anyways. There isn't an obviously easy way to implement this in ABCL due to the structure of our code and I need to think through the various permutations, but that doesn't mean that its implementation isn't a win for the end user. I'll see what I can do.
So I would ask the ASDF developers to consider extending the output translation DSL to allow something like
(initialize-output-translations '(:output-translations :ignore-inherited-configuration (#p"jar:file:/**/*.jar!/**/*.*" :function SYS::ASDF-JAR-OUTPUT-TRANSLATE)))
I suggest that either you make (:function foo) a valid second value for an output-translations list, or you make said list accept a&key function argument in addition to its two current positional arguments. But having weird non-standard vararg convention for such lists is probably a bad API. The default function would be TRANSLATE-PATHNAME and the calling convention would be the same.
I'd favor the first proposal, as making :FUNCTION a real key argument would imply that the second positional parameter is optional, whereas it is actually being replaced by the (:FUNCTION ARG) parameter. When I get time to revisit the patch, I'll implement your first proposal.
[…]
Thanks, Mark
On 29 March 2010 04:33, Mark Evenson evenson@panix.com wrote:
Couldn't you extend the ABCL pathname matching algorithm to allow for wildcards and all in the device component?
Wildcards do work in DEVICE but you can't currently use unused results of the wildcard matches on the DEVICE components to "carry over" to the translation of the DIRECTORY component. I suppose that's what you were suggesting anyways. There isn't an obviously easy way to implement this in ABCL due to the structure of our code and I need to think through the various permutations, but that doesn't mean that its implementation isn't a win for the end user. I'll see what I can do.
If you're going to use a lot of such magic DEVICE components, that could be a win.
So I would ask the ASDF developers to consider extending the output translation DSL to allow something like
(initialize-output-translations '(:output-translations :ignore-inherited-configuration (#p"jar:file:/**/*.jar!/**/*.*" :function SYS::ASDF-JAR-OUTPUT-TRANSLATE)))
I suggest that either you make (:function foo) a valid second value for an output-translations list, or you make said list accept a&key function argument in addition to its two current positional arguments. But having weird non-standard vararg convention for such lists is probably a bad API. The default function would be TRANSLATE-PATHNAME and the calling convention would be the same.
I'd favor the first proposal, as making :FUNCTION a real key argument would imply that the second positional parameter is optional, whereas it is actually being replaced by the (:FUNCTION ARG) parameter. When I get time to revisit the patch, I'll implement your first proposal.
I'd favor either a keyword argument or :function as a magic marker to be inserted in FIRST position, followed by a function designator (and additionally, parameters to curry?). Unless of course, you symmetrically accept a FUNCTION in first position to denote a "recognize if this matching rule applies".
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Boxer’s law of economics: “The economy interprets taxation and regulation as damage and routes around it.” — Jessica Boxer, http://esr.ibiblio.org/?p=1752
On Mar 29, 2010, at 10:39 AM, Faré wrote: […]
So I would ask the ASDF developers to consider extending the output translation DSL to allow something like
(initialize-output-translations '(:output-translations :ignore-inherited-configuration (#p"jar:file:/**/*.jar!/**/*.*" :function SYS::ASDF-JAR-OUTPUT-TRANSLATE)))
I suggest that either you make (:function foo) a valid second value for an output-translations list, or you make said list accept a&key function argument in addition to its two current positional arguments. But having weird non-standard vararg convention for such lists is probably a bad API. The default function would be TRANSLATE-PATHNAME and the calling convention would be the same.
I'd favor the first proposal, as making :FUNCTION a real key argument would imply that the second positional parameter is optional, whereas it is actually being replaced by the (:FUNCTION ARG) parameter. When I get time to revisit the patch, I'll implement your first proposal.
I'd favor either a keyword argument or :function as a magic marker to be inserted in FIRST position, followed by a function designator (and additionally, parameters to curry?). Unless of course, you symmetrically accept a FUNCTION in first position to denote a "recognize if this matching rule applies".
I would think that we want to preserve the current ability for the user to vistually scan down the list of translations to determine what should match, so I would favor going for your second proposal. It doesn't make sense that there be a match predicate paired with a translation pathname right? So the two new possibilities for output translation lists would be:
("/**/foo/*.*" (:function FOO-OUTPUT-TRANSLATION)) ((:function BAR-PATHNAME-P) (:function BAR-OUTPUT-TRANSLATION))
--
"A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 3/29/10 11:33 AM, Mark Evenson wrote: […]
I'd favor either a keyword argument or :function as a magic marker to be inserted in FIRST position, followed by a function designator (and additionally, parameters to curry?). Unless of course, you symmetrically accept a FUNCTION in first position to denote a "recognize if this matching rule applies".
I would think that we want to preserve the current ability for the user to vistually scan down the list of translations to determine what should match, so I would favor going for your second proposal. It doesn't make sense that there be a match predicate paired with a translation pathname right? So the two new possibilities for output translation lists would be:
("/**/foo/*.*" (:function FOO-OUTPUT-TRANSLATION)) ((:function BAR-PATHNAME-P) (:function BAR-OUTPUT-TRANSLATION))
Attached is a patch against ASDF that allows a user specified function to take the place of TRANSLATE-PATHNAME as the ASDF2 output translation function.
Additionally it adds an ABCL specific extension to ASDF2 that uses this ability by providing the following translations by default
(#p"jar:file:/**/*.jar!/**/*.*" (:function translate-jar-pathname)) (#p"/:jar:file/**/*.*" (:user-cache #p"**/*.*")))
that will map all ASDF loads from jar pathnames to an output location under the per user cache location. This will allow the loading of ASDF systems packaged in jar files which first compiles and then loads FASLs to the local filesystem.
To run this with ABCL you will need to build a custom build of ABCL with the attached patch which is currently necessary to OPEN a jar pathname. This patch is part of a large set of modifications that allow ABCL to open a pathname specified as a url, that I am developing and testing as a unit. These modiictions will be placed in ABCL trunk soon, but first I need to add create more unit tests for the introduced functionality and implement a few more suggestions (like URI encoding functions).
I did not implement Faré's suggestion of including a syntax for a predicate function as the first argument for a output translation pair, because I couldn't come up with a use case in which the current default function PATHNAME-MATCH-P would not work. If someone can come up with such a case, or this deficiency prevents ASDF from accepting this patch, I can have a go at it again.
Thanks for applying my patches to ASDF, as it now appears that ABCL trunk (svn r12551 and later) runs pretty well with ASDF-1.643. The previous problem I noted with the binary locations getting "collapsed" was our problem which I fixed in ABCL.
Attached is a (trivial) patch so that 'run-tests.sh' actually uses the flags argument.
ABCL now passes the ASDF test suite:
-#--------------------------------------- Using abcl --noinit Ran 21 tests: 21 passing and 0 failing all tests apparently successful -#---------------------------------------
After discussion on IRC and further thought, I no longer advocate that ASDF2 not make the relocating the FASLs the default. There are certainly decent arguments for it, and we do want progress in ASDF2, right? Instead, I would advise that you make a section in the (really excellent looking) manual that explicates the changes that an ASDF1 user—both with and without using ASDF-BINARY-LOCATIONS—should expect in ASDF2. And I would include the *LOAD-TRUENAME*/*LOAD-PATHNAME* issue (i.e. use a conditionalized ASDF:SYSTEM-DEFINITION-PATHNAME invocation) in that section.
Otherwise, ASDF2 looks quite cool. What's your timeframe for an official release? I'd include it as-is in ABCL, but don't want to chase all the release candidates to ASDF2.
Attached is a (trivial) patch so that 'run-tests.sh' actually uses the flags argument.
Thanks. It was applied, and many small bugs were fixed in ASDF 1.647 as I did more testing and improved the test infrastructure.
ABCL now passes the ASDF test suite:
-#--------------------------------------- Using abcl --noinit Ran 21 tests: 21 passing and 0 failing all tests apparently successful -#---------------------------------------
Yay!
After discussion on IRC and further thought, I no longer advocate that ASDF2 not make the relocating the FASLs the default. There are certainly decent arguments for it, and we do want progress in ASDF2, right? Instead, I would advise that you make a section in the (really excellent looking) manual that explicates the changes that an ASDF1 user—both with and without using ASDF-BINARY-LOCATIONS—should expect in ASDF2. And I would include the *LOAD-TRUENAME*/*LOAD-PATHNAME* issue (i.e. use a conditionalized ASDF:SYSTEM-DEFINITION-PATHNAME invocation) in that section.
Yes. We need a FAQ section for migration from ASDF1 to ASDF2, with questions each in its own subsection, and a rationale for each change. Sigh.
Otherwise, ASDF2 looks quite cool. What's your timeframe for an official release? I'd include it as-is in ABCL, but don't want to chase all the release candidates to ASDF2.
Hopefully we'd like to release ASDF2 before summer. I suggest that at some point (e.g. now) you pick one of the release candidates that you consider not too broken after testing, then upgrade to the real ASDF2 release when it's out, and otherwise only upgrade when there's a good reason to (i.e. fixing major brokenness, or adding major feature, or there's been a stable version for 6 months that you haven't upgraded to yet).
Thanks a lot for your support!
And don't forget to implement the long DEFINE-METHOD-COMBINATION :)
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] What a lot of trouble to prove in political economy that two and two make four; and if you succeed in doing so, people cry, 'It is so clear that it is boring.' Then they vote as if you had never proved anything at all. — Frederic Bastiat, "What Is Seen and What is Not Seen", 1850
-
Alan Ruttenberg -
Erik Huelsmann -
Faré -
Mark Evenson -
Robert Goldman -
Stelian Ionescu