Plans for ASDF 3.3
Dear Robert, dear Lispers, I'd like to know what are the release plans will be for 3.3. My branch for proper phase separation is basically ready to be merged, but I'd like to see how compatible it is with next month's Quicklisp before I can recommend releasing it upon the masses. Note that the branch is called "plan" because it started as a refactoring how asdf builds its plan, but the changes run deeper than that — 970 lines were added or modified all over, not counting those moved around — that's double the number of lines of the original ASDF. Additional questions: should the syntax-control branch be worked on for 3.3.0? 3.2.2? 3.3.1? 3.4.0? It's a low-hanging fruit to making ASDF a more robust build tool that supports working with non-trivial syntax extension. PS: Are any (preferrably younger) lispers interested in becoming proficient at ASDF maintenance? There are plenty of TODO items of all sizes and all difficulties that could use some love. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "I'm going to live forever, or die trying!" — Spider Robinson
Dear Robert, dear Lispers,
I'd like to know what are the release plans will be for 3.3.
My branch for proper phase separation is basically ready to be merged, but I'd like to see how compatible it is with next month's Quicklisp before I can recommend releasing it upon the masses.
Note that the branch is called "plan" because it started as a refactoring how asdf builds its plan, but the changes run deeper than that — 970 lines were added or modified all over, not counting those moved around — that's double the number of lines of the original ASDF.
Additional questions: should the syntax-control branch be worked on for 3.3.0? 3.2.2? 3.3.1? 3.4.0? It's a low-hanging fruit to making ASDF a more robust build tool that supports working with non-trivial syntax extension.
PS: Are any (preferrably younger) lispers interested in becoming proficient at ASDF maintenance? There are plenty of TODO items of all sizes and all difficulties that could use some love.
Hello Fare, I think your message is missing a bit of context: * what's wrong with the previous way of performing phase separation ? Why does the new branch make it proper ? What are the day-to-day consequences for ASDF users ? * what does the syntax-control branch do ? -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On Tue, Apr 25, 2017 at 11:33 PM, Stelian Ionescu <sionescu@cddr.org> wrote:
Dear Robert, dear Lispers,
I'd like to know what are the release plans will be for 3.3.
My branch for proper phase separation is basically ready to be merged, but I'd like to see how compatible it is with next month's Quicklisp before I can recommend releasing it upon the masses.
Note that the branch is called "plan" because it started as a refactoring how asdf builds its plan, but the changes run deeper than that — 970 lines were added or modified all over, not counting those moved around — that's double the number of lines of the original ASDF.
Additional questions: should the syntax-control branch be worked on for 3.3.0? 3.2.2? 3.3.1? 3.4.0? It's a low-hanging fruit to making ASDF a more robust build tool that supports working with non-trivial syntax extension.
PS: Are any (preferrably younger) lispers interested in becoming proficient at ASDF maintenance? There are plenty of TODO items of all sizes and all difficulties that could use some love.
Hello Fare,
I think your message is missing a bit of context:
* what's wrong with the previous way of performing phase separation ? Why does the new branch make it proper ? What are the day-to-day consequences for ASDF users ? * what does the syntax-control branch do ?
OK, here's some context. Phase Separation: A basic design idea in ASDF 1.0 to 3.2 is that you first plan your entire build, then you perform the plan. The plan is a list of actions (pair of OPERATION and COMPONENT), obtained by walking the action dependency graph implicitly defined by the COMPONENT-DEPENDS-ON methods. Performing the plan is achieved by calling PERFORM gf on each action, which in turn will call INPUT-FILES and OUTPUT-FILES to locate the inputs and outputs. This works perfectly fine as long as you don't need ASDF extensions (such as, e.g. cffi-grovel, or f2l). Now, if you do have an extension, how do you load it? Well, it's written in Lisp, so you use a Lisp build system for that, i.e. ASDF! And so people either use load-system (or older equivalent) from their .asd files, or more declaratively use :defsystem-depends-on in their (defsystem ...) form. Now, since ASDF up until 3.2 has no notion of multiple phases, what happens is that a brand new plan is created then performed every time you use this feature. This kind of works well in simple cases, when you load well-behaved systems from scratch: some actions may be planned then performed in multiple phases, but performing should be idempotent (or else you deserve to lose), therefore ASDF wastes some time rebuilding a few actions that were planned before an extension was loaded that also depended on them. However, the real problems arise when something causes an extension to be invalidated: then the behavior of the extension may change (even subtly) due its modified dependency, and the extension and all the systems that directly or indirectly depend on must be invalidated and recomputed. But ASDF up until 3.2 fail to do so, and the resulting build can thus be incorrect. The bug is quite subtle: to experience it, you must be attempting an incremental build, while meaningful changes were made that affect the behavior of an ASDF extension. This kind of situation is rare enough in the small. And it is easily remedied by manually building from scratch. In the small, you can afford to always build from scratch the few systems that you modify, anyway. But when programming in the large, the bug may become very serious. What more, it was a hurdle on the road to making a future ASDF a robust system with deterministic builds. Syntax Control: The current ASDF has no notion of syntax, and uses whatever *readtable*, *print-pprint-dispatch* or *read-default-float-format* are ambient at the time ASDF is called. This means that if you ever side-effect those variables and/or the underlying tables (e.g. to enable fare-quasiquote for the sake of matching with optima or trivia), then call ASDF, the code will be compiled with those modified tables, which will make fasl that are unloadable unless the same side-effects are present. If systems are modified and compiled that do not have explicit dependencies on those side-effects, or worse, that those side-effects depend on (e.g. fare-utils, that fare-quasiquote depends on), then your fasl cache will be polluted and the only way out will be to rm -rf the contaminated parts of the fasl cache and/or to build with :force :all until all parts are overwritten. Which is surprising and painful. In practice, this means that using ASDF is not compatible with making non-additive modifications to the syntax. Back in the 3.1 days, I wrote a branch whereby each system has its own bindings for the syntax variables. But this is not compatible with a few legacy systems that explicitly depend on modifying the syntax for the next system to use, which some do as ugly as that is, so the branch was never merged. A more moderate change to ASDF would be to have global settings for the variables that are bound around any ASDF session, and trust that at least for THOSE values of *readtable* and *print-pprint-dispatch*, users never do non-additive changes then call ASDF again (which they probably don't, or they would already be experiencing lots of trouble). Then, if you bind *readtable* to a different value, e.g. using named-readtables:in-readtable, then you can freely make non-additive changes (such as enable fare-quasiquote) and it won't adversely affect the ASDF build. The problem with not having any syntax-control in ASDF is that it forces Lispers to always be conservative about modifying the readtable and calling ASDF (or having it called indirectly by any function whatsoever that they call), which in practice makes hacking Lisp code hostile to interactive development with non-additive syntax modification. If syntax-control is added to ASDF, then you can freely do your syntax modifications and be confident that building code won't be adversely affected. There again, this modification is a low-hanging fruit in making ASDF a more robust system. Conclusion: Both these changes are (retrospectively) easy ways to make ASDF more robust. Yet in both cases they depend on users to not go wild with side-effects, in a way that essentially cannot be enforced. CL is this kind of hippie language that disallows disallowing. And so both issues illustrate how the "everything is global side-effects" model of CL is ultimately a huge impediment to building code in a robust deterministic way.
On 4/26/17 Apr 26 -6:33 AM, Faré wrote:
What more, it was a hurdle on the road to making a future ASDF a robust system with deterministic builds.
I wanted to ask about this. Right now ASDF systems are (in general) partially ordered. Is it your plan to make sure there are no implementation or state dependencies in how these partial orders are linearized, in order to make it deterministic? Also, I know you are interested in parallel variants of ASDF. It seems like this would conflict with the desire to enforce deterministic orderings, since if you have A must be before B, C, and D, but B, C, and D are unordered in the system definition, determinism would lead you to want to have a deterministic algorithm for ordering B, C, and D, but parallelism would suggest you separately compile B, C, and D in different images (each of which would have A loaded). Best, r
Also, I know you are interested in parallel variants of ASDF. It seems like this would conflict with the desire to enforce deterministic orderings, since if you have A must be before B, C, and D, but B, C, and D are unordered in the system definition, determinism would lead you to want to have a deterministic algorithm for ordering B, C, and D, but parallelism would suggest you separately compile B, C, and D in different images (each of which would have A loaded).
while we are at it, i would welcome a change/feature where anything unspecified would be randomly ordered. so that issues come up while developing, not when the live system has a different filesystem order and boom, things just don't build... -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “Political speech and writing are largely the defense of the indefensible.” — George Orwell (1903–1950), 'Politics and the English Language' (1946)
On Wed, Apr 26, 2017 at 7:17 PM, Robert Goldman <rpgoldman@sift.net> wrote:
On 4/26/17 Apr 26 -6:33 AM, Faré wrote:
What more, it was a hurdle on the road to making a future ASDF a robust system with deterministic builds.
I wanted to ask about this. Right now ASDF systems are (in general) partially ordered. Is it your plan to make sure there are no implementation or state dependencies in how these partial orders are linearized, in order to make it deterministic?
Well, a really deterministic variant of ASDF, if it ever happens, would be based on cross-compilation in separate processes, similar to how Lisp builds currently happen with Bazel. Handling self-extension and phase separation properly in this context might be "interesting".
Also, I know you are interested in parallel variants of ASDF. It seems like this would conflict with the desire to enforce deterministic orderings, since if you have A must be before B, C, and D, but B, C, and D are unordered in the system definition, determinism would lead you to want to have a deterministic algorithm for ordering B, C, and D, but parallelism would suggest you separately compile B, C, and D in different images (each of which would have A loaded).
Indeed, different images would be used for compiling. Loading would happen in the textual order of the target at hand, and/or in some arbitrary global order (e.g. asciibetical) for systems. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Sheep walk to the slaughterhouse. They don't say anything, and they don't have any hope. But at least they don't vote for the butcher who will kill them and for the rich man who will eat them. Dumber than dumb beasts, more sheepish than sheep, the voter appoints his butcher and chooses his rich man. He made Revolutions to conquer this right. — Octave Mirbeau
On 4/26/17 Apr 26 -7:46 PM, Faré wrote:
On Wed, Apr 26, 2017 at 7:17 PM, Robert Goldman <rpgoldman@sift.net> wrote:
What more, it was a hurdle on the road to making a future ASDF a robust system with deterministic builds. I wanted to ask about this. Right now ASDF systems are (in general)
On 4/26/17 Apr 26 -6:33 AM, Faré wrote: partially ordered. Is it your plan to make sure there are no implementation or state dependencies in how these partial orders are linearized, in order to make it deterministic?
Well, a really deterministic variant of ASDF, if it ever happens, would be based on cross-compilation in separate processes, similar to how Lisp builds currently happen with Bazel. Handling self-extension and phase separation properly in this context might be "interesting".
OK, but I think that should be a different thing -- perhaps POIU version 2, rather than an ASDF version. It's ok to adapt ASDF to support programming in the large, but not, I think, too much at the expense of conventional, smaller projects. At some point, probably the tool support for very large and smaller, more conventional systems, should diverge.
On Thu, Apr 27, 2017 at 10:35 AM, Robert Goldman <rpgoldman@sift.net> wrote:
On 4/26/17 Apr 26 -7:46 PM, Faré wrote:
On Wed, Apr 26, 2017 at 7:17 PM, Robert Goldman <rpgoldman@sift.net> wrote:
What more, it was a hurdle on the road to making a future ASDF a robust system with deterministic builds. I wanted to ask about this. Right now ASDF systems are (in general)
On 4/26/17 Apr 26 -6:33 AM, Faré wrote: partially ordered. Is it your plan to make sure there are no implementation or state dependencies in how these partial orders are linearized, in order to make it deterministic?
Well, a really deterministic variant of ASDF, if it ever happens, would be based on cross-compilation in separate processes, similar to how Lisp builds currently happen with Bazel. Handling self-extension and phase separation properly in this context might be "interesting".
OK, but I think that should be a different thing -- perhaps POIU version 2, rather than an ASDF version.
It's ok to adapt ASDF to support programming in the large, but not, I think, too much at the expense of conventional, smaller projects.
At some point, probably the tool support for very large and smaller, more conventional systems, should diverge.
I agree that one essential constraint of ASDF is that it should keep working in-image in the small and not depend on external processes or additional libraries. Any "deterministic build" extension should remain an extension indeed (though one you'd load early). However, if this extension is to remain compatible with ASDF and its .asd file, modifications and cleanups may have to be done to ASDF itself so it behaves well: even keeping that hypothetical deterministic build separate, I expect non-trivial changes to the ASDF API to enable, e.g. cross-compilation (i.e. a perform-form to replace perform, with perform as the fallback). I hope a backward-compatible trick can be found for a smooth transition. But you do well to mention POIU: adapting ASDF so POIU would be a nice extension was decisive in simplifying the internals of ASDF 2 and ASDF 3. Indeed quite a few incompatible modifications of the internals were informed by this it, from the fine structure of the component-depends-on, operate and perform-with-restart methods to the internal representation of timestamps to the structure of traversals (notably replacing do-first with needed-in-image-p) and the upward- and downward- (and more) operation classes, etc. Thus, I expect the design of the base ASDF to keep being influenced by such tools. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. — John F. Woods
I took the content of this thread and edited it into a blog post: http://fare.livejournal.com/188940.html Robert, can you tell me what the plan should be for ASDF 3.3, the "plan" (phase separation) branch and the "syntax-control" branch? Should I still prepare to merge the phase separation in ASDF 3.3.0? What about the syntax-control branch? ASDF 3.2.2? ASDF 3.3.0? ASDF 3.3.1? The branch needs to be rebased, reviewed, etc. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org You think minimum height restrictions make children taller? — Luke McCormick, regarding minimum wage restrictions
On 5/1/17 03:45, Faré wrote:
I took the content of this thread and edited it into a blog post: http://fare.livejournal.com/188940.html
Robert, can you tell me what the plan should be for ASDF 3.3, the "plan" (phase separation) branch and the "syntax-control" branch?
Thanks for the detailed write-up, Faré, as such exposition from your ASDF wrangling helps me greatly in understanding why things are the way they are in ASDF. Bonus points for the snark in "Common Lisp is a hippie language that disallows disallowing", at which I am still chuckling. In the JVM ecosystem, the standard way to identify packaged binary artifacts ("jar files") and their dependencies is via the Maven ["coordinate system" in the POM model][1] which associates every artifact with a group-id:artifact-id:version triple. An example of such a triple would be ["org.abcl:abcl:1.4.0" ][2]. The use of the POM model has been adopted by almost (?) all other current "Java the platform" build systems, including Clojure's Leiningen and ABCL's extensions to ASDF, [abcl-asdf][3]. [1]: https://maven.apache.org/pom.html [2]: https://search.maven.org/#artifactdetails%7Corg.abcl%7Cabcl%7C1.4.0%7Cjar [3]: https://gitlab.common-lisp.net/abcl/abcl/blob/master/contrib/abcl-asdf/abcl-... Unfortunately, the current implementation of ABCL-ASDF took a naive shortcut that needs fixing, as it works on a per-dependency basis as opposed to properly sorting and reducing declared dependencies to a minimal working set for a given ASDF system definition. Alan Ruttenberg has [proposed a short term fix][5] that we are currently experimenting with, but the proper way forward as I understand it, would be to push more of this dependency logic into ASDF where the "plan" phase would be able to properly reason about dependencies. [4]: https://mailman.common-lisp.net/pipermail/armedbear-devel/2017-April/003810.... [5]: https://mailman.common-lisp.net/pipermail/armedbear-devel/2017-April/003838.... An example to illuminate the problem as I understand it: Currently, when ABCL encounters the following definition (asdf:defsystem :log4j :defsystem-depends-on (abcl-asdf) :components ((:module log4j.jar :components ((:mvn "log4j/log4j/1.2.17"))))) a request is made to the Maven code to satisfy the dependency for "log4j/log4j/1.2.17" and add all transitive dependencies to the JVM's classpath. This graph is discovered at request time by parsing all the POM models from the network. Issuing this request at an ABCL repl just returned the following for me: "/Users/evenson/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/evenson/.m2/repository/javax/mail/mail/1.4.3/mail-1.4.3.jar:/Users/evenson/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/evenson/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.0/geronimo-jms_1.1_spec-1.0.jar" which means a request for a logging framework dependency (`log4j-1.2.17.jar`) has also pulled in an SMTP implementation (`mail-1.4.3.jar`) amongst other things (a common problem with the JVM, apparently not easily solved as the "resolution" of the packaging in jar files is too coarse grained). ABCL-ASDF currently only creates a single ASDF:MVN object that represents the entire request, not representing the other binary artifacts it introduces in a way that is in any way amendable to "planning". I figure that the right way forward would be to create additional ASDF:MVN objects for each discrete dependency in-memory that would not have a corresponding ASDF:MVN definition in a file on the filesystem. To continue the example, the ABCL-ASDF code would create the discovered "javax.mail/mail/1.4.3", "javax.activation/activation/1.4.3", and "org.apache.geronimo.specs/gerionimo-jms/1.1" ASDF:MVN objects, using them to reason about what to load. Questions for consideration by the ASDF community: 1. As I understand it, my problem doesn't have the problems of redefining ASDF behavior during ASDF's load phase that Faré wishes to eliminate in asdf-3.3. If I were to slog through asdf-3.2.x to implement the planning which I need would such an effort be impacted in any way that you can foresee by what you need to change for asdf-3.3? 2. Does anyone know if there an existing analogy for the ASDF:MVN component in an ASDF extension that I could profitably study? Currently, ABCL-ASDF hackishily neuters the ASDF:COMPONENT association with a pathname, mainly because in the current implementation a given ASDF:MVN component results in one or more jar archives. Does creating additional ASDF:MVN (a subclass of ASDF:COMPONENT) instances in the in-memory ASDF that aren't reflected in an *.asd file raise any problems that anyone is aware of? 3. In the last few months, I think I remember that there has been discussion around the possible use of ASDF to locate and download shared objects for CFFI definitions (or maybe this was within the CFFI community?). The ABCL-ASDF case is a little simpler in that the needed Maven binary objects are identical, unlike the CFFI problem which has per-operating systems and (possibly) per-dynamic linker implementation dependencies. But still, as I remember the outcome of that discussion, the general feeling was that such a mechanism does not belong in ASDF. Does the ASDF cognoscenti think that what I am proposing here for ABCL-ASDF also seem to be "too much"? Note, that ABCL-ASDF will only ever work on ABCL (unless, of course, we get another JVM CL implementation), and as such, is intended to be written as an ASDF extension that should have no impact on other's usage of ASDF. Still, what has been implemented (and what I am proposing), seems to violate Faré's description of ASDF as a build system that should only deal with Common Lisp artifacts. Thanks for the attention and the solid re-engineering of ASDF, Mark <evenson@panix.com> -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Dear Mark, thanks for your request. I'm not sure I understand how your ASDF dependencies do or don't map to MVN entities, though. I also don't understand the maven model very well.
Questions for consideration by the ASDF community:
1. As I understand it, my problem doesn't have the problems of redefining ASDF behavior during ASDF's load phase that Faré wishes to eliminate in asdf-3.3. If I were to slog through asdf-3.2.x to implement the planning which I need would such an effort be impacted in any way that you can foresee by what you need to change for asdf-3.3?
If you're going to go deep inside ASDF's planning code, I would strongly recommend starting off the 'plan' branch (due to be committed in ASDF 3.3) rather than off 3.2, as there was significant refactoring and merging would be a huge pain. Also, if some of your dependencies are from defsystem-depends-on, you *definitely* need to start off the 'plan' branch.
2. Does anyone know if there an existing analogy for the ASDF:MVN component in an ASDF extension that I could profitably study? Currently, ABCL-ASDF hackishily neuters the ASDF:COMPONENT association with a pathname, mainly because in the current implementation a given ASDF:MVN component results in one or more jar archives. Does creating additional ASDF:MVN (a subclass of ASDF:COMPONENT) instances in the in-memory ASDF that aren't reflected in an *.asd file raise any problems that anyone is aware of?
The current ASDF model is that every component has only one pathname, though that pathname can be NIL, or can be a directory. If you need multiple files, you can create multiple components indeed. I don't understand what an MVN component is or does, but I don't see any problem a priori with having multiple of them. Is it supposed to represent (a) a dependency on an external maven package? Or is it supposed to represent (b) a jar that is included as pre-built in the current source? Or (c) the ability to build a jar from source in the current system, that can then be pushed to maven? In case (a), it probably should be a special subclass of SYSTEM. In case (b) a component with a single file pathname should be fine. In case (c) I would probably also have some subclass of SYSTEM do it, and use one or many secondary systems.
3. In the last few months, I think I remember that there has been discussion around the possible use of ASDF to locate and download shared objects for CFFI definitions (or maybe this was within the CFFI community?). The ABCL-ASDF case is a little simpler in that the needed Maven binary objects are identical, unlike the CFFI problem which has per-operating systems and (possibly) per-dynamic linker implementation dependencies. But still, as I remember the outcome of that discussion, the general feeling was that such a mechanism does not belong in ASDF. Does the ASDF cognoscenti think that what I am proposing here for ABCL-ASDF also seem to be "too much"? Note, that ABCL-ASDF will only ever work on ABCL (unless, of course, we get another JVM CL implementation), and as such, is intended to be written as an ASDF extension that should have no impact on other's usage of ASDF. Still, what has been implemented (and what I am proposing), seems to violate Faré's description of ASDF as a build system that should only deal with Common Lisp artifacts.
I don't remember any discussion about downloading CFFI shared objects (though I have vague recollections of discussions about cl-test-grid and/or Quicklisp doing automatic installation of Ubuntu packages or such.)
Thanks for the attention and the solid re-engineering of ASDF, Thanks for your appreciation.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Anyone who says he can see through women is missing a lot. — Groucho Marx
On 5/4/17 17:08, Faré wrote:
Dear Mark,
thanks for your request. I'm not sure I understand how your ASDF dependencies do or don't map to MVN entities, though. I also don't understand the maven model very well.
[…]
In case (a), it probably should be a special subclass of SYSTEM. In case (b) a component with a single file pathname should be fine. In case (c) I would probably also have some subclass of SYSTEM do it, and use one or many secondary systems.
Exactly what I needed to know. I'll rebase my inheritance off of ASDF:SYSTEM instead of ASDF:COMPONENT, working off your 'plan' branch. More questions when I have an implementation to kick around… […]
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Anyone who says he can see through women is missing a lot. — Groucho Marx
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
After a bit of analysis, I have [my first issue to report][1] with running asdf-3.3 (aka [the plan branch][2]) on [ABCL][3]. The problem stems from the new--and quite sensible--restriction that any [code in a ASDF:PERFORM cannot call into another ASDF:PERFORM operation][4]. QUICKLISP-ABCL's ability to encapsulate the load of Quicklisp in an ASDF definition has been quite useful to me, and I'd hate to abandon it. Of course I could stuff the code ensuring Quicklisp's local installation in an the ABCL specific package, but maybe there is another way to encapsulate this situation in ASDF? Otherwise, it would seem that ASDF loses a little functionality here with the current asdf-3.3, which possibly might be the tradeoff for a more deterministic plan over all operations. But it would be nice to "allow allowing" here… [1]: https://gitlab.common-lisp.net/mevenson/asdf/issues/1 [2]: https://gitlab.common-lisp.net/asdf/asdf/tree/plan [3]: https://gitlab.common-lisp.net/mevenson/asdf/uploads/3039ebfe40dc111a3ab6d06... [4]: https://gitlab.common-lisp.net/asdf/asdf/blob/plan/doc/best_practices.md#mor... -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Here is what I see a problem is: ASDF only resolves dependencies on a system by system basis. The cross-system link is that if system B depends on system A we can avoid loading A if it has already been loaded. If an identical source file happens to be in both B and A that isn't recognized, but also doesn't have consequence - the functions in it are redefined. Resolving the maven dependencies in java needs to be a globally done across ASDF systems. If one ASDF system A depends on log4j version 2.1 or higher and another B depends on 2.9, and the range of versions is 2.1-2.12 then if A is loaded first there is no way to satisfy the second dependence in a predictable way. If you choose the highest available version A can take it is too high for B. If you choose the lowest A can choose it is too low for B. Java systems manage this because they are built from a single root POM file. That POM can include other POMs which have additional dependencies, but solving the full set of dependencies consistently is done at one fell swoop. While there may be seeming conflicts, there are mechanisms for making choices - the including POM can specify a version for a dependency an included POM specifies - "managed dependencies" and exclusions. There's also a fundamental difference in that what maven is doing with the dependencies is collecting them into a classpath, and then the process is run with the classpath. Because of this there aren't load-time side effects, as one has with lisp. There are static initializers that depend on classes in other jars, but these are never called for until the process is started. I'm honestly not sure how to handle this generally, or specifically with ASDF. One piece could be having a persistent registry of components (as opposed to just systems). Then, instead of just reloading a component that happens to be the same, we could detect whether the requested component's version is compatible with what was previously loaded. Another piece could be to distinguish between dependencies needed for loading the system versus runtime, and then handling the planning for a system and all the systems it depends on at once, instead of one planning one system at a time, with the default being load-time independence. That default would be analogous to the need to make exceptions for within file by use eval-when :compile-toplevel. Hope this helps. Perhaps both these capabilities are already easily available. My use of ASDF doesn't extend far enough to know, and I haven't spent a lot of time studying ASDF. Any ideas would certainly be welcome. Alan On Thu, May 4, 2017 at 11:08 AM, Faré <fahree@gmail.com> wrote:
Dear Mark,
thanks for your request. I'm not sure I understand how your ASDF dependencies do or don't map to MVN entities, though. I also don't understand the maven model very well.
Questions for consideration by the ASDF community:
1. As I understand it, my problem doesn't have the problems of redefining ASDF behavior during ASDF's load phase that Faré wishes to eliminate in asdf-3.3. If I were to slog through asdf-3.2.x to implement the planning which I need would such an effort be impacted in any way that you can foresee by what you need to change for asdf-3.3?
If you're going to go deep inside ASDF's planning code, I would strongly recommend starting off the 'plan' branch (due to be committed in ASDF 3.3) rather than off 3.2, as there was significant refactoring and merging would be a huge pain.
Also, if some of your dependencies are from defsystem-depends-on, you *definitely* need to start off the 'plan' branch.
2. Does anyone know if there an existing analogy for the ASDF:MVN component in an ASDF extension that I could profitably study? Currently, ABCL-ASDF hackishily neuters the ASDF:COMPONENT association with a pathname, mainly because in the current implementation a given ASDF:MVN component results in one or more jar archives. Does creating additional ASDF:MVN (a subclass of ASDF:COMPONENT) instances in the in-memory ASDF that aren't reflected in an *.asd file raise any problems that anyone is aware of?
The current ASDF model is that every component has only one pathname, though that pathname can be NIL, or can be a directory. If you need multiple files, you can create multiple components indeed.
I don't understand what an MVN component is or does, but I don't see any problem a priori with having multiple of them. Is it supposed to represent (a) a dependency on an external maven package? Or is it supposed to represent (b) a jar that is included as pre-built in the current source? Or (c) the ability to build a jar from source in the current system, that can then be pushed to maven?
In case (a), it probably should be a special subclass of SYSTEM. In case (b) a component with a single file pathname should be fine. In case (c) I would probably also have some subclass of SYSTEM do it, and use one or many secondary systems.
3. In the last few months, I think I remember that there has been discussion around the possible use of ASDF to locate and download shared objects for CFFI definitions (or maybe this was within the CFFI community?). The ABCL-ASDF case is a little simpler in that the needed Maven binary objects are identical, unlike the CFFI problem which has per-operating systems and (possibly) per-dynamic linker implementation dependencies. But still, as I remember the outcome of that discussion, the general feeling was that such a mechanism does not belong in ASDF. Does the ASDF cognoscenti think that what I am proposing here for ABCL-ASDF also seem to be "too much"? Note, that ABCL-ASDF will only ever work on ABCL (unless, of course, we get another JVM CL implementation), and as such, is intended to be written as an ASDF extension that should have no impact on other's usage of ASDF. Still, what has been implemented (and what I am proposing), seems to violate Faré's description of ASDF as a build system that should only deal with Common Lisp artifacts.
I don't remember any discussion about downloading CFFI shared objects (though I have vague recollections of discussions about cl-test-grid and/or Quicklisp doing automatic installation of Ubuntu packages or such.)
Thanks for the attention and the solid re-engineering of ASDF, Thanks for your appreciation.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Anyone who says he can see through women is missing a lot. — Groucho Marx
On 5/4/17 May 4 -11:01 AM, Alan Ruttenberg wrote:
Here is what I see a problem is: ASDF only resolves dependencies on a system by system basis. The cross-system link is that if system B depends on system A we can avoid loading A if it has already been loaded. If an identical source file happens to be in both B and A that isn't recognized, but also doesn't have consequence - the functions in it are redefined.
That's correct. If you want to build a composite system with A and B, you should create a system C that depends on both. Note that this problem you cite for Maven really exists for ASDF anyway. If you have two lisp systems that collide with each other (e.g., package collision), then you lose with ASDF. So I think what you have to do here is track the Maven dependencies, detect incompatible versions of a single library, and throw an error for the programmer to fix if it happens. Note that this reminds me that ASDF currently (for Lisp) doesn't support a maximum version number, only a minimum. I keep meaning to fix that, but never get around to it. And yes, it does happen, I have had systems that relied on, e.g., older versions of XMLS. For now we handled that by pinning revisions in the RCS, but that can be problematic if you get the ASDF config wrong. Cheers, r
On 5/4/17 May 4 -11:01 AM, Alan Ruttenberg wrote:
Here is what I see a problem is: ASDF only resolves dependencies on a system by system basis. The cross-system link is that if system B depends on system A we can avoid loading A if it has already been loaded. If an identical source file happens to be in both B and A that isn't recognized, but also doesn't have consequence - the functions in it are redefined.
That's correct. If you want to build a composite system with A and B, you should create a system C that depends on both.
Note that this problem you cite for Maven really exists for ASDF anyway. If you have two lisp systems that collide with each other (e.g., package collision), then you lose with ASDF.
So I think what you have to do here is track the Maven dependencies, detect incompatible versions of a single library, and throw an error for the programmer to fix if it happens.
Note that this reminds me that ASDF currently (for Lisp) doesn't support a maximum version number, only a minimum. I keep meaning to fix that, but never get around to it.
I believe this would be doubling down on the error of specifying versions in ASDF.
And yes, it does happen, I have had systems that relied on, e.g., older versions of XMLS. For now we handled that by pinning revisions in the RCS, but that can be problematic if you get the ASDF config wrong.
Getting it wrong in what sense ? -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On 5/4/17 18:41, Stelian Ionescu wrote: […]
ASDF currently (for Lisp) doesn't support
a maximum version number, only a minimum. I keep meaning to fix that, but never get around to it.
I believe this would be doubling down on the error of specifying versions in ASDF.
But as long as we have a notion of ASDF:VERSION it should certainly have a possible upper bound capability for its assertions, whatever your local mapping to such assertions may be… -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 5/4/17 18:41, Stelian Ionescu wrote: […]
ASDF currently (for Lisp) doesn't support
a maximum version number, only a minimum. I keep meaning to fix that, but never get around to it.
I believe this would be doubling down on the error of specifying versions in ASDF.
But as long as we have a notion of ASDF:VERSION it should certainly have a possible upper bound capability for its assertions, whatever your local mapping to such assertions may be…
I would rather start by deprecating it. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On 5/4/17 May 4 -12:16 PM, Stelian Ionescu wrote:
On 5/4/17 18:41, Stelian Ionescu wrote: […]
ASDF currently (for Lisp) doesn't support
a maximum version number, only a minimum. I keep meaning to fix that, but never get around to it.
I believe this would be doubling down on the error of specifying versions in ASDF.
But as long as we have a notion of ASDF:VERSION it should certainly have a possible upper bound capability for its assertions, whatever your local mapping to such assertions may be…
I would rather start by deprecating it.
This will not be happening unless you take over ASDF maintainership.
On 5/4/17 May 4 -11:41 AM, Stelian Ionescu wrote:
On 5/4/17 May 4 -11:01 AM, Alan Ruttenberg wrote:
Here is what I see a problem is: ASDF only resolves dependencies on a system by system basis. The cross-system link is that if system B depends on system A we can avoid loading A if it has already been loaded. If an identical source file happens to be in both B and A that isn't recognized, but also doesn't have consequence - the functions in it are redefined.
That's correct. If you want to build a composite system with A and B, you should create a system C that depends on both.
Note that this problem you cite for Maven really exists for ASDF anyway. If you have two lisp systems that collide with each other (e.g., package collision), then you lose with ASDF.
So I think what you have to do here is track the Maven dependencies, detect incompatible versions of a single library, and throw an error for the programmer to fix if it happens.
Note that this reminds me that ASDF currently (for Lisp) doesn't support a maximum version number, only a minimum. I keep meaning to fix that, but never get around to it.
I believe this would be doubling down on the error of specifying versions in ASDF.
No, I comprehensively disagree. If you have a stable system that people are using, and one of your libraries changes incompatibly, until you have the time to update to the new API, you should be able to declaratively specify that the new API will break your system, so that users don't get incomprehensible errors. In a perfect world, all systems would be updated to the latest versions of their dependencies. This is not a perfect world.
And yes, it does happen, I have had systems that relied on, e.g., older versions of XMLS. For now we handled that by pinning revisions in the RCS, but that can be problematic if you get the ASDF config wrong.
Getting it wrong in what sense ?
If you work on multiple projects, so have multiple different ASDF configurations, and one leaks into another. I had this with a novice user who was told to put in a shell config file, then forgot, and didn't understand why they got the wrong libraries.
On Thu, May 4, 2017 at 12:01 PM, Alan Ruttenberg <alanruttenberg@gmail.com> wrote:
Resolving the maven dependencies in java needs to be a globally done across ASDF systems. If one ASDF system A depends on log4j version 2.1 or higher and another B depends on 2.9, and the range of versions is 2.1-2.12 then if A is loaded first there is no way to satisfy the second dependence in a predictable way. If you choose the highest available version A can take it is too high for B. If you choose the lowest A can choose it is too low for B.
The best I can see you do with ASDF is as follows: 0- Aim for a single coherent set of jars within a given build, because otherwise lies madness. 1- Within a given plan-then-perform phase, collect the set of jars and their version intervals in a :before method on perform-plan that talks to maven and uses some heuristic to resolve versions. 2- Across plan-then-perform phases of a same build session, record which versions were loaded, and issue an error if it's incompatible. 3- Across build sessions, remember what version was previously loaded, and unload/shadow it if it is incompatible (or throw an error if you can't), and decide whether to keep or upgrade (if possible) if it's compatible. As to writing a system that solves issues across phases, in your worst case, your toplevel system would :defsystem-depends-on a system that loads all the proper versions of all the jar files (or at least the problematic ones), and then whichever phase causes some version to be loaded will have been preempted by that first defsystem-depends-on system. Which reminds me that load-systems should be amended to do only one plan-then-perform phase for all the requested systems. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Anyone who says he can see through women is missing a lot. — Groucho Marx
On Thu, May 4, 2017 at 7:56 AM, Mark Evenson <evenson@panix.com> wrote:
Still, what has been implemented (and what I am proposing), seems to violate Faré's description of ASDF as a build system that should only deal with Common Lisp artifacts.
I don't remember writing that ASDF *should* only deal with CL artifacts. Historically, that's what it always did; but "CL artifacts" has included "C code linked into the CL runtime" for a *long* time. And this year, this includes static linking as well as dynamic linking. Moreover, ASDF has the potential of doing more than that. Indeed, cl-unicode drives processing of data files using ASDF. QRes also used to massage its configuration files or to generate CL code from ASDF. ASDF *should* do whatever you need to build your CL application. Now to actually make that happens can be easy or hard, depending on what that is. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The older I grow, the more I distrust the familiar doctrine that age brings wisdom. — H.L. Mencken
On 5/4/17 May 4 -2:05 PM, Faré wrote:
On Thu, May 4, 2017 at 7:56 AM, Mark Evenson <evenson@panix.com> wrote:
Still, what has been implemented (and what I am proposing), seems to violate Faré's description of ASDF as a build system that should only deal with Common Lisp artifacts.
I don't remember writing that ASDF *should* only deal with CL artifacts. Historically, that's what it always did; but "CL artifacts" has included "C code linked into the CL runtime" for a *long* time. And this year, this includes static linking as well as dynamic linking. Moreover, ASDF has the potential of doing more than that. Indeed, cl-unicode drives processing of data files using ASDF. QRes also used to massage its configuration files or to generate CL code from ASDF.
ASDF *should* do whatever you need to build your CL application. Now to actually make that happens can be easy or hard, depending on what that is.
Seconded. We routinely use ASDF to load systems in other programming languages that are built with MAKE, but that our CL code interacts with....
participants (6)
-
Alan Ruttenberg -
Attila Lendvai -
Faré -
Mark Evenson -
Robert Goldman -
Stelian Ionescu