On 5/1/17 03:45, Faré wrote:
I took the content of this thread and edited it into a blog post: http://fare.livejournal.com/188940.html
Robert, can you tell me what the plan should be for ASDF 3.3, the "plan" (phase separation) branch and the "syntax-control" branch?
Thanks for the detailed write-up, Faré, as such exposition from your ASDF wrangling helps me greatly in understanding why things are the way they are in ASDF. Bonus points for the snark in "Common Lisp is a hippie language that disallows disallowing", at which I am still chuckling.
In the JVM ecosystem, the standard way to identify packaged binary artifacts ("jar files") and their dependencies is via the Maven ["coordinate system" in the POM model][1] which associates every artifact with a group-id:artifact-id:version triple. An example of such a triple would be ["org.abcl:abcl:1.4.0" ][2]. The use of the POM model has been adopted by almost (?) all other current "Java the platform" build systems, including Clojure's Leiningen and ABCL's extensions to ASDF, [abcl-asdf][3].
[1]: https://maven.apache.org/pom.html [2]: https://search.maven.org/#artifactdetails%7Corg.abcl%7Cabcl%7C1.4.0%7Cjar [3]: https://gitlab.common-lisp.net/abcl/abcl/blob/master/contrib/abcl-asdf/abcl-...
Unfortunately, the current implementation of ABCL-ASDF took a naive shortcut that needs fixing, as it works on a per-dependency basis as opposed to properly sorting and reducing declared dependencies to a minimal working set for a given ASDF system definition. Alan Ruttenberg has [proposed a short term fix][5] that we are currently experimenting with, but the proper way forward as I understand it, would be to push more of this dependency logic into ASDF where the "plan" phase would be able to properly reason about dependencies.
[4]: https://mailman.common-lisp.net/pipermail/armedbear-devel/2017-April/003810.... [5]: https://mailman.common-lisp.net/pipermail/armedbear-devel/2017-April/003838....
An example to illuminate the problem as I understand it:
Currently, when ABCL encounters the following definition
(asdf:defsystem :log4j :defsystem-depends-on (abcl-asdf) :components ((:module log4j.jar :components ((:mvn "log4j/log4j/1.2.17")))))
a request is made to the Maven code to satisfy the dependency for "log4j/log4j/1.2.17" and add all transitive dependencies to the JVM's classpath. This graph is discovered at request time by parsing all the POM models from the network. Issuing this request at an ABCL repl just returned the following for me:
"/Users/evenson/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/evenson/.m2/repository/javax/mail/mail/1.4.3/mail-1.4.3.jar:/Users/evenson/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/evenson/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.0/geronimo-jms_1.1_spec-1.0.jar"
which means a request for a logging framework dependency (`log4j-1.2.17.jar`) has also pulled in an SMTP implementation (`mail-1.4.3.jar`) amongst other things (a common problem with the JVM, apparently not easily solved as the "resolution" of the packaging in jar files is too coarse grained). ABCL-ASDF currently only creates a single ASDF:MVN object that represents the entire request, not representing the other binary artifacts it introduces in a way that is in any way amendable to "planning".
I figure that the right way forward would be to create additional ASDF:MVN objects for each discrete dependency in-memory that would not have a corresponding ASDF:MVN definition in a file on the filesystem. To continue the example, the ABCL-ASDF code would create the discovered "javax.mail/mail/1.4.3", "javax.activation/activation/1.4.3", and "org.apache.geronimo.specs/gerionimo-jms/1.1" ASDF:MVN objects, using them to reason about what to load.
Questions for consideration by the ASDF community:
1. As I understand it, my problem doesn't have the problems of redefining ASDF behavior during ASDF's load phase that Faré wishes to eliminate in asdf-3.3. If I were to slog through asdf-3.2.x to implement the planning which I need would such an effort be impacted in any way that you can foresee by what you need to change for asdf-3.3?
2. Does anyone know if there an existing analogy for the ASDF:MVN component in an ASDF extension that I could profitably study? Currently, ABCL-ASDF hackishily neuters the ASDF:COMPONENT association with a pathname, mainly because in the current implementation a given ASDF:MVN component results in one or more jar archives. Does creating additional ASDF:MVN (a subclass of ASDF:COMPONENT) instances in the in-memory ASDF that aren't reflected in an *.asd file raise any problems that anyone is aware of?
3. In the last few months, I think I remember that there has been discussion around the possible use of ASDF to locate and download shared objects for CFFI definitions (or maybe this was within the CFFI community?). The ABCL-ASDF case is a little simpler in that the needed Maven binary objects are identical, unlike the CFFI problem which has per-operating systems and (possibly) per-dynamic linker implementation dependencies. But still, as I remember the outcome of that discussion, the general feeling was that such a mechanism does not belong in ASDF. Does the ASDF cognoscenti think that what I am proposing here for ABCL-ASDF also seem to be "too much"? Note, that ABCL-ASDF will only ever work on ABCL (unless, of course, we get another JVM CL implementation), and as such, is intended to be written as an ASDF extension that should have no impact on other's usage of ASDF. Still, what has been implemented (and what I am proposing), seems to violate Faré's description of ASDF as a build system that should only deal with Common Lisp artifacts.
Thanks for the attention and the solid re-engineering of ASDF, Mark evenson@panix.com