Here is what I see a problem is: ASDF only resolves dependencies on a system by system basis. The cross-system link is that if system B depends on system A we can avoid loading A if it has already been loaded. If an identical source file happens to be in both B and A that isn't recognized, but also doesn't have consequence - the functions in it are redefined.
Resolving the maven dependencies in java needs to be a globally done across ASDF systems. If one ASDF system A depends on log4j version 2.1 or higher and another B depends on 2.9, and the range of versions is 2.1-2.12 then if A is loaded first there is no way to satisfy the second dependence in a predictable way. If you choose the highest available version A can take it is too high for B. If you choose the lowest A can choose it is too low for B.
Java systems manage this because they are built from a single root POM file. That POM can include other POMs which have additional dependencies, but solving the full set of dependencies consistently is done at one fell swoop. While there may be seeming conflicts, there are mechanisms for making choices - the including POM can specify a version for a dependency an included POM specifies - "managed dependencies" and exclusions. There's also a fundamental difference in that what maven is doing with the dependencies is collecting them into a classpath, and then the process is run with the classpath. Because of this there aren't load-time side effects, as one has with lisp. There are static initializers that depend on classes in other jars, but these are never called for until the process is started.
I'm honestly not sure how to handle this generally, or specifically with ASDF. One piece could be having a persistent registry of components (as opposed to just systems). Then, instead of just reloading a component that happens to be the same, we could detect whether the requested component's version is compatible with what was previously loaded. Another piece could be to distinguish between dependencies needed for loading the system versus runtime, and then handling the planning for a system and all the systems it depends on at once, instead of one planning one system at a time, with the default being load-time independence. That default would be analogous to the need to make exceptions for within file by use eval-when :compile-toplevel.
Hope this helps. Perhaps both these capabilities are already easily available. My use of ASDF doesn't extend far enough to know, and I haven't spent a lot of time studying ASDF.
Any ideas would certainly be welcome.
Alan
On Thu, May 4, 2017 at 11:08 AM, Faré fahree@gmail.com wrote:
Dear Mark,
thanks for your request. I'm not sure I understand how your ASDF dependencies do or don't map to MVN entities, though. I also don't understand the maven model very well.
Questions for consideration by the ASDF community:
- As I understand it, my problem doesn't have the problems of redefining ASDF behavior during ASDF's load phase that Faré wishes to eliminate in asdf-3.3. If I were to slog through asdf-3.2.x to implement the planning which I need would such an effort be impacted in any way that you can foresee by what you need to change for asdf-3.3?
If you're going to go deep inside ASDF's planning code, I would strongly recommend starting off the 'plan' branch (due to be committed in ASDF 3.3) rather than off 3.2, as there was significant refactoring and merging would be a huge pain.
Also, if some of your dependencies are from defsystem-depends-on, you *definitely* need to start off the 'plan' branch.
- Does anyone know if there an existing analogy for the ASDF:MVN component in an ASDF extension that I could profitably study? Currently, ABCL-ASDF hackishily neuters the ASDF:COMPONENT association with a pathname, mainly because in the current implementation a given ASDF:MVN component results in one or more jar archives. Does creating additional ASDF:MVN (a subclass of ASDF:COMPONENT) instances in the in-memory ASDF that aren't reflected in an *.asd file raise any problems that anyone is aware of?
The current ASDF model is that every component has only one pathname, though that pathname can be NIL, or can be a directory. If you need multiple files, you can create multiple components indeed.
I don't understand what an MVN component is or does, but I don't see any problem a priori with having multiple of them. Is it supposed to represent (a) a dependency on an external maven package? Or is it supposed to represent (b) a jar that is included as pre-built in the current source? Or (c) the ability to build a jar from source in the current system, that can then be pushed to maven?
In case (a), it probably should be a special subclass of SYSTEM. In case (b) a component with a single file pathname should be fine. In case (c) I would probably also have some subclass of SYSTEM do it, and use one or many secondary systems.
- In the last few months, I think I remember that there has been discussion around the possible use of ASDF to locate and download shared objects for CFFI definitions (or maybe this was within the CFFI community?). The ABCL-ASDF case is a little simpler in that the needed Maven binary objects are identical, unlike the CFFI problem which has per-operating systems and (possibly) per-dynamic linker implementation dependencies. But still, as I remember the outcome of that discussion, the general feeling was that such a mechanism does not belong in ASDF. Does the ASDF cognoscenti think that what I am proposing here for ABCL-ASDF also seem to be "too much"? Note, that ABCL-ASDF will only ever work on ABCL (unless, of course, we get another JVM CL implementation), and as such, is intended to be written as an ASDF extension that should have no impact on other's usage of ASDF. Still, what has been implemented (and what I am proposing), seems to violate Faré's description of ASDF as a build system that should only deal with Common Lisp artifacts.
I don't remember any discussion about downloading CFFI shared objects (though I have vague recollections of discussions about cl-test-grid and/or Quicklisp doing automatic installation of Ubuntu packages or such.)
Thanks for the attention and the solid re-engineering of ASDF,
Thanks for your appreciation.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Anyone who says he can see through women is missing a lot. — Groucho Marx