I'm using :mvn ASDF components and ran into a problem with having multiple versions of the same library. The issue boils down to the fact that ABCL currently processes dependencies for :mvn components one at a time, which doesn't take into account mutual dependencies. Attached is a function resolve-multiple-maven-dependencies, which takes a set of maven components and computes the dependencies. As an example, here is a comparison of treating the dependencies individually vs at once. (length (mapcan 'resolve-multiple-maven-dependencies '(("net.sourceforge.owlapi:owlapi-distribution:4.2.6") ("net.sourceforge.owlapi:org.semanticweb.hermit:1.3.8.413") ("org.semanticweb.elk/elk-reasoner/0.4.3") ("net.sourceforge.owlapi/pellet-cli-ignazio1977/2.4.0-ignazio1977") ("net.sourceforge.owlapi/owlexplanation/2.0.0")))) -> 210 (length (mapcan 'resolve-multiple-maven-dependencies '(("net.sourceforge.owlapi:owlapi-distribution:4.2.6" "net.sourceforge.owlapi:org.semanticweb.hermit:1.3.8.413" "org.semanticweb.elk/elk-reasoner/0.4.3" "net.sourceforge.owlapi/pellet-cli-ignazio1977/2.4.0-ignazio1977" "net.sourceforge.owlapi/owlexplanation/2.0.0")))) -> 90 In particular there are several versions of owlapi-distribution that are put on the classpath. Which one is actually used is not easily predictable. In my case even though I specified version 4.2.6 in my system definition, 4.1.3 was actually used. The attached code also has features not yet supported by the :mvn syntax that are really needed: The ability to override versions of a particular dependency that a downstream dependency specifies, and the ability to exclude certain artifacts altogether. Mark has suggested that the maven resolution could be moved to the ASDF planning stage, which sounds like a good idea. However there is still an issue regarding maven dependencies in different loaded ASDF systems. One idea is to remember dependencies across systems and then uses them as constraints on subsequent maven resolutions. Comments and ideas would be appreciated. Alan
Here's a (perhaps hacky) fix for the moment. https://github.com/alanruttenberg/abcl/commit/fab11a1b35ff143ed28d4c2f08f4fc... Lets me write: (asdf:defsystem owl2libs-mvn2 :description "Non-Lisp dependencies necessary for OWL to function." :components ((:mvn-module "maven" :dependencies ("net.sourceforge.owlapi/pellet-cli-ignazio1977/2.4.0-ignazio1977" "org.semanticweb.elk/elk-reasoner/0.4.3" "net.sourceforge.owlapi/org.semanticweb.hermit/1.3.8.413" "net.sourceforge.owlapi/owlapi-distribution/4.2.6" "net.sourceforge.owlapi/owlexplanation/2.0.0" "de.sciss/prefuse-core/1.0.1" "de.sciss/prefuse-demos/1.0.1") :managed-dependencies ("org.slf4j/slf4j-api/1.7.21" "net.sourceforge.owlapi:owlapi-distribution:4.2.6") :exclusions ("net.sourceforge.owlapi:owlapi-osgidistribution" "edu.stanford.protege:org.protege.editor.owl")) (:module rest :pathname "lib" :components ((:jar-file "factplusplus-1.6.4-SNAPSHOT") (:jar-file "LSWTreeview-1.0.0") (:jar-file "QuotedStringAnnotationVisitor-1.0.0"))) (:module lib :pathname "lib" :depends-on ("maven" rest)))) It doesn't address what to do about potentially conflicting maven artifacts loaded by distinct asdf systems. On Fri, Apr 7, 2017 at 11:50 AM, Alan Ruttenberg <alanruttenberg@gmail.com> wrote:
I'm using :mvn ASDF components and ran into a problem with having multiple versions of the same library. The issue boils down to the fact that ABCL currently processes dependencies for :mvn components one at a time, which doesn't take into account mutual dependencies. Attached is a function resolve-multiple-maven-dependencies, which takes a set of maven components and computes the dependencies. As an example, here is a comparison of treating the dependencies individually vs at once.
(length (mapcan 'resolve-multiple-maven-dependencies '(("net.sourceforge.owlapi:owlapi-distribution:4.2.6") ("net.sourceforge.owlapi:org.semanticweb.hermit:1.3.8.413") ("org.semanticweb.elk/elk-reasoner/0.4.3") ("net.sourceforge.owlapi/pellet-cli-ignazio1977/2.4.0-ignazio1977") ("net.sourceforge.owlapi/owlexplanation/2.0.0")))) -> 210
(length (mapcan 'resolve-multiple-maven-dependencies '(("net.sourceforge.owlapi:owlapi-distribution:4.2.6" "net.sourceforge.owlapi:org.semanticweb.hermit:1.3.8.413" "org.semanticweb.elk/elk-reasoner/0.4.3" "net.sourceforge.owlapi/pellet-cli-ignazio1977/2.4.0-ignazio1977" "net.sourceforge.owlapi/owlexplanation/2.0.0"))))
-> 90
In particular there are several versions of owlapi-distribution that are put on the classpath. Which one is actually used is not easily predictable. In my case even though I specified version 4.2.6 in my system definition, 4.1.3 was actually used.
The attached code also has features not yet supported by the :mvn syntax that are really needed: The ability to override versions of a particular dependency that a downstream dependency specifies, and the ability to exclude certain artifacts altogether.
Mark has suggested that the maven resolution could be moved to the ASDF planning stage, which sounds like a good idea. However there is still an issue regarding maven dependencies in different loaded ASDF systems. One idea is to remember dependencies across systems and then uses them as constraints on subsequent maven resolutions.
Comments and ideas would be appreciated.
Alan
On 4/7/17 17:50, Alan Ruttenberg wrote:
I'm using :mvn ASDF components and ran into a problem with having multiple versions of the same library. The issue boils down to the fact that ABCL currently processes dependencies for :mvn components one at a time, which doesn't take into account mutual dependencies. Attached is a function resolve-multiple-maven-dependencies, which takes a set of maven components and computes the dependencies.
[…]
The attached code also has features not yet supported by the :mvn syntax that are really needed: The ability to override versions of a particular dependency that a downstream dependency specifies, and the ability to exclude certain artifacts altogether.
Mark has suggested that the maven resolution could be moved to the ASDF planning stage, which sounds like a good idea.
[…] I have [updated abcl-1.5.0 to][maven-3.2.1] which should serve as a good basis for work on pushing the resolution into the ASDF planning layer. In addition, [t/maven][] contains Alan's test code to follow along at home. [asdf-3.2.1]: https://gitlab.common-lisp.net/abcl/abcl/commit/a479be0771b037b091bfc75e1630... [t/maven]: https://gitlab.common-lisp.net/abcl/abcl/commit/4247dfe96284fb90faced00da1e3... -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Fri, Apr 7, 2017 at 4:27 PM, Alan Ruttenberg <alanruttenberg@gmail.com> wrote:
It doesn't address what to do about potentially conflicting maven artifacts loaded by distinct asdf systems.
Here's one approach. :managed-dependencies are specifications of versions that should take precedence over versions that are in the transitive closure of some dependency. As each asdf system which has used maven dependencies computes its set of satisfying dependencies, the particular versions chosen become default managed-dependencies for subsequent systems. If two managed dependencies conflict (e.g. I include a version 1.1 as managed dependency in system A, which is loaded. Subsequently system B specifies version 1.2 as a managed dependency, then an error is signaled. Many times this can be fixed by modifying the specification of the managed dependency to be looser, and changing the order of loaded systems. I think moving this into the planning phase is probably the right thing. I'm just not familiar enough with asdf's model to whip up a solution. Hopefully asdf's planning runs over all systems that are depended on before loading anything, in which case I think this will be a viable solution, module the case of conflicting managed dependencies, or dependency hell you would arrive at if you were using maven alone. Alan
On 4/7/17 Apr 7 -4:06 PM, Alan Ruttenberg wrote:
On Fri, Apr 7, 2017 at 4:27 PM, Alan Ruttenberg <alanruttenberg@gmail.com <mailto:alanruttenberg@gmail.com>> wrote:
It doesn't address what to do about potentially conflicting maven artifacts loaded by distinct asdf systems.
Here's one approach.
:managed-dependencies are specifications of versions that should take precedence over versions that are in the transitive closure of some dependency.
As each asdf system which has used maven dependencies computes its set of satisfying dependencies, the particular versions chosen become default managed-dependencies for subsequent systems. If two managed dependencies conflict (e.g. I include a version 1.1 as managed dependency in system A, which is loaded. Subsequently system B specifies version 1.2 as a managed dependency, then an error is signaled. Many times this can be fixed by modifying the specification of the managed dependency to be looser, and changing the order of loaded systems.
I think moving this into the planning phase is probably the right thing. I'm just not familiar enough with asdf's model to whip up a solution. Hopefully asdf's planning runs over all systems that are depended on before loading anything, in which case I think this will be a viable solution, module the case of conflicting managed dependencies, or dependency hell you would arrive at if you were using maven alone.
Yes, ASDF has a "plan, then execute" model of operation, so it won't load anything until inspecting all the systems involved.
Hi Robert, If you are familiar with the architecture, perhaps you can give me some hints where to intervene. The behavior I want is: Assume I load a system at top-level and sprinkled in it or its dependencies there are a number of :mvn-modules as described earlier. During preparation, the dependencies, exclusions, and managed-dependencies are collected at the level of the top-level system. Then, before any lisp or other files are loaded a single call is made to compute the combined maven dependencies and they are resolved and added to the classpath. Finally the rest of the load goes as it usually does, although the :mvn-module components compile and loads are no-ops since their work has already been done up-front. Thanks, Alan On Fri, Apr 7, 2017 at 5:48 PM, Robert Goldman <rpgoldman@sift.net> wrote:
On 4/7/17 Apr 7 -4:06 PM, Alan Ruttenberg wrote:
On Fri, Apr 7, 2017 at 4:27 PM, Alan Ruttenberg <alanruttenberg@gmail.com <mailto:alanruttenberg@gmail.com>> wrote:
It doesn't address what to do about potentially conflicting maven artifacts loaded by distinct asdf systems.
Here's one approach.
:managed-dependencies are specifications of versions that should take precedence over versions that are in the transitive closure of some dependency.
As each asdf system which has used maven dependencies computes its set of satisfying dependencies, the particular versions chosen become default managed-dependencies for subsequent systems. If two managed dependencies conflict (e.g. I include a version 1.1 as managed dependency in system A, which is loaded. Subsequently system B specifies version 1.2 as a managed dependency, then an error is signaled. Many times this can be fixed by modifying the specification of the managed dependency to be looser, and changing the order of loaded
systems.
I think moving this into the planning phase is probably the right thing. I'm just not familiar enough with asdf's model to whip up a solution. Hopefully asdf's planning runs over all systems that are depended on before loading anything, in which case I think this will be a viable solution, module the case of conflicting managed dependencies, or dependency hell you would arrive at if you were using maven alone.
Yes, ASDF has a "plan, then execute" model of operation, so it won't load anything until inspecting all the systems involved.
On 4/8/17 01:15, Alan Ruttenberg wrote:
Hi Robert,
If you are familiar with the architecture, perhaps you can give me some hints where to intervene. The behavior I want is:
Assume I load a system at top-level and sprinkled in it or its dependencies there are a number of :mvn-modules as described earlier. During preparation, the dependencies, exclusions, and managed-dependencies are collected at the level of the top-level system. Then, before any lisp or other files are loaded a single call is made to compute the combined maven dependencies and they are resolved and added to the classpath. Finally the rest of the load goes as it usually does, although the :mvn-module components compile and loads are no-ops since their work has already been done up-front.
The architecture of ASDF can be likened to that of the [ship of Theseus][1], being replaced as it sails the seas of compatibility; Robert is just the latest in the line of captains who have attempted to navigate its passage. [1]: https://en.wikipedia.org/wiki/Ship_of_Theseus It was suggested at [ELS 2017][2], that while ASDF has a firm abstraction at the upper level in the CLOS abstraction and a recently concertized cross-implementation compatibility layer in UIOP, it may be said to be missing large swathes of reusable abstraction to bridge the two. [2]: urn:"Loading Multiple Versions of an ASDF System in the Same Lisp Image"/"10th European Lisp Symposium Session: Session I: Tools"/"Vsevolod Domkin". I think the first step forward will be recognizing that the current [ABCL-ASDF:RESOLVE][3] machinery works on an "internal state", opaque to ASDF for resolution. What is needed is to somehow explicitly return all the dependencies in a format that the properly overloaded ASDF CLOS machinery can understand. [3]: https://gitlab.common-lisp.net/abcl/abcl/blob/master/contrib/abcl-asdf/abcl-... -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 4/7/17 17:50, Alan Ruttenberg wrote:
I'm using :mvn ASDF components and ran into a problem with having multiple versions of the same library.
The clearest path forward for conflicting dependencies would be for JAVA:ADD-TO-CLASSPATH to take an environment which would contain a new, dynamically added classloader for all java artifacts loaded in scope. This mechanism is used by Apache Tomcat to introduce a separate classloader for each Java Servlet instance. Even though the Tomcat developers presumably thought such a method would allow some degree of security, they failed. Partly because of the need for each servlet to use global resources in-process for efficiency gains (think JDBC), and partly because the security guarantees of the underlying JVM can be trivially broken (as can be seen by [abcl-servlet][1] which boots a swank servlet binding a listening socket on localhost). Symbolic bindings in ennvironments via the [five currently missing methods in ABCL for CtLv2][cltl2-env] could be presumably mimic the scoping of all possible desired classloaders. Some abstraction of this sort will be needed to run on Java 9, so we might as take a stab at supporting both at a profit to ultimate complexity. [cltl2-env]: https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html [1]: https://bitbucket.org/easye/abcl-servlet [java:add-to-classpath]: :has """" 4.1 JAVA 4.1.1 Modifying the JVM CLASSPATH The JAVA:ADD-TO-CLASSPATH generic functions allows one to add the specified pathname or list of pathnames to the current classpath used by ABCL, allowing the dynamic loading of JVM objects: CL-USER > ( add-to-classpath " / path / to / some . jar " ) N.b ADD-TO-CLASSPATH only affects the classloader used by ABCL (the value of the special variable JAVA:*CLASSLOADER*. It has no effect on Java code outside ABCL. """" . -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Sun, Apr 9, 2017 at 6:10 AM, Mark Evenson <evenson@panix.com> wrote:
On 4/7/17 17:50, Alan Ruttenberg wrote:
I'm using :mvn ASDF components and ran into a problem with having multiple versions of the same library.
The clearest path forward for conflicting dependencies would be for JAVA:ADD-TO-CLASSPATH to take an environment which would contain a new, dynamically added classloader for all java artifacts loaded in scope. This mechanism is used by Apache Tomcat to introduce a separate classloader for each Java Servlet instance.
There is OSGI which is able to do this sort of thing, and there is already prototype code in JSS to use it. However OSGI, or multiple classloaders, needs a fair amount of thought to use correctly, because you can easily arrive at situations where compatible classes can't be used because they were loaded into different classloaders. The first step is to enable managing dependencies when it *is* possible to run without conflicts, as with the :mvn-module I showed above. In this case it turns out that the dependencies only *seemed* to conflict. With the appropriate use of Maven there is no conflict in this case. The thing that is different in our environment as compared to the standard maven situation is that we don't have a central pom for an application that gathers all dependencies and resolves them at once. Instead we have separate ASDF systems which are loaded independently and yet may have intertwined dependencies. I can't comment on the Java 9 issues as I've not studied them as yet. Alan
Even though the Tomcat developers presumably thought such a method would allow some degree of security, they failed. Partly because of the need for each servlet to use global resources in-process for efficiency gains (think JDBC), and partly because the security guarantees of the underlying JVM can be trivially broken (as can be seen by [abcl-servlet][1] which boots a swank servlet binding a listening socket on localhost).
Symbolic bindings in ennvironments via the [five currently missing methods in ABCL for CtLv2][cltl2-env] could be presumably mimic the scoping of all possible desired classloaders.
Some abstraction of this sort will be needed to run on Java 9, so we might as take a stab at supporting both at a profit to ultimate complexity.
[cltl2-env]: https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html
[1]: https://bitbucket.org/easye/abcl-servlet
[java:add-to-classpath]: :has """" 4.1 JAVA 4.1.1 Modifying the JVM CLASSPATH The JAVA:ADD-TO-CLASSPATH generic functions allows one to add the specified pathname or list of pathnames to the current classpath used by ABCL, allowing the dynamic loading of JVM objects: CL-USER > ( add-to-classpath " / path / to / some . jar " ) N.b ADD-TO-CLASSPATH only affects the classloader used by ABCL (the value of the special variable JAVA:*CLASSLOADER*. It has no effect on Java code outside ABCL. """" .
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 4/18/17 04:14, Alan Ruttenberg wrote: […]
There is OSGI which is able to do this sort of thing, and there is already prototype code in JSS to use it. However OSGI, or multiple classloaders, needs a fair amount of thought to use correctly, because you can easily arrive at situations where compatible classes can't be used because they were loaded into different classloaders.
The first step is to enable managing dependencies when it *is* possible to run without conflicts, as with the :mvn-module I showed above. In this case it turns out that the dependencies only *seemed* to conflict. With the appropriate use of Maven there is no conflict in this case.
The thing that is different in our environment as compared to the standard maven situation is that we don't have a central pom for an application that gathers all dependencies and resolves them at once. Instead we have separate ASDF systems which are loaded independently and yet may have intertwined dependencies.
I agree that incrementally fixing the current situation along the lines you have prototyped should occur before any dramatic overhaul with segregated classloaders. Currently, I am working towards a [abcl-1.5.0][] next week with the following priorities: 1) stabilizing support for Java6 (dropping support for Java5, which in practice given the absence of an open Java6 VM doesn't exist) 2) Resurrect the ABCL-BUILD system to build ABCL from Lisp by downloading and installing Ant and Maven as needed. 3) Provide a restart for failure to find Maven, that will download and install it in the XDG filesystem. [abcl-1.5.0]: http://abcl.org/trac/milestone/1.5.0 Maybe we can try to squeeze your :MVN-MODULE proposal in as well. I've seen [your code that demonstrates the problem][1], but as I understand it there is no exact code as you need more understanding of the underlying ASDF mechanisms, right? If you could take the time in the next day or so to write up the minimal proposal for your current needs, I will see what I can do to take a stab at its implementation. [1]: https://gitlab.common-lisp.net/abcl/abcl/blob/master/t/resolve-multiple-mave... -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Should have responded to this sooner. The proposed change is to have a single form in the system definition that deals with all maven dependencies - that's the :mvn-module. It has the most important expressivity available in a POM, namely to list all dependencies, exclusions, and overrides (called managed depencies) which let you specify a specific version for a transitively gathered dependency. Having these easily accessible makes the code that asks Maven to resolve the dependencies all at once easy to invoke. That part is implemented. For the case of managing multiple asdf systems that have maven dependency one way to go forward would be to have resolves dependencies in a system added (and overriding) managed dependencies in subsequent :mvn module resolution. This has the benefit that you can coordinate version choices across multiple systems by first loading a system definition that sets managed dependencies that stick across all subsequent mvn resolutions. There's no guarantee that doing so will be adequate - if the collected set of maven dependencies is fundamentally conflicted you are still screwed. But we can't do anything to prevent that. however in my case that wasn't so and so this strategy solved my issue. I think other parts of ABCL have maven dependencies that overlap the ones in LSW. If these load before user code and the versions chosen aren't optimal for downstream use dependencies we might need a mechanism for overriding them (e.g. The resolution during abcl chooses v2.4 but would work with v2.7, but user code might not work with 2.4. To reduce the incidence of such cases we should avoid using version specifications that are restrictive - say only to major version to next major version - and, when possible, defer resolution until needed at runtime. I do that in a couple of cases already. I think in OSGI, and in some code that uses a Java parser library. In those cases the dependencies are not recorded in the system definition. Instead resolve is called only when code that needs is executed. Alan On Tue, Apr 18, 2017 at 1:57 AM Mark Evenson <evenson@panix.com> wrote:
There is OSGI which is able to do this sort of thing, and there is already prototype code in JSS to use it. However OSGI, or multiple classloaders, needs a fair amount of thought to use correctly, because you can easily arrive at situations where compatible classes can't be used because they were loaded into different classloaders.
The first step is to enable managing dependencies when it *is* possible to run without conflicts, as with the :mvn-module I showed above. In this case it turns out that the dependencies only *seemed* to conflict. With the appropriate use of Maven there is no conflict in this case.
The thing that is different in our environment as compared to the standard maven situation is that we don't have a central pom for an application
On 4/18/17 04:14, Alan Ruttenberg wrote: […] that
gathers all dependencies and resolves them at once. Instead we have separate ASDF systems which are loaded independently and yet may have intertwined dependencies.
I agree that incrementally fixing the current situation along the lines you have prototyped should occur before any dramatic overhaul with segregated classloaders.
Currently, I am working towards a [abcl-1.5.0][] next week with the following priorities:
1) stabilizing support for Java6 (dropping support for Java5, which in practice given the absence of an open Java6 VM doesn't exist)
2) Resurrect the ABCL-BUILD system to build ABCL from Lisp by downloading and installing Ant and Maven as needed.
3) Provide a restart for failure to find Maven, that will download and install it in the XDG filesystem.
[abcl-1.5.0]: http://abcl.org/trac/milestone/1.5.0
Maybe we can try to squeeze your :MVN-MODULE proposal in as well. I've seen [your code that demonstrates the problem][1], but as I understand it there is no exact code as you need more understanding of the underlying ASDF mechanisms, right? If you could take the time in the next day or so to write up the minimal proposal for your current needs, I will see what I can do to take a stab at its implementation.
[1]:
https://gitlab.common-lisp.net/abcl/abcl/blob/master/t/resolve-multiple-mave...
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 4/27/17 22:48, Alan Ruttenberg wrote:
Should have responded to this sooner.
The proposed change is to have a single form in the system definition that deals with all maven dependencies - that's the :mvn-module. It has the most important expressivity available in a POM, namely to list all dependencies, exclusions, and overrides (called managed depencies) which let you specify a specific version for a transitively gathered dependency.
After discussion, Alan's implementation of [MVN-MODULE][asdf-mvn-module] has been added to abcl-1.5.0-dev. To use MVN-MODULE, after ABCL-CONTRIB has been loaded, simply declare the DEFSYSTEM dependency on ASDF-MVN-MODULE, e.g. (defsystem foo :defsystem-depends-on (asdf-mvn-module) :components ((:mvn-module maven :dependencies ("net.sourceforge.owlapi/pellet-cli-ignazio1977/2.4.0-ignazio1977" "org.semanticweb.elk/elk-owlapi/0.4.3" "net.sourceforge.owlapi/org.semanticweb.hermit/1.3.8.413" "net.sourceforge.owlapi/owlapi-distribution/4.2.6" "net.sourceforge.owlapi/owlexplanation/2.0.0" "de.sciss/prefuse-core/1.0.1" "de.sciss/prefuse-demos/1.0.1") :managed-dependencies ("org.slf4j/slf4j-api/1.7.21" "net.sourceforge.owlapi:owlapi-distribution:4.2.6") :exclusions ("net.sourceforge.owlapi:owlapi-osgidistribution" "edu.stanford.protege:org.protege.editor.owl"))) [asdf-mvn-module]: https://github.com/armedbear/abcl/commit/b1d3df487f425f8edb55c48107b60c4190b... I consider ASDF-MVN-MODULE to be an interm solution, to eventually be replaced by code which pushes dependency resolution into the ASDF plan phase. But [since integrating ASDF 3.3 into ABCL still has a few kinks to work out][1], abcl-1.5.0 is long overdue, and Alan reports not being able to use ABCL-ASDF's MVN component on loads of large Maven POM dependency graphs without specifying exclusions, we include his working code to ship with ABCL 1.5.0 (any day now…). We will support the ASDF:MVN-MODULE component syntax via the appropriate hooks into the "better" underlying implementation when (if?) it arrives. [1]: https://gitlab.common-lisp.net/mevenson/asdf/issues -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
participants (3)
-
Alan Ruttenberg -
Mark Evenson -
Robert Goldman