On Fri, Apr 7, 2017 at 4:27 PM, Alan Ruttenberg alanruttenberg@gmail.com wrote:

It doesn't address what to do about potentially conflicting maven artifacts loaded by distinct asdf systems.

Here's one approach.

:managed-dependencies are specifications of versions that should take precedence over versions that are in the transitive closure of some dependency.

As each asdf system which has used maven dependencies computes its set of satisfying dependencies, the particular versions chosen become default managed-dependencies for subsequent systems. If two managed dependencies conflict (e.g. I include a version 1.1 as managed dependency in system A, which is loaded. Subsequently system B specifies version 1.2 as a managed dependency, then an error is signaled. Many times this can be fixed by modifying the specification of the managed dependency to be looser, and changing the order of loaded systems.

I think moving this into the planning phase is probably the right thing. I'm just not familiar enough with asdf's model to whip up a solution. Hopefully asdf's planning runs over all systems that are depended on before loading anything, in which case I think this will be a viable solution, module the case of conflicting managed dependencies, or dependency hell you would arrive at if you were using maven alone.

Alan

Hi Robert,

If you are familiar with the architecture, perhaps you can give me some hints where to intervene. The behavior I want is:

Assume I load a system at top-level and sprinkled in it or its dependencies there are a number of :mvn-modules as described earlier. During preparation, the dependencies, exclusions, and managed-dependencies are collected at the level of the top-level system. Then, before any lisp or other files are loaded a single call is made to compute the combined maven dependencies and they are resolved and added to the classpath. Finally the rest of the load goes as it usually does, although the :mvn-module components compile and loads are no-ops since their work has already been done up-front.

Thanks, Alan

On Fri, Apr 7, 2017 at 5:48 PM, Robert Goldman rpgoldman@sift.net wrote:

On 4/7/17 Apr 7 -4:06 PM, Alan Ruttenberg wrote:

...

On Fri, Apr 7, 2017 at 4:27 PM, Alan Ruttenberg <alanruttenberg@gmail.com mailto:alanruttenberg@gmail.com> wrote:

It doesn't address what to do about potentially conflicting maven
artifacts loaded by distinct asdf systems.

Here's one approach.

:managed-dependencies are specifications of versions that should take precedence over versions that are in the transitive closure of some dependency.

As each asdf system which has used maven dependencies computes its set of satisfying dependencies, the particular versions chosen become default managed-dependencies for subsequent systems. If two managed dependencies conflict (e.g. I include a version 1.1 as managed dependency in system A, which is loaded. Subsequently system B specifies version 1.2 as a managed dependency, then an error is signaled. Many times this can be fixed by modifying the specification of the managed dependency to be looser, and changing the order of loaded

systems.

...

I think moving this into the planning phase is probably the right thing. I'm just not familiar enough with asdf's model to whip up a solution. Hopefully asdf's planning runs over all systems that are depended on before loading anything, in which case I think this will be a viable solution, module the case of conflicting managed dependencies, or dependency hell you would arrive at if you were using maven alone.

Yes, ASDF has a "plan, then execute" model of operation, so it won't load anything until inspecting all the systems involved.

On Sun, Apr 9, 2017 at 6:10 AM, Mark Evenson evenson@panix.com wrote:

On 4/7/17 17:50, Alan Ruttenberg wrote:

...

I'm using :mvn ASDF components and ran into a problem with having

multiple

...

versions of the same library.

The clearest path forward for conflicting dependencies would be for JAVA:ADD-TO-CLASSPATH to take an environment which would contain a new, dynamically added classloader for all java artifacts loaded in scope. This mechanism is used by Apache Tomcat to introduce a separate classloader for each Java Servlet instance.

There is OSGI which is able to do this sort of thing, and there is already prototype code in JSS to use it. However OSGI, or multiple classloaders, needs a fair amount of thought to use correctly, because you can easily arrive at situations where compatible classes can't be used because they were loaded into different classloaders.

The first step is to enable managing dependencies when it *is* possible to run without conflicts, as with the :mvn-module I showed above. In this case it turns out that the dependencies only *seemed* to conflict. With the appropriate use of Maven there is no conflict in this case.

The thing that is different in our environment as compared to the standard maven situation is that we don't have a central pom for an application that gathers all dependencies and resolves them at once. Instead we have separate ASDF systems which are loaded independently and yet may have intertwined dependencies.

I can't comment on the Java 9 issues as I've not studied them as yet.

Alan

Even though the Tomcat developers presumably thought such a method would allow some degree of security, they failed. Partly because of the need for each servlet to use global resources in-process for efficiency gains (think JDBC), and partly because the security guarantees of the underlying JVM can be trivially broken (as can be seen by [abcl-servlet][1] which boots a swank servlet binding a listening socket on localhost).

Symbolic bindings in ennvironments via the [five currently missing methods in ABCL for CtLv2][cltl2-env] could be presumably mimic the scoping of all possible desired classloaders.

Some abstraction of this sort will be needed to run on Java 9, so we might as take a stab at supporting both at a profit to ultimate complexity.

"""" 4.1 JAVA 4.1.1 Modifying the JVM CLASSPATH The JAVA:ADD-TO-CLASSPATH generic functions allows one to add the specified pathname or list of pathnames to the current classpath used by ABCL, allowing the dynamic loading of JVM objects: CL-USER > ( add-to-classpath " / path / to / some . jar " ) N.b ADD-TO-CLASSPATH only affects the classloader used by ABCL (the value of the special variable JAVA:*CLASSLOADER*. It has no effect on Java code outside ABCL. """" .

-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."

Should have responded to this sooner.

The proposed change is to have a single form in the system definition that deals with all maven dependencies - that's the :mvn-module. It has the most important expressivity available in a POM, namely to list all dependencies, exclusions, and overrides (called managed depencies) which let you specify a specific version for a transitively gathered dependency.

Having these easily accessible makes the code that asks Maven to resolve the dependencies all at once easy to invoke.

That part is implemented.

For the case of managing multiple asdf systems that have maven dependency one way to go forward would be to have resolves dependencies in a system added (and overriding) managed dependencies in subsequent :mvn module resolution. This has the benefit that you can coordinate version choices across multiple systems by first loading a system definition that sets managed dependencies that stick across all subsequent mvn resolutions.

There's no guarantee that doing so will be adequate - if the collected set of maven dependencies is fundamentally conflicted you are still screwed. But we can't do anything to prevent that. however in my case that wasn't so and so this strategy solved my issue.

I think other parts of ABCL have maven dependencies that overlap the ones in LSW. If these load before user code and the versions chosen aren't optimal for downstream use dependencies we might need a mechanism for overriding them (e.g. The resolution during abcl chooses v2.4 but would work with v2.7, but user code might not work with 2.4. To reduce the incidence of such cases we should avoid using version specifications that are restrictive - say only to major version to next major version - and, when possible, defer resolution until needed at runtime. I do that in a couple of cases already. I think in OSGI, and in some code that uses a Java parser library. In those cases the dependencies are not recorded in the system definition. Instead resolve is called only when code that needs is executed.

Alan

On Tue, Apr 18, 2017 at 1:57 AM Mark Evenson evenson@panix.com wrote:

On 4/18/17 04:14, Alan Ruttenberg wrote: […]

...

There is OSGI which is able to do this sort of thing, and there is

already

...

prototype code in JSS to use it. However OSGI, or multiple classloaders, needs a fair amount of thought to use correctly, because you can easily arrive at situations where compatible classes can't be used because they were loaded into different classloaders.

The first step is to enable managing dependencies when it *is* possible

to

...

run without conflicts, as with the :mvn-module I showed above. In this

case

...

it turns out that the dependencies only *seemed* to conflict. With the appropriate use of Maven there is no conflict in this case.

The thing that is different in our environment as compared to the

standard

...

maven situation is that we don't have a central pom for an application

that

...

gathers all dependencies and resolves them at once. Instead we have separate ASDF systems which are loaded independently and yet may have intertwined dependencies.

I agree that incrementally fixing the current situation along the lines you have prototyped should occur before any dramatic overhaul with segregated classloaders.

Currently, I am working towards a [abcl-1.5.0][] next week with the following priorities:

1. stabilizing support for Java6 (dropping support for Java5, which in

practice given the absence of an open Java6 VM doesn't exist)

2. Resurrect the ABCL-BUILD system to build ABCL from Lisp by

downloading and installing Ant and Maven as needed.

3. Provide a restart for failure to find Maven, that will download and

install it in the XDG filesystem.

Maybe we can try to squeeze your :MVN-MODULE proposal in as well. I've seen [your code that demonstrates the problem][1], but as I understand it there is no exact code as you need more understanding of the underlying ASDF mechanisms, right? If you could take the time in the next day or so to write up the minimal proposal for your current needs, I will see what I can do to take a stab at its implementation.

[1]:

https://gitlab.common-lisp.net/abcl/abcl/blob/master/t/resolve-multiple-mave...

-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."