Re: SYS:RUN-PROGRAM does not work on pre-Java7 JVMs (was Re: abcl-1.4.0 released)
On 11/18/16 18:25, Scott Burson wrote:
It's not a miracle at all. JVMs have always been backward-compatible with older classfiles. The big Java app I work on for my day job is mostly compiled for 1.8, but ships with libraries compiled for earlier targets. We've never had the slightest trouble as a result. You only need to generate a newer bytecode version if you want to use a new feature (like InvokeDynamic -- wasn't Alessio working on that?).
I now accept Scott's reasoning on this: ORCL will probably have to maintain bytecode compatibility for a long time, otherwise there would be chaos with existing libraries that are available binary-only. In practice, most of the libraries are distributed by Maven are effectively binary-only, as the steps to build from source are not always clear or documented. Plus many commercial vendors distribute applications as jars. When we compile our Java, we do get the following warnings: abcl.compile.java: Compiling 1 source file to /Users/evenson/work/abcl/build/classes warning: [options] bootstrap class path not set in conjunction with -source 1.5 warning: [options] source value 1.5 is obsolete and will be removed in a future release warning: [options] target value 1.5 is obsolete and will be removed in a future release warning: [options] To suppress warnings about obsolete options, use -Xlint:-options. which indicates to me that at some point in the future, JDK-N (where N is greater than 9) will not compile to JDK5 bytecode anymore. Can anyone point to a statement of what ORCL intends here for obsolescence? --- As for the compatiblity problems with SYS:RUN-PROGRAM, my current propose is from <http://abcl.org/trac/ticket/422#comment:1>: 1. Un-deprecate SYS:RUN-SHELL-COMMAND. Use the pre Java 7 APIs as best we can to support invoking programs, which essentially boils down to being able to take a string, attempt to invoke it, and return its stdout as a string. 2. Re-code the SYS:RUN-PROGRAM Java callsite linkages so that ABCL may be compiled on pre-Java7 JDKs. Produce an intelligible error if it is invoked on a pre-Java7 JVM. 3. Create an API to determine runtime JVM version. Wrap UIOP/RUN-PROGRAM invocation in some sort of handler that will "fall back" to using SYS:RUN-SHELL-COMMAND if it has compatible behavior via specified args. This will potentially be fairly ugly code ASDF-side so we might provide a shim that UIOP/RUN-PROGRAM invokes that contains the logic outside the ASDF code base. Any objections to this path forward? -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
A quick note I have some projects in which I use ABCL on that still need java1.6 I was unable to use the new code as well
1. Un-deprecate SYS:RUN-SHELL-COMMAND. Use the pre Java 7 APIs as best we can to support invoking programs, which essentially boils down to being able to take a string, attempt to invoke it, and return its stdout as a string.
2. Re-code the SYS:RUN-PROGRAM Java callsite linkages so that ABCL may be compiled on pre-Java7 JDKs. Produce an intelligible error if it is invoked on a pre-Java7 JVM.
3. Create an API to determine runtime JVM version. Wrap UIOP/RUN-PROGRAM invocation in some sort of handler that will "fall back" to using SYS:RUN-SHELL-COMMAND if it has compatible behavior via specified args. This will potentially be fairly ugly code ASDF-side so we might provide a shim that UIOP/RUN-PROGRAM invokes that contains the logic outside the ASDF code base.
Any objections to this path forward?
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 2016/11/19 12:50, Douglas Miles wrote:
A quick note I have some projects in which I use ABCL on that still need java1.6 I was unable
to use the new code as well
A [provisional patch][1] that restores compilation in JDK6 is now available. [1]: http://abcl.org/trac/attachment/ticket/422/run-program-20161120a.diff -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
After further analysis, the new plan of attack: 1. Continue to deprecate SYS:RUN-SHELL-COMMAND, as the JVM-[56] actually seem to provide all the behavior we need. 2. Fully support the current signature of the abcl-1.5.0-dev SYS:RUN-PROGRAM as extended by Pipping and Olof. The environment clearing/setting will work without alterations. "All that is needed" is to create an org.abcl.util.AsyncIOManager using the non-blocking file descriptor framework in java.jnio (introduced in JVM5) to drain/fill input/output org.armedbear.lisp.Stream instances. Pipping has collected a thorough test suite for the SYS:RUN-PROGRAM behavior, which will be of great use in implementation. The use of SYS:RUN-PROGRAM as invoked by UIOP/RUN-PROGRAM as the implementation interface should mean that ASDF should "just work" when the new code passes Pipping's test suite. TODO: transcribe details of SYS:RUN-PROGAM test suite invocation; implement ASyncIOManager. Such work is intended for an abcl-1.4.1 release by the end of November 2016. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
I don't care for 1.6 compatibility. Personally I'd like to see the compiler generate more modern bytecode (which is loaded more quickly btw because the verification phase is lighter) and I wouldn't mind if it used ASM or other established bytecode generation libraries rather than mucking with bytes by itself. On 20 November 2016 at 10:28, Mark Evenson <evenson@panix.com> wrote:
On 2016/11/19 12:50, Douglas Miles wrote:
A quick note I have some projects in which I use ABCL on that still need java1.6 I was unable
to use the new code as well
A [provisional patch][1] that restores compilation in JDK6 is now available.
[1]: http://abcl.org/trac/attachment/ticket/422/run-program-20161120a.diff
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
I agree. On Mon, Nov 21, 2016 at 11:14 AM, Alessio Stalla <alessiostalla@gmail.com> wrote:
I don't care for 1.6 compatibility.
Personally I'd like to see the compiler generate more modern bytecode (which is loaded more quickly btw because the verification phase is lighter) and I wouldn't mind if it used ASM or other established bytecode generation libraries rather than mucking with bytes by itself.
On 20 November 2016 at 10:28, Mark Evenson <evenson@panix.com> wrote:
On 2016/11/19 12:50, Douglas Miles wrote:
A quick note I have some projects in which I use ABCL on that still need java1.6 I was unable
to use the new code as well
A [provisional patch][1] that restores compilation in JDK6 is now available.
[1]: http://abcl.org/trac/attachment/ticket/422/run-program- 20161120a.diff
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Ideally, it would be nice to compare this design with what ASDF (UIOP) does in RUN-PROGRAM and LAUNCH-PROGRAM, to minimize issues at that API. Elias Pipping is now the person who understands that best. These APIs try to support both invocation through the shell and direct invocation, and separate redirection of input, output, and error output. Best, R Sent from my iPad
On Nov 19, 2016, at 02:04, Mark Evenson <evenson@panix.com> wrote:
On 11/18/16 18:25, Scott Burson wrote: It's not a miracle at all. JVMs have always been backward-compatible with older classfiles. The big Java app I work on for my day job is mostly compiled for 1.8, but ships with libraries compiled for earlier targets. We've never had the slightest trouble as a result. You only need to generate a newer bytecode version if you want to use a new feature (like InvokeDynamic -- wasn't Alessio working on that?).
I now accept Scott's reasoning on this: ORCL will probably have to maintain bytecode compatibility for a long time, otherwise there would be chaos with existing libraries that are available binary-only. In practice, most of the libraries are distributed by Maven are effectively binary-only, as the steps to build from source are not always clear or documented. Plus many commercial vendors distribute applications as jars.
When we compile our Java, we do get the following warnings:
abcl.compile.java: Compiling 1 source file to /Users/evenson/work/abcl/build/classes warning: [options] bootstrap class path not set in conjunction with -source 1.5 warning: [options] source value 1.5 is obsolete and will be removed in a future release warning: [options] target value 1.5 is obsolete and will be removed in a future release warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
which indicates to me that at some point in the future, JDK-N (where N is greater than 9) will not compile to JDK5 bytecode anymore. Can anyone point to a statement of what ORCL intends here for obsolescence?
---
As for the compatiblity problems with SYS:RUN-PROGRAM, my current propose is from <http://abcl.org/trac/ticket/422#comment:1>:
1. Un-deprecate SYS:RUN-SHELL-COMMAND. Use the pre Java 7 APIs as best we can to support invoking programs, which essentially boils down to being able to take a string, attempt to invoke it, and return its stdout as a string.
2. Re-code the SYS:RUN-PROGRAM Java callsite linkages so that ABCL may be compiled on pre-Java7 JDKs. Produce an intelligible error if it is invoked on a pre-Java7 JVM.
3. Create an API to determine runtime JVM version. Wrap UIOP/RUN-PROGRAM invocation in some sort of handler that will "fall back" to using SYS:RUN-SHELL-COMMAND if it has compatible behavior via specified args. This will potentially be fairly ugly code ASDF-side so we might provide a shim that UIOP/RUN-PROGRAM invokes that contains the logic outside the ASDF code base.
Any objections to this path forward?
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
participants (5)
-
Alessio Stalla -
Blake McBride -
Douglas Miles -
Mark Evenson -
Robert P. Goldman