August 2020 - armedbear-devel - mailman3.common-lisp.net

Setting different class loader
by Steven Nunez 06 Feb '23

06 Feb '23

Greetings all, I have what I think is a problem with the ABCL class loader. I am working with a 'big data' library, Spark, but run into an issue on line 2 of the programming guide example. I am able to load the JARs from Maven with the ASDF system definition: (asdf:defsystem #:spark :description "Wrapper for Spark 3.0" :serial t :defsystem-depends-on (abcl-asdf) :depends-on (#:jss #:javaparser) :components ((:mvn "org.apache.spark/spark-core_2.12" :version "3.0.0") (:file "package") (:file "spark"))) and can create a SparkConf object: (defvar *spark-conf* #1"new SparkConf() .setAppName("abcl-app") .setMaster("local")" ) But when I try to create a 'context' (defvar *sc* (new 'JavaSparkContext *spark-conf*)) I get an error in the initialisation: Java exception 'java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.package$'. There isn't much on this except from the Jenkins guys, who have attempted to put Spark and Spark applications into a CI system. They seem to think that it's related to a call to get a properties file in the package class and on a StackOverflow discussion suggested that "you should make sure that you set the classloader that Spark was loaded through using the Thread.currentThread().setContextClassLoader(myCustomLoader) call". I've verified with (java:dump-classpath) that the JAR is on the ABCL classpath, and the JAR file does contain the spark-version-info.properties file. I've also tried getting the file myself with: (defvar rs #1"Thread.currentThread() .getContextClassLoader() .getResourceAsStream("spark-version-info.properties")" ) which returns nil, so their theory may be correct. Messing around with class loaders is a bit beyond my 20 year old Java knowledge so I thought I'd ask here if anyone has any ideas on how I can load Spark in way to use the default Java class loader. Alternatively it occurs to me to ask why the ABCL class loader isn't able to find the properties file if the JAR is on the classpath and then to correct whatever that problem is. Cheers, Steve

3 6

Challenge with ABCL on a commercial project: Licensing
by Steven Nunez 07 Sep '20

07 Sep '20

Gents, I nearly had ABCL into an upcoming project, until I checked the license. I should have looked at this before I embarked down this path, but for some reason assumed that since ABCL is a relatively young implementation it was using Apache/MIT/BSD or something similar. Here's the opensource clause from the contract: 1.1 OPEN SOURCE SOFTWARE. To the extent that the Company develops orprovides any software to <Customer> under this Agreement, without <Customer’s> priorwritten consent, that <Customer> may grant or withhold in its sole discretion, nosuch deliverable shall incorporate, link to or call upon any Open SourceSoftware. “Open Source Software” means any software that is generally availableto the public in source code form under (i) licenses substantially similar tothose approved by the Open Source Initiative and listed athttp://www.opensource.org/licenses, or (ii) any copyleft license (including aReciprocal License), or (iii) any other license that contains terms thatrequire, whether or not conditionally, disclosure of any source code. This clause, or one very much like it, has been in every contract for the last 10 years or so, or at least since the BusyBox debacle. Most of the time companies have a list of pre-approved licenses, with MIT/Apache/BSD almost always being on it. Note the specific exclusion "any copyleft license" and any license in any form that might cause disclosure of source code. Reading the ABCL FAQ and web page, it seems the intention is to avoid compelling a company from releasing their source code: The use of Armed Bear Common Lisp (ABCL) is covered by the GNU General Public License with Classpath exception, meaning that you can use ABCL in your application without the requirement to open the sources to your application. However there's a few problem with this: - It has the words "GNU General Public License", which rings loud alarm bells with corporate lawyers (and is disqualified by clauses ii/iii anyway) - Whilst the intention is to allow ABCL to be mixed with proprietary code, the IP equivalent of ambulance chasing lawyers may still see it as an opportunity for profit. It's worth reading the BusyBox saga if you're not familiar with it. - It's never been tested in court The last point is worth elaborating on. I once cornered a friendly corporate lawyer at one of our company drinks functions and showed him the LLGPL and asked what he thought. His response put it all in perspective. He said: "It looks fine to me, but it's never been tested in court. Would you take an open source project and deploy it to production and bet the company's livelihood on it, without testing it first?" I guess I can't blame them really. Would you want stick your neck out for something with potentially catastrophic consequences, when there's plenty of alternatives? Lisp has enough of an uphill climb just on the technology front, this is the nail in the coffin. Questions: - Would the current maintainers of ABCL consider relicensing it under a MIT, BSD or Apache license? - If so, do we have a list of all the contributors? - If so, how many are there and are they contactable?

6 9

Test
by Steven Nunez 08 Aug '20

08 Aug '20

Testing. I sent out a message last night but haven't seen it yet; neither in my mailbox nor the archives.

1 0

Am I at a dead-end?
by Steven Nunez 07 Aug '20

07 Aug '20

I've been looking at a few other ways to accomplish wrapping Spark, and none look good. I could generate Java functions, which seem to be inherently serialisable, but that defeats the point of using ABCL. I also considered an approach using Java to call into ABCL, but then I've got to ensure that the ABCL functions are distributed to each node, defeating the point of using Spark. @Alessio Stalla, your most recent github commit suggests that things might have moved along to the point that function and closure serialisation is working. Are things stable enough to give it a try? Not this week, but perhaps on Monday? I'm happy to debug anything that I turn up. There's a new project on the horizon and I'd like to use the Java compatibility angle to try and get Lisp into it. If I'm lucky, I may not even have to mention lisp, depending on how/if the contract terms work out.

2 3

Regarding Steps Towards ABCL STEP Functionality
by loveleague 03 Aug '20

03 Aug '20

Hello, While reading through the release manual, I came across the realization -- from section 1.1.2 -- that ABCL still needs an implementation of the STEP function. I would like to look into the details of the hurdles one might need to overcome to bring STEP functionality to ABCL. I imagine there may be some additional challenges from dealing with the JVM, but I'm not well versed as of yet on the topic of debuggers or JVM internals. If it would be possible to share some details on where one would get started in trying to approach this, that would be appreciated. - Luke

4 3