armedbear-devel

armedbear-devel@common-lisp.net

2 participants
1202 discussions

Start a nNew thread

Test
by Steven Nunez 08 Aug '20

08 Aug '20

Testing. I sent out a message last night but haven't seen it yet; neither in my mailbox nor the archives.

1 0

Am I at a dead-end?
by Steven Nunez 07 Aug '20

07 Aug '20

I've been looking at a few other ways to accomplish wrapping Spark, and none look good. I could generate Java functions, which seem to be inherently serialisable, but that defeats the point of using ABCL. I also considered an approach using Java to call into ABCL, but then I've got to ensure that the ABCL functions are distributed to each node, defeating the point of using Spark. @Alessio Stalla, your most recent github commit suggests that things might have moved along to the point that function and closure serialisation is working. Are things stable enough to give it a try? Not this week, but perhaps on Monday? I'm happy to debug anything that I turn up. There's a new project on the horizon and I'd like to use the Java compatibility angle to try and get Lisp into it. If I'm lucky, I may not even have to mention lisp, depending on how/if the contract terms work out.

2 3

Regarding Steps Towards ABCL STEP Functionality
by loveleague 03 Aug '20

03 Aug '20

Hello, While reading through the release manual, I came across the realization -- from section 1.1.2 -- that ABCL still needs an implementation of the STEP function. I would like to look into the details of the hurdles one might need to overcome to bring STEP functionality to ABCL. I imagine there may be some additional challenges from dealing with the JVM, but I'm not well versed as of yet on the topic of debuggers or JVM internals. If it would be possible to share some details on where one would get started in trying to approach this, that would be appreciated. - Luke

4 3

run-program issues with OpenJDK 11
by Eric Timmons 30 Jul '20

30 Jul '20

It appears that java.lang.UNIXProcess was removed in OpenJDK 9 or thereabouts, assuming I'm interpreting https://github.com/openjdk/jdk/commit/aa6b19f38ee4dc69cac664d1211bff0e69b31… correctly (/definitely/ not a Java developer here...). This seems to make run-program unusable when using prebuilt jars on recent JREs. Using the prebuilt 1.7.0 release and OpenJDK 11, I get the errors shown in prebuilt-openjdk11.txt (attached). If compiling using OpenJDK 11, run-program seems usable, but any attempt to get the PID of a process gives the error in from-source-openjdk11.txt (attached). -Eric VM settings: Max. Heap Size (Estimated): 7.79G Using VM: OpenJDK 64-Bit Server VM Armed Bear Common Lisp 1.7.0 Java 11.0.7 Oracle Corporation OpenJDK 64-Bit Server VM Low-level initialization completed in 0.198 seconds. Startup completed in 1.107 seconds. Type ":help" for a list of available commands. CL-USER(1): (sys:run-program "ls" nil) Error loading jar:file:/home/abcl/work/abcl/dist/abcl.jar!/org/armedbear/lisp/run-program.abcl at line 278 (offset 19142) #<THREAD "interpreter" {5C1AE0C4}>: Debugger invoked on condition of type ERROR Class not found: java.lang.UNIXProcess Restarts: 0: TOP-LEVEL Return to top level. [1] SYS(2): 0 java.lang.ExceptionInInitializerError at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/java.lang.Class.newInstance(Class.java:584) at org.armedbear.lisp.FaslClassLoader.loadFunction(FaslClassLoader.java:130) at org.armedbear.lisp.FaslClassLoader$pf_get_fasl_function.execute(FaslClassLoader.java:165) at org.armedbear.lisp.LispThread.execute(LispThread.java:832) at org.armedbear.lisp.Lisp.evalCall(Lisp.java:582) at org.armedbear.lisp.Lisp.eval(Lisp.java:540) at org.armedbear.lisp.Lisp.evalCall(Lisp.java:577) at org.armedbear.lisp.Lisp.eval(Lisp.java:540) at org.armedbear.lisp.Lisp.progn(Lisp.java:709) at org.armedbear.lisp.SpecialOperators$sf_progn.execute(SpecialOperators.java:273) at org.armedbear.lisp.Lisp.eval(Lisp.java:530) at org.armedbear.lisp.Load.faslLoadStream(Load.java:667) at org.armedbear.lisp.Load$init_fasl.execute(Load.java:457) at org.armedbear.lisp.LispThread.execute(LispThread.java:832) at org.armedbear.lisp.Lisp.evalCall(Lisp.java:582) at org.armedbear.lisp.Lisp.eval(Lisp.java:540) at org.armedbear.lisp.Load.loadStream(Load.java:629) at org.armedbear.lisp.Load.loadFileFromStream(Load.java:597) at org.armedbear.lisp.Load.loadFileFromStream(Load.java:477) at org.armedbear.lisp.Load.loadSystemFile(Load.java:375) at org.armedbear.lisp.Load.loadSystemFile(Load.java:255) at org.armedbear.lisp.Autoload.effectiveLoad(Autoload.java:110) at org.armedbear.lisp.Autoload.load(Autoload.java:147) at org.armedbear.lisp.Lisp.eval(Lisp.java:537) at org.armedbear.lisp.Primitives$pf__eval.execute(Primitives.java:345) at org.armedbear.lisp.LispThread.execute(LispThread.java:814) at org.armedbear.lisp.Lisp.evalCall(Lisp.java:575) at org.armedbear.lisp.Lisp.eval(Lisp.java:540) at org.armedbear.lisp.Lisp.progn(Lisp.java:709) at org.armedbear.lisp.Primitives$sf_block.execute(Primitives.java:3743) at org.armedbear.lisp.Lisp.eval(Lisp.java:530) at org.armedbear.lisp.Lisp.progn(Lisp.java:709) at org.armedbear.lisp.Closure.execute(Closure.java:220) at org.armedbear.lisp.Closure.execute(Closure.java:148) at org.armedbear.lisp.LispThread.execute(LispThread.java:814) at org.armedbear.lisp.Lisp$1.execute(Lisp.java:285) at org.armedbear.lisp.Symbol.execute(Symbol.java:803) at org.armedbear.lisp.LispThread.execute(LispThread.java:814) at org.armedbear.lisp.top_level_47.execute(top-level.lisp:407) at org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:89) at org.armedbear.lisp.Symbol.execute(Symbol.java:793) at org.armedbear.lisp.LispThread.execute(LispThread.java:798) at org.armedbear.lisp.top_level_48.execute(top-level.lisp:415) at org.armedbear.lisp.LispThread.execute(LispThread.java:798) at org.armedbear.lisp.Interpreter.run(Interpreter.java:361) at org.armedbear.lisp.Main$1.run(Main.java:48) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.armedbear.lisp.Go Error loading jar:file:/home/abcl/work/abcl/dist/abcl.jar!/org/armedbear/lisp/run-program.abcl at line 278 (offset 19142) #<THREAD "interpreter" {5C1AE0C4}>: Debugger invoked on condition of type ERROR Compiled function can't be loaded: org.armedbear.lisp.run_program_60 from org.armedbear.lisp.Pathname@49bd2fd0 Restarts: 0: TOP-LEVEL Return to top level. [1] SYS(3): 0 CL-USER(4): (exit) VM settings: Max. Heap Size (Estimated): 7.79G Using VM: OpenJDK 64-Bit Server VM Armed Bear Common Lisp 1.7.1-dev Java 11.0.7 Oracle Corporation OpenJDK 64-Bit Server VM Low-level initialization completed in 0.205 seconds. Startup completed in 1.224 seconds. Type ":help" for a list of available commands. CL-USER(1): (sys:run-program "ls" nil :wait nil) #S(SYSTEM:PROCESS :JPROCESS #<java.lang.ProcessImpl Process[pid=125, exitValue=0] {1E464845}> :%INPUT #S(SYSTEM::SYSTEM-STREAM) :%OUTPUT #S(SYSTEM::SYSTEM-STREAM) :%ERROR #S(SYSTEM::SYSTEM-STREAM)) CL-USER(2): (sys:process-pid *) #<THREAD "interpreter" {7278E1D6}>: Debugger invoked on condition of type ERROR Class not found: java.lang.UNIXProcess Restarts: 0: TOP-LEVEL Return to top level. [1] CL-USER(3): 0 CL-USER(4): (exit)

3 8

java:jinterface-implementation: multiple interfaces?
by Steven Nunez 30 Jul '20

30 Jul '20

Is there a way to implement multiple interfaces on a single Java proxy? This code almost works: (java:jinterface-implementation "org.apache.spark.api.java.function.Function" "call" (lambda (s) (length s))) except that the proxy also needs to implement Serializable. The jproxy code in java.lisp seems to suggest that multiple implementations are allowed: (defgeneric jmake-proxy (interface implementation &optional lisp-this) (:documentation "Returns a proxy Java object implementing the provided interface(s)... but I can't see adding multiple implementations in the code. I see there's a few jmake-proxy methods in there though: are there any documentation or examples for their usage? Lsw2 doesn't use this at all and I can't find any other good examples of using ABCL. Multiple interfaces from the jinterface-implementation function would be ideal, as the above code could then be wrapped with a macro to produce a 'spark-lambda' and be used nearly like the regular ABCL lambda.

3 12

JDK8 Style Lambda Functions and ABCL
by Steven Nunez 30 Jul '20

30 Jul '20

Greetings all, I'm trying to convert the following Java code in the "Basics" section of the Spark Programming Guide to ABCL: JavaRDD<Integer> lineLengths = lines.map(s -> s.length()); I know that "s -> s.length" is a JDK8 style lambda function with one parameter, returning the result of calling length() on 's'. What I'd like to be able to do is write: (let ((line-lengths (#"map" *lines* (lambda (s) (#"length" s))))) but this isn't getting me anywhere, with Java saying there is no applicable method 'map' on *lines* (an instance of JavaRDD). There is such a method (if it matters, it is inherited by JavaRDD from interface JavaRDDLike). Investigating that map method a bit further, it seems to want an org.apache.spark.api.java.function. Here's a clip from the Spark description: Spark’s API relies heavily on passing functions in the driver program to run on the cluster. In Java, functions are represented by classes implementing the interfaces in the org.apache.spark.api.java.function package. There are two ways to create such functions: - Implement the Function interfaces in your own class, either as an anonymous inner class or a named one, and pass an instance of it to Spark. - Use lambda expressions to concisely define an implementation. I think what I'm getting from the ABCL lambda expression is a java.util.function: SPARK> (describe (lambda (s) #"length" s))#<FUNCTION #<FUNCTION (LAMBDA (S)) {70B82AD0}> {70B82AD0}> is an object of type FUNCTION.The function's lambda list is: (S) I do wonder though how the Java lambda "s -> s.length()" manages to produce the correct result, so this theory may not be correct. The Spark guide goes on to say: While much of this guide uses lambda syntax for conciseness, it is easy to use all the same APIs in long-form. For example, we could have written our code above as follows: JavaRDD<String> lines = sc.textFile("data.txt"); JavaRDD<Integer> lineLengths = lines.map(new Function<String, Integer>() { public Integer call(String s) { return s.length(); } }); int totalLength = lineLengths.reduce(new Function2<Integer, Integer, Integer>() { public Integer call(Integer a, Integer b) { return a + b; } }); Or, if writing the functions inline is unwieldy: class GetLength implements Function<String, Integer> { public Integer call(String s) { return s.length(); } } class Sum implements Function2<Integer, Integer, Integer> { public Integer call(Integer a, Integer b) { return a + b; } } JavaRDD<String> lines = sc.textFile("data.txt"); JavaRDD<Integer> lineLengths = lines.map(new GetLength()); int totalLength = lineLengths.reduce(new Sum()); To me, both of those examples look unwieldy. There was a stack overflow discussion on creating Java classes with ABCL from 9 years ago, and I've read the java interface examples on abcl.org, but both of those techniques look like they will produce code just as unwieldy as the Java syntax above. Is there any way to use a lisp-style lambda syntax to produce a function that will satisfy Spark's requirement to implement the org.apache.spark.api.java.function interface? If not, the obvious route would be some macrology to create a defsparkfun, defsparkfun2, etc. that wrap the ABCL function I want to use with something that implements 'call' in the org.apache.spark.api.java.function interface. I'm hoping there's a better way.

2 3

Third Brick Wall
by Steven Nunez 27 Jul '20

27 Jul '20

OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL: List<Integer> data = Arrays.asList(1, 2, 3, 4, 5); JavaRDD<Integer> distData = sc.parallelize(data); it looks like it should be easy. Heck, I can do that in one line: (#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it: (#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5))) No instance method named copytoArray found for type org.armedbear.lisp.Cons' And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5))) Sigh It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well. I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered. Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious. I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated. @easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.

4 9

Create instance of generic class at runtime, supplying specializer?
by Alan Ruttenberg 21 Jul '20

21 Jul '20

For instance in http://owlcs.github.io/owlapi/apidocs_4/org/semanticweb/owlapi/util/Inferre… It says: Type Parameters:A - the axiom type Question is: how do I supply the axiom type when creating an instance. Thanks ALan

3 5

abcl-1.7.1 released
by Mark Evenson 18 Jul '20

18 Jul '20

With gentle prodding, we have released ABCL 1.7.1, a decidedly minor release correcting a few bugs resulting from the overhaul of arrays specialized on unsigned byte types. The brief list of CHANGES is a available for your perusal. <https://abcl.org/releases/1.7.1/> <https://abcl.org/svn/tags/1.7.1/CHANGES>. "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."

1 0

ABCL-1.7.0 released
by Mark Evenson 05 Jun '20

05 Jun '20

We are pleased to announce the immediate availablity of the [ABCL 1.7.0 release][]. After consuming a steady diet of java.nio.ByteBuffer objects over the past month, the Bear has managed to incorporate the use of these abstractions for arrays specialized on the commonly used unsigned-byte types (or (unsigned-byte 8) (unsigned-byte 16) (unsigned-byte 32)). This replacement of the use arrays of primitive bytes is denoted by the presence of the :NIO keyword in CL:*FEATURES*. With this :NIO overhaul, we have extended our implementation of ANSI Common Lisp [CL:MAKE-ARRAY][] with two additional keywords, viz. :NIO-BUFFER and :NIO-DIRECT. The :NIO-BUFFER keyword argument allows one to construct a vector directly utilizing the contents of an already allocated java.nio.ByteBuffer object. When combined with the ability of JNA to allocate memory on the heap via a malloc() system call, we implemented shareable byte vectors in [CFFI-SYS:MAKE-SHAREABLE-BYTE-VECTOR][]. (let* ((length 16) (byte-buffer (java:jstatic "allocate" "java.nio.ByteBuffer" length))) (make-array length :element-type ’(unsigned-byte 8) :nio-buffer byte-buffer)) When the :NIO-DIRECT keyword argument is called with a non-NIL value, the implementation creates a byte vector with a "directly allocated" java.nio.ByteBuffer object. Such direct buffers typically have somewhat higher allocation and deallocation costs than non-direct buffers. The contents of direct buffers may reside outside of the normal garbage-collected heap, and so their impact upon the memory footprint of an application might not be obvious. It is therefore recommended that direct buffers be allocated primarily for large, long-lived buffers that are subject to the underlying system’s native I/O operations. In general it is best to allocate direct buffers only when they yield a measureable gain in program performance. In the near future, we intend to exploring the performance gains available CL:LOAD by accessing direct buffers memory mapped to our on-disk fasl representation. Our fasls, as zipped archives, currently require a new seek() from the beginning of the fasl for each component they contain. With a memory mapped direct buffer we should be able to simply read from the appropiate byte offset for each component. A complete overview of the accumulated fixes and changes since the previous release may be viewed in [CHANGES][]. [ABCL 1.7.0 release]: https://abcl.org/releases/1.7.0/ [java.nio.ByteBuffer]: https://www.doc.ic.ac.uk/csg-old/java/jdk6docs/api/java/nio/ByteBuffer.html [CFFI-SYS:MAKE-SHAREABLE-BYTE-VECTOR]: https://github.com/cffi/cffi/commit/47136ad9a97c2df98dbcd13a068e14489ced5b03 [CHANGES]: https://abcl.org/svn/tags/1.7.0/CHANGES [CL:MAKE-ARRAY]: https://github.com/armedbear/abcl/commit/6c4b7f3c11b2c8151f4599c9bddf5c0991… -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."

3 3

← Newer
1
...
5
6
7
8
9
10
11
...
121
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

armedbear-devel