I see. So when you mention proxies, are you referring to wrapping java:jnew-runtime-class with some macros? Something like this: (java:jnew-runtime-class "get-length" :interfaces (list "org.apache.spark.api.java.function") :methods `(("call" ,"java.lang.Integer" (,"java.lang.String") (lambda (s) (length s)) :modifiers (:public))) :access-flags '(:public :static :final)) ? I'll give that a try today and see how it goes, but I notice that I've got to specify the RETURN-TYPE, where the JDK8 lambda's do not. Looking at how the well existing lambda works, barring returning java.util.function, I can't help but wonder if a variant of the ABCL lambda that returns an implementation of org.apache.spark.api.java.function and the 'call' method might not be the most elegant route, the one with the most natural syntax.
On Wednesday, July 29, 2020, 4:11:40 PM GMT+8, Alessio Stalla alessiostalla@gmail.com wrote:
Java lambdas are syntax sugar to implement interfaces with a single non-default method. ABCL doesn't translate them automatically and doing so dynamically would be slow (creating a proxy each time is costly) without some advanced optimizations, but it could be done.You could use some functions and macros to hide the complexity of creating the proxy.
On Wed, 29 Jul 2020 at 09:30, Steven Nunez steve_nunez@yahoo.com wrote:
Greetings all, I'm trying to convert the following Java code in the "Basics" section of the Spark Programming Guide to ABCL:
JavaRDD<Integer> lineLengths = lines.map(s -> s.length()); I know that "s -> s.length" is a JDK8 style lambda function with one parameter, returning the result of calling length() on 's'. What I'd like to be able to do is write: (let ((line-lengths (#"map" *lines* (lambda (s) (#"length" s)))))
but this isn't getting me anywhere, with Java saying there is no applicable method 'map' on *lines* (an instance of JavaRDD). There is such a method (if it matters, it is inherited by JavaRDD from interface JavaRDDLike). Investigating that map method a bit further, it seems to want an org.apache.spark.api.java.function. Here's a clip from the Spark description: Spark’s API relies heavily on passing functions in the driver program to run on the cluster. In Java, functions are represented by classes implementing the interfaces in the org.apache.spark.api.java.function package. There are two ways to create such functions: - Implement the Function interfaces in your own class, either as an anonymous inner class or a named one, and pass an instance of it to Spark. - Use lambda expressions to concisely define an implementation.
I think what I'm getting from the ABCL lambda expression is a java.util.function: SPARK> (describe (lambda (s) #"length" s))#<FUNCTION #<FUNCTION (LAMBDA (S)) {70B82AD0}> {70B82AD0}> is an object of type FUNCTION.The function's lambda list is: (S) I do wonder though how the Java lambda "s -> s.length()" manages to produce the correct result, so this theory may not be correct. The Spark guide goes on to say:
While much of this guide uses lambda syntax for conciseness, it is easy to use all the same APIs in long-form. For example, we could have written our code above as follows: JavaRDD<String> lines = sc.textFile("data.txt"); JavaRDD<Integer> lineLengths = lines.map(new Function<String, Integer>() { public Integer call(String s) { return s.length(); } }); int totalLength = lineLengths.reduce(new Function2<Integer, Integer, Integer>() { public Integer call(Integer a, Integer b) { return a + b; } });
Or, if writing the functions inline is unwieldy: class GetLength implements Function<String, Integer> { public Integer call(String s) { return s.length(); } } class Sum implements Function2<Integer, Integer, Integer> { public Integer call(Integer a, Integer b) { return a + b; } } JavaRDD<String> lines = sc.textFile("data.txt"); JavaRDD<Integer> lineLengths = lines.map(new GetLength()); int totalLength = lineLengths.reduce(new Sum());
To me, both of those examples look unwieldy. There was a stack overflow discussion on creating Java classes with ABCL from 9 years ago, and I've read the java interface examples on abcl.org, but both of those techniques look like they will produce code just as unwieldy as the Java syntax above.
Is there any way to use a lisp-style lambda syntax to produce a function that will satisfy Spark's requirement to implement the org.apache.spark.api.java.function interface? If not, the obvious route would be some macrology to create a defsparkfun, defsparkfun2, etc. that wrap the ABCL function I want to use with something that implements 'call' in the org.apache.spark.api.java.function interface. I'm hoping there's a better way.