OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL: List<Integer> data = Arrays.asList(1, 2, 3, 4, 5); JavaRDD<Integer> distData = sc.parallelize(data); it looks like it should be easy. Heck, I can do that in one line: (#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it: (#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5))) No instance method named copytoArray found for type org.armedbear.lisp.Cons' And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5))) Sigh It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well. I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered. Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious. I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated. @easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
On Jul 24, 2020, at 05:01, Steven Nunez <steve_nunez@yahoo.com> wrote:
[…]
@easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
Sorry to hear that you weren’t able to bootstrap your project in time. I will definitely continue with Ember to wrap Apache Spark as time permits to start crafting the sort of documentation/tutorial that you would have needed a week ago… -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
Lisp lists are not Java List<T> instances. There are some automatic conversions (e.g., from Lisp numbers to Java numbers) but not for lists or hash tables. You could invoke Arrays.asList(...) in Lisp as well, I don't remember the exact syntax to do that as I haven't been using ABCL for years. Granted, it wouldn't be much work to either have Cons implement List<LispObject> or provide a cons-backed List<T extends LispObject> implementation. However, the devil is in the details – how do you convert those LispObject's to the appropriate Java type? Is it Integer, Long, Double, ...? Generics are erased at runtime, ABCL couldn't possibly know, so we'd need another primitive, e.g., (jcoerce-collection collection &key element-type (collection-type "java.util.ArrayList")). But then one would have to know about it in order to use it. ABCL could also do type inference and deduce the type of the elements of the list from how they're used in the code, e.g., from the call to parallelize. But 1) it's a lot of work to implement 2) it doesn't solve all problems, e.g., I guess parallelize is itself generic like <T> List<T> parallelize(List<T> data) or something like that, in that case, you as a user would still have to spell a type name. BTW – it gives me warm fuzzy feelings to read that you see extensible sequences as a feature worth mentioning. I did the porting from SBCL back then but it didn't look like it was used much. FWIW, back then I also integrated CLOS dispatch with the Java class hierarchy – it may have bit rotted, but you could write, e.g., (defmethod foo ((bar (jclass "java.lang.List"))) ...) and it would do the "right thing" wrt Java inheritance. But hey, maybe it's undocumented as well. On Fri, 24 Jul 2020 at 09:55, Mark Evenson <evenson@panix.com> wrote:
On Jul 24, 2020, at 05:01, Steven Nunez <steve_nunez@yahoo.com> wrote:
[…]
@easye, you mentioned your ember project. If you're going to continue
with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
Sorry to hear that you weren’t able to bootstrap your project in time.
I will definitely continue with Ember to wrap Apache Spark as time permits to start crafting the sort of documentation/tutorial that you would have needed a week ago…
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
If you send a smallish example of the code that doesn't work, and a list of the dependencies, I can have a look. I've been dealing with some stuff recently that might make it easier to debug. You definitely can't use CopyToArray that way. You need to work with a java array (jarray-from-list '(1 2 3 4 5)) Maybe: (#"parallelize" *sc* (jarray-from-list '(1 2 3 4 5))) Alan On Thu, Jul 23, 2020 at 11:03 PM Steven Nunez <steve_nunez@yahoo.com> wrote:
OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL:
List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);JavaRDD<Integer> distData = sc.parallelize(data);
it looks like it should be easy. Heck, I can do that in one line:
(#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct
But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it:
(#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5)))
No instance method named copytoArray found for type org.armedbear.lisp.Cons'
And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5)))
*Sigh*
It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well.
I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered.
Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious.
I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated.
@easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
I think what parallelize really needs is a java.util.list. Alessio mentioned some reasons why an automatic conversion is challenging; perhaps a restart is easier? I.e. search for a method of the given name that takes a java.util.list, and if you're giving it an abcl.cons, the restart asks if you want an automatic conversion. Trying such a conversion manually, it seems I need a jlist-from-list, but this doesn't exist in the JAVA package. How do I get a java.util.list from an abcl.cons ? On Saturday, July 25, 2020, 5:20:30 AM GMT+8, Alan Ruttenberg <alanruttenberg@gmail.com> wrote: If you send a smallish example of the code that doesn't work, and a list of the dependencies, I can have a look. I've been dealing with some stuff recently that might make it easier to debug. You definitely can't use CopyToArray that way. You need to work with a java array(jarray-from-list '(1 2 3 4 5))Maybe:(#"parallelize" *sc* (jarray-from-list '(1 2 3 4 5))) Alan On Thu, Jul 23, 2020 at 11:03 PM Steven Nunez <steve_nunez@yahoo.com> wrote: OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL: List<Integer> data = Arrays.asList(1, 2, 3, 4, 5); JavaRDD<Integer> distData = sc.parallelize(data); it looks like it should be easy. Heck, I can do that in one line: (#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it: (#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5))) No instance method named copytoArray found for type org.armedbear.lisp.Cons' And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5))) Sigh It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well. I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered. Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious. I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated. @easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
Try just using a java list. As I understand it, that should work the same way. Since Java.util.List is abstract, you need to choose a concrete class, such as ArrayList. (let ((jlist (jss::new 'arraylist))) (loop for el in '(1 2 3 4 5) do (#"add" jlist el)) (print (jss::jlist-to-list jlist)) jlist) The print statement is to verify that we got what was expected, and to demonstrate jlist-to-list. Alan On Fri, Jul 24, 2020 at 10:21 PM Steven Nunez <steve_nunez@yahoo.com> wrote:
I think what parallelize <https://spark.apache.org/docs/latest/api/java/org/apache/spark/api/java/JavaSparkContext.html#parallelize-java.util.List->really needs is a java.util.list. Alessio mentioned some reasons why an automatic conversion is challenging; perhaps a restart is easier? I.e. search for a method of the given name that takes a java.util.list, and if you're giving it an abcl.cons, the restart asks if you want an automatic conversion.
Trying such a conversion manually, it seems I need a jlist-from-list, but this doesn't exist in the JAVA package. How do I get a java.util.list from an abcl.cons ?
On Saturday, July 25, 2020, 5:20:30 AM GMT+8, Alan Ruttenberg < alanruttenberg@gmail.com> wrote:
If you send a smallish example of the code that doesn't work, and a list of the dependencies, I can have a look. I've been dealing with some stuff recently that might make it easier to debug. You definitely can't use CopyToArray that way. You need to work with a java array (jarray-from-list '(1 2 3 4 5)) Maybe: (#"parallelize" *sc* (jarray-from-list '(1 2 3 4 5)))
Alan
On Thu, Jul 23, 2020 at 11:03 PM Steven Nunez <steve_nunez@yahoo.com> wrote:
OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL:
List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);JavaRDD<Integer> distData = sc.parallelize(data);
it looks like it should be easy. Heck, I can do that in one line:
(#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct
But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it:
(#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5)))
No instance method named copytoArray found for type org.armedbear.lisp.Cons'
And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5)))
*Sigh*
It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well.
I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered.
Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious.
I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated.
@easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
Thanks Alan. What finally worked is: (#"parallelize" *sc* (#"asList" 'Arrays (java:jarray-from-list '(1 2 3 4 5)))) But I can't help but think that going from a list to an array to a list is the long way 'round. Is there a better or more idiomatic way to do this? More generally, I think much of the confusion is that a Lisper will enter into the FFI with a certain set of assumptions that don't hold. An array is a type of list? No Array class? Only specific types of array classes (e.g. array of Int)? I was hoping that this was or could be papered over in a manner similar to CFFI where I barely notice I'm using an external library most of the time. On Saturday, July 25, 2020, 12:35:08 PM GMT+8, Alan Ruttenberg <alanruttenberg@gmail.com> wrote: Try just using a java list. As I understand it, that should work the same way. Since Java.util.List is abstract, you need to choose a concrete class, such as ArrayList. (let ((jlist (jss::new 'arraylist))) (loop for el in '(1 2 3 4 5) do (#"add" jlist el)) (print (jss::jlist-to-list jlist)) jlist) The print statement is to verify that we got what was expected, and to demonstrate jlist-to-list. Alan On Fri, Jul 24, 2020 at 10:21 PM Steven Nunez <steve_nunez@yahoo.com> wrote: I think what parallelize really needs is a java.util.list. Alessio mentioned some reasons why an automatic conversion is challenging; perhaps a restart is easier? I.e. search for a method of the given name that takes a java.util.list, and if you're giving it an abcl.cons, the restart asks if you want an automatic conversion. Trying such a conversion manually, it seems I need a jlist-from-list, but this doesn't exist in the JAVA package. How do I get a java.util.list from an abcl.cons ? On Saturday, July 25, 2020, 5:20:30 AM GMT+8, Alan Ruttenberg <alanruttenberg@gmail.com> wrote: If you send a smallish example of the code that doesn't work, and a list of the dependencies, I can have a look. I've been dealing with some stuff recently that might make it easier to debug. You definitely can't use CopyToArray that way. You need to work with a java array(jarray-from-list '(1 2 3 4 5))Maybe:(#"parallelize" *sc* (jarray-from-list '(1 2 3 4 5))) Alan On Thu, Jul 23, 2020 at 11:03 PM Steven Nunez <steve_nunez@yahoo.com> wrote: OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL: List<Integer> data = Arrays.asList(1, 2, 3, 4, 5); JavaRDD<Integer> distData = sc.parallelize(data); it looks like it should be easy. Heck, I can do that in one line: (#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it: (#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5))) No instance method named copytoArray found for type org.armedbear.lisp.Cons' And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5))) Sigh It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well. I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered. Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious. I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated. @easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
On Sat, Jul 25, 2020 at 5:15 AM Steven Nunez <steve_nunez@yahoo.com> wrote:
Thanks Alan. What finally worked is:
(#"parallelize" *sc* (#"asList" 'Arrays (java:jarray-from-list '(1 2 3 4 5))))
Did you try the code I sent to create the list? Here it is rephrased to be analogous to jarrary-from-list (defun jlist-from-list (list) (let ((jlist (jss::new 'arraylist))) (loop for el in list do (#"add" jlist el)) jlist)) I would expect to be able to write (#"parallelize" *sc* (jlist-from-list '(1 2 3 4 5))) It would be helpful to know if this doesn't work, as it means there's something I need to learn.
But I can't help but think that going from a list to an array to a list is the long way 'round. Is there a better or more idiomatic way to do this?
In this case it's not really going the long way around. #"asList" takes a variable number of arguments - its java method signature is (T... a). JSS doesn't yet know about varargs. The way java implements varags is to actually create a method that takes an *array* of the arguments, and then when calling the method, add code to pack the arguments into an array. That's what you did - pack the arguments into an array, doing what the java compiler would do. See https://stackoverflow.com/questions/21746663/how-does-jvm-implement-the-vara... I'm going to think about how to make varargs work as expected so you could use a more natural syntax (#"parallelize" *sc* (#"asList" 'Arrays 1 2 3 4 5)). But, as I said, the implementation of jlist-from-list should be adequate. If you can verify that then we can add (an optimized version of) it to abcl. More generally, I think much of the confusion is that a Lisper will enter
into the FFI with a certain set of assumptions that don't hold. An array is a type of list?
What's the basis for thinking this? No Array class? Only specific types of array classes (e.g. array of Int)?
Not following. It would be useful for us if you to unpack how someone would come to these conclusions. We could make the documentation better so as to try to avoid the confusion. I was hoping that this was or could be papered over in a manner similar to
CFFI where I barely notice I'm using an external library most of the time.
Well, that's the intent of JSS (I was the original author of JSS, BTW). In this case I think you are effectively arguing that we should coerce lisp lists and arrays passed as java arguments to a java equivalent. That's also something I can think about. The downside of this is that sometimes I *want* to pass a cons. I suppose we could provide both, making coercing be the default and add syntax to escape it, so (#"parallelize" *sc* '(1 2 3 4 5)) and (#"add" jlist (the cons '(1 2 3 4 5)) if I wanted to have an element of jlist be a cons. There's still an issue that there are multiple implementations of java's List and we aren't indicating which one is desired. So we'd have to pick a default, like java.util.ArrayList. We'd have to document that if a different type of list was wanted it needs to be created explicitly, ala jlist-to-list. Similarly for arrays. If one writes: (#"myMethod" ob #(1 2 3 4 5)) should it pass a byte array? an integer array? an array of Objects? Again we'd have to choose a default, presumably what jarray-from list does - The array type is the java type of the first argument. In this case that would be java.lang.Integer. Again I suppose we could add syntax to help. (#"myMethod" ob (the (array byte) #(1 2 3 4 5))) Any JSS users have an opinion? Alan On Saturday, July 25, 2020, 12:35:08 PM GMT+8, Alan Ruttenberg <
alanruttenberg@gmail.com> wrote:
Try just using a java list. As I understand it, that should work the same way. Since Java.util.List is abstract, you need to choose a concrete class, such as ArrayList.
(let ((jlist (jss::new 'arraylist))) (loop for el in '(1 2 3 4 5) do (#"add" jlist el)) (print (jss::jlist-to-list jlist)) jlist)
The print statement is to verify that we got what was expected, and to demonstrate jlist-to-list. Alan
On Fri, Jul 24, 2020 at 10:21 PM Steven Nunez <steve_nunez@yahoo.com> wrote:
I think what parallelize <https://spark.apache.org/docs/latest/api/java/org/apache/spark/api/java/JavaSparkContext.html#parallelize-java.util.List->really needs is a java.util.list. Alessio mentioned some reasons why an automatic conversion is challenging; perhaps a restart is easier? I.e. search for a method of the given name that takes a java.util.list, and if you're giving it an abcl.cons, the restart asks if you want an automatic conversion.
Trying such a conversion manually, it seems I need a jlist-from-list, but this doesn't exist in the JAVA package. How do I get a java.util.list from an abcl.cons ?
On Saturday, July 25, 2020, 5:20:30 AM GMT+8, Alan Ruttenberg < alanruttenberg@gmail.com> wrote:
If you send a smallish example of the code that doesn't work, and a list of the dependencies, I can have a look. I've been dealing with some stuff recently that might make it easier to debug. You definitely can't use CopyToArray that way. You need to work with a java array (jarray-from-list '(1 2 3 4 5)) Maybe: (#"parallelize" *sc* (jarray-from-list '(1 2 3 4 5)))
Alan
On Thu, Jul 23, 2020 at 11:03 PM Steven Nunez <steve_nunez@yahoo.com> wrote:
OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL:
List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);JavaRDD<Integer> distData = sc.parallelize(data);
it looks like it should be easy. Heck, I can do that in one line:
(#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct
But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it:
(#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5)))
No instance method named copytoArray found for type org.armedbear.lisp.Cons'
And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5)))
*Sigh*
It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well.
I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered.
Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious.
I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated.
@easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
The jlist-from-list works fine, thank you. More generally, I think much of the confusion is that a Lisper will enter into the FFI with a certain set of assumptions that don't hold. An array is a type of list? What's the basis for thinking this? In an earlier message you wrote "Since Java.util.List is abstract, you need to choose a concrete class, such as ArrayList." I took this to mean that there was an is-a relationship between Array and List. No Array class? Only specific types of array classes (e.g. array of Int)? Not following. It would be useful for us if you to unpack how someone would come to these conclusions. We could make the documentation better so as to try to avoid the confusion. This came about when I searched for Array classes and a stack overflow discussion seemed to suggest "There are array classes, one per element type used in the program" along with some other information that probably confused me further. What helped was when I realised that java.lang.list is probably like a lisp sequence and although perhaps not a direct superclass, may act like one. I still could be wrong, but it helped me conceptualise things enough to 'solve' the problem. I've put what I've learned so far into an annotated hello-world.lisp on github so that it can be turned into a tutorial when it's finished. It's a work in progress, but by the end of the Spark hello world, I hope that most of the basics will have been demonstrated. The code is almost certainly not yet a good exemplar, and would benefit greatly from a review by someone(s) more knowledgeable about ABCL than I. On Sunday, July 26, 2020, 6:37:27 AM GMT+8, Alan Ruttenberg <alanruttenberg@gmail.com> wrote: On Sat, Jul 25, 2020 at 5:15 AM Steven Nunez <steve_nunez@yahoo.com> wrote: Thanks Alan. What finally worked is: (#"parallelize" *sc* (#"asList" 'Arrays (java:jarray-from-list '(1 2 3 4 5)))) Did you try the code I sent to create the list? Here it is rephrased to be analogous to jarrary-from-list (defun jlist-from-list (list) (let ((jlist (jss::new 'arraylist))) (loop for el in list do (#"add" jlist el)) jlist)) I would expect to be able to write (#"parallelize" *sc* (jlist-from-list '(1 2 3 4 5))) It would be helpful to know if this doesn't work, as it means there's something I need to learn. But I can't help but think that going from a list to an array to a list is the long way 'round. Is there a better or more idiomatic way to do this? In this case it's not really going the long way around. #"asList" takes a variable number of arguments - its java method signature is (T... a). JSS doesn't yet know about varargs. The way java implements varags is to actually create a method that takes an array of the arguments, and then when calling the method, add code to pack the arguments into an array. That's what you did - pack the arguments into an array, doing what the java compiler would do. See https://stackoverflow.com/questions/21746663/how-does-jvm-implement-the-vara... I'm going to think about how to make varargs work as expected so you could use a more natural syntax (#"parallelize" *sc* (#"asList" 'Arrays 1 2 3 4 5)). But, as I said, the implementation of jlist-from-list should be adequate. If you can verify that then we can add (an optimized version of) it to abcl. More generally, I think much of the confusion is that a Lisper will enter into the FFI with a certain set of assumptions that don't hold. An array is a type of list? What's the basis for thinking this? No Array class? Only specific types of array classes (e.g. array of Int)? Not following. It would be useful for us if you to unpack how someone would come to these conclusions. We could make the documentation better so as to try to avoid the confusion. I was hoping that this was or could be papered over in a manner similar to CFFI where I barely notice I'm using an external library most of the time. Well, that's the intent of JSS (I was the original author of JSS, BTW). In this case I think you are effectively arguing that we should coerce lisp lists and arrays passed as java arguments to a java equivalent. That's also something I can think about. The downside of this is that sometimes I *want* to pass a cons. I suppose we could provide both, making coercing be the default and add syntax to escape it, so (#"parallelize" *sc* '(1 2 3 4 5))and(#"add" jlist (the cons '(1 2 3 4 5)) if I wanted to have an element of jlist be a cons. There's still an issue that there are multiple implementations of java's List and we aren't indicating which one is desired. So we'd have to pick a default, like java.util.ArrayList. We'd have to document that if a different type of list was wanted it needs to be created explicitly, ala jlist-to-list. Similarly for arrays. If one writes: (#"myMethod" ob #(1 2 3 4 5)) should it pass a byte array? an integer array? an array of Objects? Again we'd have to choose a default, presumably what jarray-from list does - The array type is the java type of the first argument. In this case that would be java.lang.Integer. Again I suppose we could add syntax to help. (#"myMethod" ob (the (array byte) #(1 2 3 4 5))) Any JSS users have an opinion? Alan On Saturday, July 25, 2020, 12:35:08 PM GMT+8, Alan Ruttenberg <alanruttenberg@gmail.com> wrote: Try just using a java list. As I understand it, that should work the same way. Since Java.util.List is abstract, you need to choose a concrete class, such as ArrayList. (let ((jlist (jss::new 'arraylist))) (loop for el in '(1 2 3 4 5) do (#"add" jlist el)) (print (jss::jlist-to-list jlist)) jlist) The print statement is to verify that we got what was expected, and to demonstrate jlist-to-list. Alan On Fri, Jul 24, 2020 at 10:21 PM Steven Nunez <steve_nunez@yahoo.com> wrote: I think what parallelize really needs is a java.util.list. Alessio mentioned some reasons why an automatic conversion is challenging; perhaps a restart is easier? I.e. search for a method of the given name that takes a java.util.list, and if you're giving it an abcl.cons, the restart asks if you want an automatic conversion. Trying such a conversion manually, it seems I need a jlist-from-list, but this doesn't exist in the JAVA package. How do I get a java.util.list from an abcl.cons ? On Saturday, July 25, 2020, 5:20:30 AM GMT+8, Alan Ruttenberg <alanruttenberg@gmail.com> wrote: If you send a smallish example of the code that doesn't work, and a list of the dependencies, I can have a look. I've been dealing with some stuff recently that might make it easier to debug. You definitely can't use CopyToArray that way. You need to work with a java array(jarray-from-list '(1 2 3 4 5))Maybe:(#"parallelize" *sc* (jarray-from-list '(1 2 3 4 5))) Alan On Thu, Jul 23, 2020 at 11:03 PM Steven Nunez <steve_nunez@yahoo.com> wrote: OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL: List<Integer> data = Arrays.asList(1, 2, 3, 4, 5); JavaRDD<Integer> distData = sc.parallelize(data); it looks like it should be easy. Heck, I can do that in one line: (#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it: (#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5))) No instance method named copytoArray found for type org.armedbear.lisp.Cons' And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5))) Sigh It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well. I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered. Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious. I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated. @easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
Arrays are primitive sequences of fixed size and element type. They are implemented in the JVM itself (presumably in C++). Each array has a (synthetic) array class represented in Java as <type>[].class, e.g., int[].class or String[].class or Object[][].class. Lists are higher-level data structures, implemented in Java. They're potentially heterogeneous in element type, have variable size, and several implementations with different characteristics, particularly wrt. performance and concurrency. ArrayList is one such implementation: a List backed by a primitive array. Then there's LinkedList, etc. On Mon, 27 Jul 2020 at 06:14, Steven Nunez <steve_nunez@yahoo.com> wrote:
The jlist-from-list works fine, thank you.
More generally, I think much of the confusion is that a Lisper will enter into the FFI with a certain set of assumptions that don't hold. An array is a type of list?
What's the basis for thinking this?
In an earlier message you wrote "*Since Java.util.List is abstract, you need to choose a concrete class, such as ArrayList.*" I took this to mean that there was an is-a relationship between Array and List.
No Array class? Only specific types of array classes (e.g. array of Int)?
Not following. It would be useful for us if you to unpack how someone would come to these conclusions. We could make the documentation better so as to try to avoid the confusion.
This came about when I searched for Array classes and a stack overflow discussion <https://stackoverflow.com/questions/8546500/why-isnt-there-a-java-lang-array-class-if-a-java-array-is-an-object-shouldnt> seemed to suggest "There *are* array classes, one per element type used in the program" along with some other information that probably confused me further.
What helped was when I realised that java.lang.list is probably like a lisp *sequence *and although perhaps not a direct superclass, may act like one. I still could be wrong, but it helped me conceptualise things enough to 'solve' the problem.
I've put what I've learned so far into an annotated hello-world.lisp <https://github.com/Symbolics/Spark.cl/blob/master/hello-world.lisp> on github so that it can be turned into a tutorial when it's finished. It's a work in progress, but by the end of the Spark hello world, I hope that most of the basics will have been demonstrated. The code is almost certainly not yet a good exemplar, and would benefit greatly from a review by someone(s) more knowledgeable about ABCL than I.
On Sunday, July 26, 2020, 6:37:27 AM GMT+8, Alan Ruttenberg < alanruttenberg@gmail.com> wrote:
On Sat, Jul 25, 2020 at 5:15 AM Steven Nunez <steve_nunez@yahoo.com> wrote:
Thanks Alan. What finally worked is:
(#"parallelize" *sc* (#"asList" 'Arrays (java:jarray-from-list '(1 2 3 4 5))))
Did you try the code I sent to create the list? Here it is rephrased to be analogous to jarrary-from-list
(defun jlist-from-list (list) (let ((jlist (jss::new 'arraylist))) (loop for el in list do (#"add" jlist el)) jlist))
I would expect to be able to write
(#"parallelize" *sc* (jlist-from-list '(1 2 3 4 5)))
It would be helpful to know if this doesn't work, as it means there's something I need to learn.
But I can't help but think that going from a list to an array to a list is the long way 'round. Is there a better or more idiomatic way to do this?
In this case it's not really going the long way around. #"asList" takes a variable number of arguments - its java method signature is (T... a). JSS doesn't yet know about varargs. The way java implements varags is to actually create a method that takes an *array* of the arguments, and then when calling the method, add code to pack the arguments into an array. That's what you did - pack the arguments into an array, doing what the java compiler would do. See https://stackoverflow.com/questions/21746663/how-does-jvm-implement-the-vara...
I'm going to think about how to make varargs work as expected so you could use a more natural syntax (#"parallelize" *sc* (#"asList" 'Arrays 1 2 3 4 5)).
But, as I said, the implementation of jlist-from-list should be adequate. If you can verify that then we can add (an optimized version of) it to abcl.
More generally, I think much of the confusion is that a Lisper will enter into the FFI with a certain set of assumptions that don't hold. An array is a type of list?
What's the basis for thinking this?
No Array class? Only specific types of array classes (e.g. array of Int)?
Not following. It would be useful for us if you to unpack how someone would come to these conclusions. We could make the documentation better so as to try to avoid the confusion.
I was hoping that this was or could be papered over in a manner similar to CFFI where I barely notice I'm using an external library most of the time.
Well, that's the intent of JSS (I was the original author of JSS, BTW). In this case I think you are effectively arguing that we should coerce lisp lists and arrays passed as java arguments to a java equivalent. That's also something I can think about. The downside of this is that sometimes I *want* to pass a cons. I suppose we could provide both, making coercing be the default and add syntax to escape it, so (#"parallelize" *sc* '(1 2 3 4 5)) and (#"add" jlist (the cons '(1 2 3 4 5)) if I wanted to have an element of jlist be a cons.
There's still an issue that there are multiple implementations of java's List and we aren't indicating which one is desired. So we'd have to pick a default, like java.util.ArrayList. We'd have to document that if a different type of list was wanted it needs to be created explicitly, ala jlist-to-list. Similarly for arrays. If one writes: (#"myMethod" ob #(1 2 3 4 5)) should it pass a byte array? an integer array? an array of Objects? Again we'd have to choose a default, presumably what jarray-from list does - The array type is the java type of the first argument. In this case that would be java.lang.Integer. Again I suppose we could add syntax to help. (#"myMethod" ob (the (array byte) #(1 2 3 4 5)))
Any JSS users have an opinion?
Alan
On Saturday, July 25, 2020, 12:35:08 PM GMT+8, Alan Ruttenberg < alanruttenberg@gmail.com> wrote:
Try just using a java list. As I understand it, that should work the same way. Since Java.util.List is abstract, you need to choose a concrete class, such as ArrayList.
(let ((jlist (jss::new 'arraylist))) (loop for el in '(1 2 3 4 5) do (#"add" jlist el)) (print (jss::jlist-to-list jlist)) jlist)
The print statement is to verify that we got what was expected, and to demonstrate jlist-to-list. Alan
On Fri, Jul 24, 2020 at 10:21 PM Steven Nunez <steve_nunez@yahoo.com> wrote:
I think what parallelize <https://spark.apache.org/docs/latest/api/java/org/apache/spark/api/java/JavaSparkContext.html#parallelize-java.util.List->really needs is a java.util.list. Alessio mentioned some reasons why an automatic conversion is challenging; perhaps a restart is easier? I.e. search for a method of the given name that takes a java.util.list, and if you're giving it an abcl.cons, the restart asks if you want an automatic conversion.
Trying such a conversion manually, it seems I need a jlist-from-list, but this doesn't exist in the JAVA package. How do I get a java.util.list from an abcl.cons ?
On Saturday, July 25, 2020, 5:20:30 AM GMT+8, Alan Ruttenberg < alanruttenberg@gmail.com> wrote:
If you send a smallish example of the code that doesn't work, and a list of the dependencies, I can have a look. I've been dealing with some stuff recently that might make it easier to debug. You definitely can't use CopyToArray that way. You need to work with a java array (jarray-from-list '(1 2 3 4 5)) Maybe: (#"parallelize" *sc* (jarray-from-list '(1 2 3 4 5)))
Alan
On Thu, Jul 23, 2020 at 11:03 PM Steven Nunez <steve_nunez@yahoo.com> wrote:
OK, I'm on lines 4 and 5 of 'hello world' and ran into yet another brick wall. Trying to convert the following two lines into ABCL:
List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);JavaRDD<Integer> distData = sc.parallelize(data);
it looks like it should be easy. Heck, I can do that in one line:
(#"parallelize" *sc* '(1 2 3 4 5)) ; *sc* defined yesterday and known to be correct
But no, it claims "no applicable method named parallelize found on JavaSparkContext" (but there is!). Reading through section 3.1.1 of the documentation, it appears that this is probably because '(1 2 3...) is a LispObject and not a Java object (why no automatic conversion?). Let's try to convert it:
(#"parallelize" *sc* (#"copytoArray" '(1 2 3 4 5)))
No instance method named copytoArray found for type org.armedbear.lisp.Cons'
And the same with using an array, e.g. (#"parallelize" *sc* (#"copytoArray" #(1 2 3 4 5)))
*Sigh*
It's been a week and my intention was to have a working prototype by now and present ABCL as a viable alternative to use in a project. I haven't got past line 5 in 'hello world'. This doesn't bode well.
I've been reading about ABCL for years, and it's impressive. Full MOP, extensible sequences, nearly 100% ANSI compliance, and the ability to deploy on the JVM are major achievements. However, as a not-inexperienced Lisp programmer, I find the barrier to entry remarkably high and the documentation and examples sparse and insufficient to surmount the hurdles I encountered.
Please take these comments in the way they are intended: constructive feedback from someone who is a fan of the project and would love to be able to use it. It's nearly impossible to get Lisp introduced into enterprise environments, and ABCL provides a wedge into those types of projects, ticking the boxes on deployment and ability to work with legacy Java code. Perhaps it makes more sense to someone approaching Lisp from the Java side, but coming from the Lisp side to Java, there's a high barrier to entry. I know that no volunteer wants to write documentation, but more and clearer docs are sorely needed here. This is probably not news, but sometimes it helps to be reminded of the obvious.
I hate giving up, so this will be a personal background project in the hopes that at the next opportunity things will have improved to the point where we can consider introducing ABCL, so if anyone has any pointers, generally (though I think I would have found any docs or examples (lsw2) by now) and explaining this problem in particular, it would be greatly appreciated.
@easye, you mentioned your ember project. If you're going to continue with that, please message me. A Spark wrapper would be useful, serve as a good exemplar for using ABCL to wrap a large library and, with a companion tutorial, help others overcome the kind of obstacles I've encountered. I'd be happy to contribute.
participants (4)
-
Alan Ruttenberg -
Alessio Stalla -
Mark Evenson -
Steven Nunez