I've been looking at a few other ways to accomplish wrapping Spark, and none look good. I could generate Java functions, which seem to be inherently serialisable, but that defeats the point of using ABCL. I also considered an approach using Java to call into ABCL, but then I've got to ensure that the ABCL functions are distributed to each node, defeating the point of using Spark. @Alessio Stalla, your most recent github commit suggests that things might have moved along to the point that function and closure serialisation is working. Are things stable enough to give it a try? Not this week, but perhaps on Monday? I'm happy to debug anything that I turn up. There's a new project on the horizon and I'd like to use the Java compatibility angle to try and get Lisp into it. If I'm lucky, I may not even have to mention lisp, depending on how/if the contract terms work out.
If you're ok with building your own abcl from my sources, then yes, you can give it a try as early as you want. Some form of closure serialisation works, and it may already be ok for your use case. I couldn't figure how to execute Lisp tests, and I'd like to write a test that spawns a new instance of ABCL to properly verify that serialisation works across different processes.
On Wed, Aug 5, 2020, 09:47 Steven Nunez steve_nunez@yahoo.com wrote:
I've been looking at a few other ways to accomplish wrapping Spark, and none look good. I could generate Java functions, which seem to be inherently serialisable, but that defeats the point of using ABCL. I also considered an approach using Java to call into ABCL, but then I've got to ensure that the ABCL functions are distributed to each node, defeating the point of using Spark.
@Alessio Stalla, your most recent github commit suggests that things might have moved along to the point that function and closure serialisation is working. Are things stable enough to give it a try? Not this week, but perhaps on Monday? I'm happy to debug anything that I turn up. There's a new project on the horizon and I'd like to use the Java compatibility angle to try and get Lisp into it. If I'm lucky, I may not even have to mention lisp, depending on how/if the contract terms work out.
I've opened a pull request: https://github.com/armedbear/abcl/pull/265 You can read some commentary there. I couldn't find a way to run Lisp tests locally so I may have missed something.
Cheers, Alessio
On Wed, 5 Aug 2020 at 10:30, Alessio Stalla alessiostalla@gmail.com wrote:
If you're ok with building your own abcl from my sources, then yes, you can give it a try as early as you want. Some form of closure serialisation works, and it may already be ok for your use case. I couldn't figure how to execute Lisp tests, and I'd like to write a test that spawns a new instance of ABCL to properly verify that serialisation works across different processes.
On Wed, Aug 5, 2020, 09:47 Steven Nunez steve_nunez@yahoo.com wrote:
I've been looking at a few other ways to accomplish wrapping Spark, and none look good. I could generate Java functions, which seem to be inherently serialisable, but that defeats the point of using ABCL. I also considered an approach using Java to call into ABCL, but then I've got to ensure that the ABCL functions are distributed to each node, defeating the point of using Spark.
@Alessio Stalla, your most recent github commit suggests that things might have moved along to the point that function and closure serialisation is working. Are things stable enough to give it a try? Not this week, but perhaps on Monday? I'm happy to debug anything that I turn up. There's a new project on the horizon and I'd like to use the Java compatibility angle to try and get Lisp into it. If I'm lucky, I may not even have to mention lisp, depending on how/if the contract terms work out.
@Nunez Steve steve_nunez@yahoo.com FYI Mark Evenson has merged my pull request, so you may try with a build of ABCL from its official repository.
On Thu, 6 Aug 2020 at 10:52, Alessio Stalla alessiostalla@gmail.com wrote:
I've opened a pull request: https://github.com/armedbear/abcl/pull/265 You can read some commentary there. I couldn't find a way to run Lisp tests locally so I may have missed something.
Cheers, Alessio
On Wed, 5 Aug 2020 at 10:30, Alessio Stalla alessiostalla@gmail.com wrote:
If you're ok with building your own abcl from my sources, then yes, you can give it a try as early as you want. Some form of closure serialisation works, and it may already be ok for your use case. I couldn't figure how to execute Lisp tests, and I'd like to write a test that spawns a new instance of ABCL to properly verify that serialisation works across different processes.
On Wed, Aug 5, 2020, 09:47 Steven Nunez steve_nunez@yahoo.com wrote:
I've been looking at a few other ways to accomplish wrapping Spark, and none look good. I could generate Java functions, which seem to be inherently serialisable, but that defeats the point of using ABCL. I also considered an approach using Java to call into ABCL, but then I've got to ensure that the ABCL functions are distributed to each node, defeating the point of using Spark.
@Alessio Stalla, your most recent github commit suggests that things might have moved along to the point that function and closure serialisation is working. Are things stable enough to give it a try? Not this week, but perhaps on Monday? I'm happy to debug anything that I turn up. There's a new project on the horizon and I'd like to use the Java compatibility angle to try and get Lisp into it. If I'm lucky, I may not even have to mention lisp, depending on how/if the contract terms work out.
armedbear-devel@common-lisp.net
-
Alessio Stalla -
Steven Nunez