Numpy and Common Lisp?
Has anyone taken the Numpy C API and created a CFFI library based on it? I see some attempts to clone it in pure CL, but I would guess that just using it through its API might be easier and more effective.
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL. But Common Lispers are like academics: the "herding cats" applies. Cheers Marco PS I am a Common Lisper AND an academic. You know what I mean... On Mon, Apr 10, 2023 at 9:09 PM Robert Goldman <rpgoldman@sift.info> wrote:
Has anyone taken the Numpy C API and created a CFFI library based on it? I see some attempts to clone it in pure CL, but I would guess that just using it through its API might be easier and more effective.
Yes, that's my concern. I take it as given that the CL community does not have the resources to fully reimplement numpy from scratch. If one grants me that premise, then building a FFI to numpy would be a better investment of hours than cloning it. Arguably investing in py4cl would be better than either. But other than for personal satisfaction, cloning is a dominated choice. I was hoping to hear that someone had gone over the C API with CFFI. That leads me to a follow-up question: am I right that SWIG no longer ships with a back end for emitting CFFI stubs? It seems like it was dropped sometimes between versions 3 and 4. Best, R On 10 Apr 2023, at 16:16, Marco Antoniotti wrote:
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.
But Common Lispers are like academics: the "herding cats" applies.
Cheers
Marco
PS I am a Common Lisper AND an academic. You know what I mean...
On Mon, Apr 10, 2023 at 9:09 PM Robert Goldman <rpgoldman@sift.info> wrote:
Has anyone taken the Numpy C API and created a CFFI library based on it? I see some attempts to clone it in pure CL, but I would guess that just using it through its API might be easier and more effective.
for completeness sake: there's also cffi/c2ffi (which is part of CFFI): it's an automatic binding generator that processes the C header files using Clang as a library. see the CFFI readme for details and examples. -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “He who refuses to do arithmetic is doomed to talk nonsense.” — John McCarthy (1927–2011), father of Lisp
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.
I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
Hi Michael I am all for it. But, as I said, I am an academic (and a cat). Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures. All the best Marco On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote:
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.
I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference
Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
There’s cl-ana, which may be a useful substitute in some cases… or april, possibly. https://www.cliki.net/cl-ana https://www.cliki.net/april If you specifically want numpy, it may be possible to have Common Lisp talking to python.
On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Hi Michael
I am all for it. But, as I said, I am an academic (and a cat).
Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures.
All the best
Marco
On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com <mailto:michael@stray-labs.com>> wrote:
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.
I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference
Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
FYI - there appears to be a library called numcl that was written to cover numpy's functionality. https://github.com/numcl/numclI've yet to try it, but thought I'd pass along the link.Regards,Elliott Johnson -------- Original message --------From: Raymond Wiker <rwiker@gmail.com> Date: 4/11/23 3:53 AM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp? There’s cl-ana, which may be a useful substitute in some cases… or april, possibly.cl-anacliki.netaprilcliki.netIf you specifically want numpy, it may be possible to have Common Lisp talking to python.On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:Hi MichaelI am all for it. But, as I said, I am an academic (and a cat).Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures.All the bestMarcoOn Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote:IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#referenceLooks rather intimidating. Less intimidating though, than doing the FFI dance, though.
There's also the Lisp-Stat ecosystem, if you don't already know about it. Data-frame, array-operations and LLA (Lisp Linear Algebra) cover much of numpy's functionality; at least enough to get significant work done. On Tuesday, April 11, 2023 at 07:45:50 PM GMT+8, Elliott Johnson <elliott@elliottjohnson.net> wrote: FYI - there appears to be a library called numcl that was written to cover numpy's functionality. https://github.com/numcl/numcl I've yet to try it, but thought I'd pass along the link. Regards,Elliott Johnson -------- Original message --------From: Raymond Wiker <rwiker@gmail.com> Date: 4/11/23 3:53 AM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp? There’s cl-ana, which may be a useful substitute in some cases… or april, possibly. | | cl-anacliki.net | | | | | aprilcliki.net | | | If you specifically want numpy, it may be possible to have Common Lisp talking to python. On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote: Hi Michael I am all for it. But, as I said, I am an academic (and a cat). Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures. All the best Marco On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote: IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL. I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
I don't mean to rain on the parade, but the development and maintenance of numpy consumes a level of resources that is simply beyond the capacity of the CL community to muster. The NUMFocus project, a non-profit, supports this and other numerical computation projects (most, but not exclusively python), drawing on substantial amounts of corporate sponsorship. I urge you to cast your eyes on this NumFOCUS sponsors list before thinking that our community could even begin to tackle this task: https://numfocus.org/sponsors On 11 Apr 2023, at 7:14, Steven Nunez wrote:
There's also the Lisp-Stat ecosystem, if you don't already know about it. Data-frame, array-operations and LLA (Lisp Linear Algebra) cover much of numpy's functionality; at least enough to get significant work done.
On Tuesday, April 11, 2023 at 07:45:50 PM GMT+8, Elliott Johnson <elliott@elliottjohnson.net> wrote:
FYI - there appears to be a library called numcl that was written to cover numpy's functionality. https://github.com/numcl/numcl I've yet to try it, but thought I'd pass along the link. Regards,Elliott Johnson
-------- Original message --------From: Raymond Wiker <rwiker@gmail.com> Date: 4/11/23 3:53 AM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp? There’s cl-ana, which may be a useful substitute in some cases… or april, possibly.
| | cl-anacliki.net | |
|
| | aprilcliki.net | |
|
If you specifically want numpy, it may be possible to have Common Lisp talking to python.
On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote: Hi Michael I am all for it. But, as I said, I am an academic (and a cat).
Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures. All the best Marco
On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote:
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.
I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
Robert,I must say that I am a big fan of your work on asdf and in awe of your professional and academic career.I agree with your assessment that numpy and the entirety of NUMFocus would be well outside the scope of the current CL community.In an effort to conserve the momentum of this thread and channel the spirit of my time at Franz Inc, I'd like to emphasize that a lot can be accomplished by a small team with clear goals and roles.I hope that if a such project arises that I can be of assistance.Best regards,ElliottSent from my T-Mobile 5G Device -------- Original message --------From: Robert Goldman <rpgoldman@sift.net> Date: 4/11/23 1:07 PM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp? I don't mean to rain on the parade, but the development and maintenance of numpy consumes a level of resources that is simply beyond the capacity of the CL community to muster. The NUMFocus project, a non-profit, supports this and other numerical computation projects (most, but not exclusively python), drawing on substantial amounts of corporate sponsorship. I urge you to cast your eyes on this NumFOCUS sponsors list before thinking that our community could even begin to tackle this task: https://numfocus.org/sponsors On 11 Apr 2023, at 7:14, Steven Nunez wrote: There's also the Lisp-Stat ecosystem, if you don't already know about it. Data-frame, array-operations and LLA (Lisp Linear Algebra) cover much of numpy's functionality; at least enough to get significant work done. On Tuesday, April 11, 2023 at 07:45:50 PM GMT+8, Elliott Johnson <elliott@elliottjohnson.net> wrote: FYI - there appears to be a library called numcl that was written to cover numpy's functionality. https://github.com/numcl/numcl I've yet to try it, but thought I'd pass along the link. Regards, Elliott Johnson -------- Original message -------- From: Raymond Wiker <rwiker@gmail.com> Date: 4/11/23 3:53 AM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp? There’s cl-ana, which may be a useful substitute in some cases… or april, possibly. cliki.net cl-ana cliki.net april If you specifically want numpy, it may be possible to have Common Lisp talking to python. On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote: Hi Michael I am all for it. But, as I said, I am an academic (and a cat). Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures. All the best Marco On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote: IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL. I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
Dear Elliot, Robert, Daniel (just the last one writing) et al. First of all, let me apologize for being snarky in my previous comments. I am getting older. Of course, we cannot hope to muster anything like NUMFocus, that goes without saying. But, as Daniel suggested, we have different needs in the community, w.r.t., mathematical and numerical issues: two outlooks could be the following. Some of us, Robert, for example if I am not wrong, want to use libraries that are already out there. This has a long history in the community: the f2cl project has been instrumental in this respect, and the Matlisp project did bring many well known Fortran libraries into the fold. The number of "math", "matrix", "statistical", "ML", libraries listed in CLiki is long. Some of us, myself for example, have, at this point, an... aesthetic approach to the matter (given my day job). While I have no expectations about the outcome, I like to make proposals for portable and foundational specifications; hence my stance about "first specify, then code". It is in this spirit that I followed up on my pamphlet "Why you cannot..." (doi:10.5281/zenodo.3759522) with an effort to provide a library implementing the "Language Independent Arithmetic" (LIA) standards ( https://github.com/marcoxa/CDR-LIA-SPEC - preliminary code is at https://gitlab.common-lisp.net/cl-lia/cl-lia ) I would welcome any help on this later effort. The rest may follow... A note on numpy. In general, my feeling is that most of it is already in the guts of Common Lisp. Yet, it will be useful to check its API. As it would be useful to study more recent APIs for math/numerical libraries, most notably, Julia. All the best Marco On Wed, Apr 12, 2023 at 3:22 AM Elliott Johnson <elliott@elliottjohnson.net> wrote:
Robert,
I must say that I am a big fan of your work on asdf and in awe of your professional and academic career.
I agree with your assessment that numpy and the entirety of NUMFocus would be well outside the scope of the current CL community.
In an effort to conserve the momentum of this thread and channel the spirit of my time at Franz Inc, I'd like to emphasize that a lot can be accomplished by a small team with clear goals and roles.
I hope that if a such project arises that I can be of assistance.
Best regards, Elliott
Sent from my T-Mobile 5G Device
-------- Original message -------- From: Robert Goldman <rpgoldman@sift.net> Date: 4/11/23 1:07 PM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp?
I don't mean to rain on the parade, but the development and maintenance of numpy consumes a level of resources that is simply beyond the capacity of the CL community to muster.
The NUMFocus project, a non-profit, supports this and other numerical computation projects (most, but not exclusively python), drawing on substantial amounts of corporate sponsorship.
I urge you to cast your eyes on this NumFOCUS sponsors list before thinking that our community could even begin to tackle this task: https://numfocus.org/sponsors
On 11 Apr 2023, at 7:14, Steven Nunez wrote:
There's also the Lisp-Stat <https://lisp-stat.dev/> ecosystem, if you don't already know about it. Data-frame, array-operations and LLA (Lisp Linear Algebra) cover much of numpy's functionality; at least enough to get significant work done.
On Tuesday, April 11, 2023 at 07:45:50 PM GMT+8, Elliott Johnson < elliott@elliottjohnson.net> wrote:
FYI - there appears to be a library called numcl that was written to cover numpy's functionality.
https://github.com/numcl/numcl
I've yet to try it, but thought I'd pass along the link.
Regards, Elliott Johnson
-------- Original message -------- From: Raymond Wiker <rwiker@gmail.com> Date: 4/11/23 3:53 AM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp?
There’s cl-ana, which may be a useful substitute in some cases… or april, possibly.
cliki.net
cl-ana <https://www.cliki.net/cl-ana> cliki.net
april <https://www.cliki.net/april>
If you specifically want numpy, it may be possible to have Common Lisp talking to python.
On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Hi Michael
I am all for it. But, as I said, I am an academic (and a cat).
Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures.
All the best
Marco
On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote:
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.
I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference
Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
Hi All, I would like to highlight another currently existing project moving towards numpy/pandas in Common Lisp and at least two libraries specifically for data frames. https://github.com/Lisp-Stat/lisp-stat https://github.com/sirherrbatka/vellum https://github.com/40ants/teddy Best, Craig On Wed, Apr 12, 2023 at 8:12 AM Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Dear Elliot, Robert, Daniel (just the last one writing) et al.
First of all, let me apologize for being snarky in my previous comments. I am getting older.
Of course, we cannot hope to muster anything like NUMFocus, that goes without saying. But, as Daniel suggested, we have different needs in the community, w.r.t., mathematical and numerical issues: two outlooks could be the following.
Some of us, Robert, for example if I am not wrong, want to use libraries that are already out there. This has a long history in the community: the f2cl project has been instrumental in this respect, and the Matlisp project did bring many well known Fortran libraries into the fold. The number of "math", "matrix", "statistical", "ML", libraries listed in CLiki is long.
Some of us, myself for example, have, at this point, an... aesthetic approach to the matter (given my day job). While I have no expectations about the outcome, I like to make proposals for portable and foundational specifications; hence my stance about "first specify, then code". It is in this spirit that I followed up on my pamphlet "Why you cannot..." (doi:10.5281/zenodo.3759522) with an effort to provide a library implementing the "Language Independent Arithmetic" (LIA) standards ( https://github.com/marcoxa/CDR-LIA-SPEC - preliminary code is at https://gitlab.common-lisp.net/cl-lia/cl-lia )
I would welcome any help on this later effort. The rest may follow...
A note on numpy. In general, my feeling is that most of it is already in the guts of Common Lisp. Yet, it will be useful to check its API. As it would be useful to study more recent APIs for math/numerical libraries, most notably, Julia.
All the best
Marco
On Wed, Apr 12, 2023 at 3:22 AM Elliott Johnson < elliott@elliottjohnson.net> wrote:
Robert,
I must say that I am a big fan of your work on asdf and in awe of your professional and academic career.
I agree with your assessment that numpy and the entirety of NUMFocus would be well outside the scope of the current CL community.
In an effort to conserve the momentum of this thread and channel the spirit of my time at Franz Inc, I'd like to emphasize that a lot can be accomplished by a small team with clear goals and roles.
I hope that if a such project arises that I can be of assistance.
Best regards, Elliott
Sent from my T-Mobile 5G Device
-------- Original message -------- From: Robert Goldman <rpgoldman@sift.net> Date: 4/11/23 1:07 PM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp?
I don't mean to rain on the parade, but the development and maintenance of numpy consumes a level of resources that is simply beyond the capacity of the CL community to muster.
The NUMFocus project, a non-profit, supports this and other numerical computation projects (most, but not exclusively python), drawing on substantial amounts of corporate sponsorship.
I urge you to cast your eyes on this NumFOCUS sponsors list before thinking that our community could even begin to tackle this task: https://numfocus.org/sponsors
On 11 Apr 2023, at 7:14, Steven Nunez wrote:
There's also the Lisp-Stat <https://lisp-stat.dev/> ecosystem, if you don't already know about it. Data-frame, array-operations and LLA (Lisp Linear Algebra) cover much of numpy's functionality; at least enough to get significant work done.
On Tuesday, April 11, 2023 at 07:45:50 PM GMT+8, Elliott Johnson < elliott@elliottjohnson.net> wrote:
FYI - there appears to be a library called numcl that was written to cover numpy's functionality.
https://github.com/numcl/numcl
I've yet to try it, but thought I'd pass along the link.
Regards, Elliott Johnson
-------- Original message -------- From: Raymond Wiker <rwiker@gmail.com> Date: 4/11/23 3:53 AM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp?
There’s cl-ana, which may be a useful substitute in some cases… or april, possibly.
cliki.net
cl-ana <https://www.cliki.net/cl-ana> cliki.net
april <https://www.cliki.net/april>
If you specifically want numpy, it may be possible to have Common Lisp talking to python.
On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Hi Michael
I am all for it. But, as I said, I am an academic (and a cat).
Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures.
All the best
Marco
On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote:
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.
I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference
Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
Thank you, you are too kind. My thinking was that there was a large investment in numpy that cannot be exploited by lispers. It has been objected that it would be better to have a library that is more lispy. I have made my argument for why that would be difficult to do. I should also add that numpy isn't particularly "pythony," either: the linear algebra computations are done by external libraries with highly efficient low-level implementations. Furthermore, a lot of python is lisp-inspired, so I think it not unreasonable that a python-suited numpy could also be quite lisp-suitable. Of course, if people want to do a capable linear algebra library for and in CL, I have no objections! Either as a community big project or hobbyist hack. Let a thousand flowers bloom! My interest, though, is in *using* such a library, not building it (numerical analysis is not my thing), so I was looking for the most direct path to the most solid capability. On 11 Apr 2023, at 20:18, Elliott Johnson wrote:
Robert,I must say that I am a big fan of your work on asdf and in awe of your professional and academic career.I agree with your assessment that numpy and the entirety of NUMFocus would be well outside the scope of the current CL community.In an effort to conserve the momentum of this thread and channel the spirit of my time at Franz Inc, I'd like to emphasize that a lot can be accomplished by a small team with clear goals and roles.I hope that if a such project arises that I can be of assistance.Best regards,ElliottSent from my T-Mobile 5G Device -------- Original message --------From: Robert Goldman <rpgoldman@sift.net> Date: 4/11/23 1:07 PM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp? I don't mean to rain on the parade, but the development and maintenance of numpy consumes a level of resources that is simply beyond the capacity of the CL community to muster. The NUMFocus project, a non-profit, supports this and other numerical computation projects (most, but not exclusively python), drawing on substantial amounts of corporate sponsorship. I urge you to cast your eyes on this NumFOCUS sponsors list before thinking that our community could even begin to tackle this task: https://numfocus.org/sponsors On 11 Apr 2023, at 7:14, Steven Nunez wrote:
There's also the Lisp-Stat ecosystem, if you don't already know about it. Data-frame, array-operations and LLA (Lisp Linear Algebra) cover much of numpy's functionality; at least enough to get significant work done.
On Tuesday, April 11, 2023 at 07:45:50 PM GMT+8, Elliott Johnson <elliott@elliottjohnson.net> wrote:
FYI - there appears to be a library called numcl that was written to cover numpy's functionality.
https://github.com/numcl/numcl
I've yet to try it, but thought I'd pass along the link.
Regards, Elliott Johnson
-------- Original message -------- From: Raymond Wiker <rwiker@gmail.com> Date: 4/11/23 3:53 AM (GMT-08:00) To: Discussion list for Common Lisp professionals <pro@common-lisp.net> Subject: Re: Numpy and Common Lisp?
There’s cl-ana, which may be a useful substitute in some cases… or april, possibly.
cliki.net
cl-ana
cliki.net
april
If you specifically want numpy, it may be possible to have Common Lisp talking to python.
On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Hi Michael
I am all for it. But, as I said, I am an academic (and a cat).
Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures.
All the best
Marco
On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote:
IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL.
I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference
Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
Isn’t april (somebody mentioned that already) a high quality implementation of such a library? I understand there’s a layer of APL there, which might be putting off some folks :)
On 12 Apr 2023, at 20:28, Robert Goldman <rpgoldman@sift.net> wrote:
Of course, if people want to do a capable linear algebra library for and in CL, I have no objections! Either as a community big project or hobbyist hack. Let a thousand flowers bloom!
Hey Antoni. A lot of Common Lisp actually comes from PL/I :) :) And getting an APL capable 3270 emulator is easily done :) :) :) On Wed, Apr 12, 2023 at 8:37 PM Antoni Grzymała <antoni@grzymala.info> wrote:
Isn’t april (somebody mentioned that already) a high quality implementation of such a library? I understand there’s a layer of APL there, which might be putting off some folks :)
On 12 Apr 2023, at 20:28, Robert Goldman <rpgoldman@sift.net> wrote:
Of course, if people want to do a capable linear algebra library for and in CL, I have no objections! Either as a community big project or hobbyist hack. Let a thousand flowers bloom!
This is, of course, true, but also points a way forward depending on requirements. Some kind of organised sponsorship and curation of libraries into a more or less coherent whole that's fit for purpose would be a start. No matter how small the start. Ravenpack? Franz? There are a few corporate users out there that would probably like an organised numeric ecosystem. On Wednesday, April 12, 2023 at 04:07:38 AM GMT+8, Robert Goldman <rpgoldman@sift.net> wrote: I don't mean to rain on the parade, but the development and maintenance of numpy consumes a level of resources that is simply beyond the capacity of the CL community to muster. The NUMFocus project, a non-profit, supports this and other numerical computation projects (most, but not exclusively python), drawing on substantial amounts of corporate sponsorship. I urge you to cast your eyes on this NumFOCUS sponsors list before thinking that our community could even begin to tackle this task: https://numfocus.org/sponsors On 11 Apr 2023, at 7:14, Steven Nunez wrote: There's also the Lisp-Stat ecosystem, if you don't already know about it. Data-frame, array-operations and LLA (Lisp Linear Algebra) cover much of numpy's functionality; at least enough to get significant work done. On Tuesday, April 11, 2023 at 07:45:50 PM GMT+8, Elliott Johnson <elliott@elliottjohnson.net> wrote: FYI - there appears to be a library called numcl that was written to cover numpy's functionality. https://github.com/numcl/numcl I've yet to try it, but thought I'd pass along the link. Regards,Elliott Johnson -------- Original message --------From: Raymond Wiker <rwiker@gmail.com>Date: 4/11/23 3:53 AM (GMT-08:00)To: Discussion list for Common Lisp professionals <pro@common-lisp.net>Subject: Re: Numpy and Common Lisp? There’s cl-ana, which may be a useful substitute in some cases… or april, possibly. | cliki.net | cl-ana | | | | cliki.net | april | | | If you specifically want numpy, it may be possible to have Common Lisp talking to python. On 11 Apr 2023, at 08:41, Marco Antoniotti <marco.antoniotti@unimib.it> wrote: Hi Michael I am all for it. But, as I said, I am an academic (and a cat). Should we (as in "a bunch of common lispers", most of whom with day jobs) want to do something like that, how would you want to proceed? Note that I have been part of many past failures. All the best Marco On Tue, Apr 11, 2023 at 1:01 AM Michael Bentley <michael@stray-labs.com> wrote: IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL. I agree. Here’s the API specification for NumPy: https://numpy.org/doc/stable/reference/index.html#reference Looks rather intimidating. Less intimidating though, than doing the FFI dance, though.
On Tue, 2023-04-11 at 15:04 -0500, Robert Goldman wrote:
I don't mean to rain on the parade, but the development and maintenance of numpy consumes a level of resources that is simply beyond the capacity of the CL community to muster.
The NUMFocus project, a non-profit, supports this and other numerical computation projects (most, but not exclusively python), drawing on substantial amounts of corporate sponsorship.
I urge you to cast your eyes on this NumFOCUS sponsors list before thinking that our community could even begin to tackle this task: https://numfocus.org/sponsors
These companies are willing and able to fund this project because NumPy is a core part of software products that yield large profits. I keep seeing big dreams about major development in CL and my answer nowadays is "go start a company and write Lisp". Even "a small team with clear goals" is still significant payroll. -- Stelian Ionescu
On 12 Apr 2023, at 7:37, Stelian Ionescu wrote:
On Tue, 2023-04-11 at 15:04 -0500, Robert Goldman wrote:
I don't mean to rain on the parade, but the development and maintenance of numpy consumes a level of resources that is simply beyond the capacity of the CL community to muster.
The NUMFocus project, a non-profit, supports this and other numerical computation projects (most, but not exclusively python), drawing on substantial amounts of corporate sponsorship.
I urge you to cast your eyes on this NumFOCUS sponsors list before thinking that our community could even begin to tackle this task: https://numfocus.org/sponsors
These companies are willing and able to fund this project because NumPy is a core part of software products that yield large profits.
I keep seeing big dreams about major development in CL and my answer nowadays is "go start a company and write Lisp". Even "a small team with clear goals" is still significant payroll.
FWIW, my philosophy is captured by the name of a Python interface library: "burgled-batteries." Rather than build something new to get this capability, I would prefer to hot-wire someone else's ride....
On 12 Apr 2023, at 7:37, Stelian Ionescu wrote: [...]
These companies are willing and able to fund this project because NumPy is a core part of software products that yield large profits.
I keep seeing big dreams about major development in CL and my answer nowadays is "go start a company and write Lisp". Even "a small team with clear goals" is still significant payroll.
FWIW, my philosophy is captured by the name of a Python interface library: "burgled-batteries."
Rather than build something new to get this capability, I would prefer to hot-wire someone else's ride....
I definitely agree that Python and Common Lisp are similar enough in their FFI approach that a C/Fortran library meant as backend for a Python could be wrapper decently well for Common Lisp. That's probably the best approach given the lack of manpower around Common Lisp. -- Stelian Ionescu
"If you specifically want numpy, it may be possible to have Common Lisp talking to python." CL can run python (uiop:run-program "python3 ~/quicklisp/local-projects/clvpy/hello.py" :output t) This isn't 1950 anymore. 1950s biases about what "efficiency" means no longer apply. (Lisp used to be concerned with "development time efficiency" and can be again) pt
Am 11.04.2023 um 14:17 schrieb paul tarvydas <paultarvydas@gmail.com>:
"If you specifically want numpy, it may be possible to have Common Lisp talking to python."
CL can run python
(uiop:run-program "python3 ~/quicklisp/local-projects/clvpy/hello.py" :output t)
This isn't 1950 anymore. 1950s biases about what "efficiency" means no longer apply.
(Lisp used to be concerned with "development time efficiency" and can be again)
pt
In that regard, there is py4cl2: https://github.com/digikar99/py4cl2 Not sure if you know. MB
On 11 Apr 2023, at 7:23, Manfred Bergmann wrote:
Am 11.04.2023 um 14:17 schrieb paul tarvydas <paultarvydas@gmail.com>:
"If you specifically want numpy, it may be possible to have Common Lisp talking to python."
CL can run python
(uiop:run-program "python3 ~/quicklisp/local-projects/clvpy/hello.py" :output t)
This isn't 1950 anymore. 1950s biases about what "efficiency" means no longer apply.
(Lisp used to be concerned with "development time efficiency" and can be again)
pt
In that regard, there is py4cl2: https://github.com/digikar99/py4cl2 Not sure if you know.
FWIW, py4cl2 in my experience is in very rough shape, and not fit for use. Stick to py4cl instead of the fork. It's simpler and doesn't do a lot of things py4cl2 wants to, but it's more solid.
On 11 Apr 2023, at 14:57, Robert Goldman wrote: On 11 Apr 2023, at 7:23, Manfred Bergmann wrote: Am 11.04.2023 um 14:17 schrieb paul tarvydas <paultarvydas@gmail.com>: "If you specifically want numpy, it may be possible to have Common Lisp talking to python." CL can run python (uiop:run-program "python3 ~/quicklisp/local-projects/clvpy/hello.py" :output t) This isn't 1950 anymore. 1950s biases about what "efficiency" means no longer apply. (Lisp used to be concerned with "development time efficiency" and can be again) pt In that regard, there is py4cl2: https://github.com/digikar99/py4cl2 Not sure if you know. FWIW, py4cl2 in my experience is in very rough shape, and not fit for use. Stick to py4cl instead of the fork. It's simpler and doesn't do a lot of things py4cl2 wants to, but it's more solid. Even more recently, there is py4cl2-cffi that provides support for passing CL arrays by reference: https://github.com/digikar99/py4cl2-cffi The support is one-directional, in that arrays returned by numpy whose data was not allocated by CL still need to be copied over. It is possible to write a wrapper around numpy arrays, so arrays can become completely owned by numpy, so this copying over becomes unnecessary. And while I have been trying to use it as my daily driver, I'd definitely say that it is in an ever rougher shape than py4cl2, and will require time and risk-takers to make it suitable for production. On the other hand there is mgl-mat: https://github.com/melisgl/mgl-mat and And magicl (backed by quil-lang?): https://github.com/quil-lang/magicl And numericals: https://github.com/digikar99/numericals And as someone already mentioned, lisp-stat: https://github.com/Lisp-Stat/lisp-stat and https://lisp-stat.dev/docs/ digikar
(Lisp used to be concerned with "development time efficiency" and can be again)
; in: LISP USED ; (LISP USED TO BE CONCERNED WITH "development time efficiency" AND CAN BE ; AGAIN) ; ; caught WARNING: ; undefined variable: COMMON-LISP-USER::AGAIN ; ; caught WARNING: ; undefined variable: COMMON-LISP:AND ; ; caught WARNING: ; undefined variable: COMMON-LISP-USER::BE ; ; caught WARNING: ; undefined variable: COMMON-LISP-USER::CAN ; ; caught WARNING: ; undefined variable: COMMON-LISP-USER::CONCERNED ; ; caught STYLE-WARNING: ; undefined function: COMMON-LISP-USER::LISP ; ; caught WARNING: ; undefined variable: COMMON-LISP-USER::TO ; ; caught WARNING: ; undefined variable: COMMON-LISP-USER::USED ; ; caught WARNING: ; undefined variable: COMMON-LISP-USER::WITH ; ; compilation unit finished ; Undefined function: ; LISP ; Undefined variables: ; AGAIN AND BE CAN CONCERNED TO USED WITH ; caught 8 WARNING conditions ; caught 1 STYLE-WARNING condition
I'm not sure I'd want a clone of a Python style API; I'd rather have something that's idiomatic lisp. I like Marco's idea of a spec, however beyond documentation (and that is valuable), it mostly means we can have multiple implementations, and I suspect they'd end up like everything else, half-baked, poorly documented implementations of the spec. I think we can do a lot of useful work without cloning the entire Numpy API. The real challenge, IMO, is what Marco outlines below. Getting the community to rally behind a single implementation that they didn't cook up themselves in their garage whilst watching TV and drinking beer. NIH disease runs rampant. There are however enough high-quality starting points (libraries) that might be worth looking at. I think tpapp's stuff is good, with generally small codebase and an API worked out via actual usage in a real-world environment. Cheers, Steve On Tuesday, April 11, 2023 at 05:19:25 AM GMT+8, Marco Antoniotti <marco.antoniotti@unimib.it> wrote: IMHO, it'd be easier and effective to band up together and FIRST write a proper API specification and THEN implement it in CL. But Common Lispers are like academics: the "herding cats" applies. Cheers Marco PS I am a Common Lisper AND an academic. You know what I mean... On Mon, Apr 10, 2023 at 9:09 PM Robert Goldman <rpgoldman@sift.info> wrote: Has anyone taken the Numpy C API and created a CFFI library based on it? I see some attempts to clone it in pure CL, but I would guess that just using it through its API might be easier and more effective.
Hi Robert, The answer to your original question appears to be no, so the conversation turned to brainstorming solutions. However your question did not provide a clear scope and purpose or measure of fitness for such a development. Answers to the following questions may help focus this conversation. Why did you ask for the CFFI binding? Do you want access to a bigger function library, better numeric performance, or something else? Why would you prefer CFFI bindings to numpy over CFFI bindings to a custom kernel? What schedule are you hoping for? How would you value traits such as portability, performance, implementation effort, user effort, and schedule? -- Daniel
Hi, Daniel -- I was looking for a system that I could use to do some numerical computations I now do in Python in CL instead. That involves a relatively large function library (or perhaps more accurately, the developers of numpy have a better idea what functions are likely to be needed than do I), and the ability to do vectorized operations. I asked for a CFFI binding because I found a number of stabs at something similar in CL, but none looked promising for use in the short term. I'm not sure what you mean by "a custom kernel," I'm afraid. Schedule was "as soon as possible," in the sense of "if it's there I will start using it right away." I guess the trait valuation is that I would like something that I could use instead of numpy, so quite portable (although if it only worked in SBCL, that would be fine), implementation effort: essentially none (I want to use a capable linear algebra library, not write one). Best, R On 11 Apr 2023, at 23:18, Daniel Herring wrote:
Hi Robert,
The answer to your original question appears to be no, so the conversation turned to brainstorming solutions. However your question did not provide a clear scope and purpose or measure of fitness for such a development.
Answers to the following questions may help focus this conversation.
Why did you ask for the CFFI binding?
Do you want access to a bigger function library, better numeric performance, or something else?
Why would you prefer CFFI bindings to numpy over CFFI bindings to a custom kernel?
What schedule are you hoping for?
How would you value traits such as portability, performance, implementation effort, user effort, and schedule?
-- Daniel
Hi Robert, To add to the list of library propositions for numpy in Lisp, we can probably also have a look at the language Hylang that is a Lisp dialect that's embedded in Python. Cheers * Robert Goldman | Hi, Daniel -- | I was looking for a system that I could use to do some numerical | computations I now do in Python in CL instead. That involves a | relatively large function library (or perhaps more accurately, the | developers of numpy have a better idea what functions are likely to be | needed than do I), and the ability to do vectorized operations. | I asked for a CFFI binding because I found a number of stabs at something similar in CL, but none looked promising for use in the short term. | I'm not sure what you mean by "a custom kernel," I'm afraid. | Schedule was "as soon as possible," in the sense of "if it's there I will start using it right away." | I guess the trait valuation is that I would like something that I | could use instead of numpy, so quite portable (although if it only | worked in SBCL, that would be fine), implementation effort: | essentially none (I want to use a capable linear algebra library, not | write one). | Best, | R
Hi As the main proponent of the utility ‘irsabol’ I am partial to Julia (which is a Common Lisp in a drag :) ). IMHO Julia is doing many things right. All the best Marco On Thu, 13 Apr 2023 at 04:18, <lpgl@free.fr> wrote:
Hi Robert,
To add to the list of library propositions for numpy in Lisp, we can probably also have a look at the language Hylang that is a Lisp dialect that's embedded in Python.
Cheers
* Robert Goldman | Hi, Daniel --
| I was looking for a system that I could use to do some numerical | computations I now do in Python in CL instead. That involves a | relatively large function library (or perhaps more accurately, the | developers of numpy have a better idea what functions are likely to be | needed than do I), and the ability to do vectorized operations.
| I asked for a CFFI binding because I found a number of stabs at something similar in CL, but none looked promising for use in the short term.
| I'm not sure what you mean by "a custom kernel," I'm afraid.
| Schedule was "as soon as possible," in the sense of "if it's there I will start using it right away."
| I guess the trait valuation is that I would like something that I | could use instead of numpy, so quite portable (although if it only | worked in SBCL, that would be fine), implementation effort: | essentially none (I want to use a capable linear algebra library, not | write one).
| Best, | R
-- Marco Antoniotti https://dcb.disco.unimib.it
Robert, When you write: "I asked for a CFFI binding because I found a number of stabs at something similar in CL, but none looked promising for use in the short term." Which libraries did you try and what was lacking? Did you provide any input or raise any issues to the authors and maintainers? Is it only linear algebra you want to do? Cheers, Steve On Thursday, April 13, 2023 at 04:11:13 AM GMT+8, Robert Goldman <rpgoldman@sift.info> wrote: Hi, Daniel -- I was looking for a system that I could use to do some numerical computations I now do in Python in CL instead. That involves a relatively large function library (or perhaps more accurately, the developers of numpy have a better idea what functions are likely to be needed than do I), and the ability to do vectorized operations. I asked for a CFFI binding because I found a number of stabs at something similar in CL, but none looked promising for use in the short term. I'm not sure what you mean by "a custom kernel," I'm afraid. Schedule was "as soon as possible," in the sense of "if it's there I will start using it right away." I guess the trait valuation is that I would like something that I could use instead of numpy, so quite portable (although if it only worked in SBCL, that would be fine), implementation effort: essentially none (I want to use a capable linear algebra library, not write one). Best, R On 11 Apr 2023, at 23:18, Daniel Herring wrote:
Hi Robert,
The answer to your original question appears to be no, so the conversation turned to brainstorming solutions. However your question did not provide a clear scope and purpose or measure of fitness for such a development.
Answers to the following questions may help focus this conversation.
Why did you ask for the CFFI binding?
Do you want access to a bigger function library, better numeric performance, or something else?
Why would you prefer CFFI bindings to numpy over CFFI bindings to a custom kernel?
What schedule are you hoping for?
How would you value traits such as portability, performance, implementation effort, user effort, and schedule?
-- Daniel
participants (17)
-
Antoni Grzymała -
Attila Lendvai -
Craig -
Daniel Herring -
Elliott Johnson -
lpgl＠free.fr -
Manfred Bergmann -
Marco Antoniotti -
Michael Bentley -
michael＠stray-hound.com -
paul tarvydas -
Raymond Wiker -
Robert Goldman -
Robert Goldman -
Shubhamkar Ayare -
Stelian Ionescu -
Steven Nunez