Hi Akshay,
Thank you for your work and your email.
I think a lot of people have had very similar thoughts to yours, and have started down the road meeting the needs of users they've identified. There are a lot of good and partially overlapping ideas within this loosely organized group of people. There was a similar discussion started in the context of common lisp stats (which I'm not a member of) a few months ago. Where I was participating, I cc'ed the antik-devel mailing list; see for example
http://lists.common-lisp.net/pipermail/antik-devel/2012q4/000019.html
and the archives in general http://lists.common-lisp.net/pipermail/antik-devel/2012q4/thread.html with the subject "numerical lisp".
My feeling is that we need to break the desired functions into relatively narrow, compatible libraries. I started Antik as a split out of GSLL to handle "grids" (generalized arrays) to start down that road. It needs to be split up further, because it now includes other things such as calculations with physical dimensions. I've started that further split, but haven't done any work on it recently.
I am interested in the work you've done; it's been a while since I looked at Matlisp. At the time, it did not meet my needs, but it seems you have done significant work on it and has a lot of potential to be part of the compatible family of libraries as I've described. I think a good first step is to identify the different areas (for example, generalized arrays as I've indicated, or statistics functions) and canvas the current state of software for features and capabilities.
I've added Tamas Papp and Mirko Vukovic onto this email because they've been very active in thinking about these things. Also I've added the Antik and GSLL mailing lists for good measure.
Liam
On Wed, Feb 27, 2013 at 1:04 AM, Akshay Srinivasan akshaysrinivasan@gmail.com wrote:
Hello All, I'm sort of the active developer of Matlisp these days (mostly whilst procrastinating from doing more "academic" work). I know all of you have created your independent system for scientific computing on Common Lisp, and it is really a pity that all of them (including the old Matlisp) are not particularly usable when compared with something like SciPy or Matlab even; more so since Lisp is so much nicer a language. I think the version of Matlisp I'm developing has a lot of promise, and is taking a very good shape (atleast I'd like to think so). I'm personally trying to use it for my work in Control theory and such; but it's still not something which is particularly usable yet. The improvements over the old version of Matlisp are:
- Use of a hacky static object system, so that functions can be generated for any user-defined "field". There is some code demonstrating the use of Maxima to work with "symbolic-tensors", there isn't code for integer matrices yet, but getting this to work is fairly trivial.
- The tensor (multi-array) structures can now be sliced inplace very easily like SciPy.
- Automatic fortran callback generation (see dlsode.lisp). There are plans to extend this to C, so that we can make use of GSL.
- Lisp implementations of BLAS functions, and support for seamlessly switching between the Lisp and Fortran versions of these routines. The lisp version of GEMM (and others) is nearly as fast as the F77 reference BLAS one, on SBCL.
- The FFI has moved to CFFI, if you haven't noticed that yet; builds on CCL and CMUCL.
The basic structure of BLAS is out there, but the LAPACK wrappers are missing. However I do think that the general structure of the code is in place.
I just wanted to throw out the idea into the open, about the feasibility of merging all your projects into Matlisp (and calling it by some other name, if that is an issue) ? Would any of you be willing to take the time to go through my code and merge yours in ? I'm writing to you now before I take time and polish things, because then there'd be little room to make architectural changes later on. The code is here if you'd like to give it a try: git://matlisp.git.sourceforge.net/gitroot/matlisp/matlisp tensor or if you just want to browse the code: https://github.com/enupten/matlisp You may want to look at the (ugly) code in tests/ folder to get an idea about the code behaviour. You'd also want to work in #:matlisp package (despite what the herald says).
Getting the maxima bits to work is kind of tricky, but you don't really need it for numerical routines to work.
Akshay
At Wed, 27 Feb 2013 12:11:26 -0500, Liam Healy wrote:
Hi Akshay,
Thank you for your work and your email.
I think a lot of people have had very similar thoughts to yours, and have started down the road meeting the needs of users they've identified. There are a lot of good and partially overlapping ideas within this loosely organized group of people. There was a similar discussion started in the context of common lisp stats (which I'm not a member of) a few months ago. Where I was participating, I cc'ed the antik-devel mailing list; see for example
http://lists.common-lisp.net/pipermail/antik-devel/2012q4/000019.html
and the archives in general http://lists.common-lisp.net/pipermail/antik-devel/2012q4/thread.html with the subject "numerical lisp".
My feeling is that we need to break the desired functions into relatively narrow, compatible libraries. I started Antik as a split out of GSLL to handle "grids" (generalized arrays) to start down that road. It needs to be split up further, because it now includes other things such as calculations with physical dimensions. I've started that further split, but haven't done any work on it recently.
I am interested in the work you've done; it's been a while since I looked at Matlisp. At the time, it did not meet my needs, but it seems you have done significant work on it and has a lot of potential to be part of the compatible family of libraries as I've described. I think a good first step is to identify the different areas (for example, generalized arrays as I've indicated, or statistics functions) and canvas the current state of software for features and capabilities.
I think its very important to get the generalized arrays working very well. I'm not particularly keen on shadowing CL's functions though; I know it sort of makes things clumsy (to work around CL), but it also makes it awkward to use it outside the package.
Getting the arrays to work quite well was really my goal, I think its easier to build functionality around the array structure. I can just polish the code that is there now to suit my needs, and then get on with my work, but that'll only lead to another isolated project in the lisp world.
It'd be useful if some of you try using matlisp and see if it is atleast technically suitable for some of your tasks. I know there are a lot of design decisions which many of you will not agree with, but if you think Matlisp is worth the effort to base your work on, I'd be more than happy to make a compromise and change the structure to suit your needs.
Akshay
I've added Tamas Papp and Mirko Vukovic onto this email because they've been very active in thinking about these things. Also I've added the Antik and GSLL mailing lists for good measure.
Liam
On Wed, Feb 27, 2013 at 1:04 AM, Akshay Srinivasan akshaysrinivasan@gmail.com wrote:
Hello All, I'm sort of the active developer of Matlisp these days (mostly whilst procrastinating from doing more "academic" work). I know all of you have created your independent system for scientific computing on Common Lisp, and it is really a pity that all of them (including the old Matlisp) are not particularly usable when compared with something like SciPy or Matlab even; more so since Lisp is so much nicer a language. I think the version of Matlisp I'm developing has a lot of promise, and is taking a very good shape (atleast I'd like to think so). I'm personally trying to use it for my work in Control theory and such; but it's still not something which is particularly usable yet. The improvements over the old version of Matlisp are:
- Use of a hacky static object system, so that functions can be generated for any user-defined "field". There is some code demonstrating the use of Maxima to work with "symbolic-tensors", there isn't code for integer matrices yet, but getting this to work is fairly trivial.
- The tensor (multi-array) structures can now be sliced inplace very easily like SciPy.
- Automatic fortran callback generation (see dlsode.lisp). There are plans to extend this to C, so that we can make use of GSL.
- Lisp implementations of BLAS functions, and support for seamlessly switching between the Lisp and Fortran versions of these routines. The lisp version of GEMM (and others) is nearly as fast as the F77 reference BLAS one, on SBCL.
- The FFI has moved to CFFI, if you haven't noticed that yet; builds on CCL and CMUCL.
The basic structure of BLAS is out there, but the LAPACK wrappers are missing. However I do think that the general structure of the code is in place.
I just wanted to throw out the idea into the open, about the feasibility of merging all your projects into Matlisp (and calling it by some other name, if that is an issue) ? Would any of you be willing to take the time to go through my code and merge yours in ? I'm writing to you now before I take time and polish things, because then there'd be little room to make architectural changes later on. The code is here if you'd like to give it a try: git://matlisp.git.sourceforge.net/gitroot/matlisp/matlisp tensor or if you just want to browse the code: https://github.com/enupten/matlisp You may want to look at the (ugly) code in tests/ folder to get an idea about the code behaviour. You'd also want to work in #:matlisp package (despite what the herald says).
Getting the maxima bits to work is kind of tricky, but you don't really need it for numerical routines to work.
Akshay
-
Akshay Srinivasan -
Liam Healy