Hi Mirko, Tamas, Tony,

I am happy to read of these developments; they align very closely with what I have in mind in building grid and Antik.

A little bit of background: I do numerical computation as part of my "day jobs," and have used Lisp for 25 years now, starting with the Symbolics. Over the decades, I developed for personal use only a rather (ahem) hackish set of numerical libraries and utilities. When GSL came out in the late 90s, I saw it as an opportunity to gain some numerical utilities without having to write them myself, but it took until CFFI's release in 2005 to spur me to start GSLL. I am of the mind that any correct library is potentially useful and should be available in CL; while I prefer to code in CL myself, if the library is already written, then I will try to use it. I realize others may differ. My goal in writing GSLL was to make the non-lispness of it as invisible as possible, so that it can be used with all the power of lisp and none of the hassles of C.

One of the hacks I had in my old personal library was that of generic arrays. With some GSL functions needing arrays, I was forced to reinvent that. I did a lot of research of existing CL libraries, and settled on Tamas's affine library (which predates xarray, I believe) as a basis, and built on that. Eventually, this was split out from GSLL into the grid package in Antik, which also includes other numerical utilities. I envision that grids could address any kind of array-like structure (including, say SQL tables, structured files like FITS, HDF, etc.) because these are the kinds of things I use in my work. They should be as transparent as possible, so that if for example there is a function in GSL that does the same thing as another library (written in C, or CL, or whatever), I can send the exact same inputs and compare the outputs.

There is a lot of work to be done to get Antik and grids to that point, and I welcome ideas, contributions, and patches. Unfortunately, my time on this has been very limited recently, but I intend to move this forward as best as I can. Mirko has sent me emails and I've been slow to respond. I think the antik-develhttp://lists.common-lisp.net/cgi-bin/mailman/listinfo/antik-devellist is a good place to discuss any ideas, and its email archive may be useful in the future. It is my intent to be inclusive of different approaches as possible, so that even working entirely within CL using arrays and displaced arrays should be supported (currently, displaced arrays are not there). I look at languages like Fortran 90, Python, and Matlab, and I think there is opportunity to borrow idioms and features, and maybe have interfaces to some of those libraries. I don't know R but suspect it too has something to offer. As Mirko noted, I use (shadowed) CL symbols where possible.

One special note about the current status: all development for the last several months or so (and there hasn't been much) is on the "cffi-libffi" branches. Over last fall and winter, working with Luís Oliveira, I have replaced FSBV with the system cffi-libffi, currently distributed on the master branch of CFFI. This is a significant improvement on FSBV because calling structures by value require nothing more than CFFI calls and having libffi (also, things like recursive structures should work). While this is in the master branch of CFFI, it is not yet in a released version of CFFI, so I have held off merging my cffi-libffi branches into master because Quicklisp only pulls from released versions of software, where such a concept is defined (it isn't for Antik and GSLL). However, I don't maintain the FSBV-based master branches any more.

(I am copying antik-devel so that other Antik users will see.)

Liam

On Fri, Oct 12, 2012 at 7:45 AM, A.J. Rossini blindglobe@gmail.com wrote:

Hi Mirko -

The other possibility is that depending on the complexity of Antik (sounds like it does a good deal, whether it should all be in the same place is another matter, which is just a question that I shouldn't ask until I read the code later this weekend, Sorry for posing it prematureley!), we just replace xarray with antik. Some of this might be pretty simple.

On Friday, October 12, 2012 2:02:06 AM UTC+2, Mirko Vukovic wrote:

...

This is a bit long-winded, but it may have some relevance to the architecture of CL-Statistics

The other day, I was writing this as part of a sequence generating library documentation that I hope to release soon:

The Intoruction to Rhttp://cran.r-project.org/doc/manuals/R-intro.html#Top describes vectorshttp://cran.r-project.org/doc/manuals/R-intro.html#Simple-manipulations-numbers-and-vectors. Of interest are also sections on factorshttp://cran.r-project.org/doc/manuals/R-intro.html#Factors, arrays and matriceshttp://cran.r-project.org/doc/manuals/R-intro.html#Arrays-and-matrices, and lists and data frameshttp://cran.r-project.org/doc/manuals/R-intro.html#Lists-and-data-frames. Finally, reading datahttp://cran.r-project.org/doc/manuals/R-intro.html#Reading-data-from-filesis of interest. At this point, this is starting to sound like Rossini's project common-lisp-stat. However note that this project uses Tamas Papp's xarray.

At that point, I went to google what Rossini was up to, and came upon this list. It was up for only a day or so when I found it.

The reason for this post is that I am using Liam Healy's gsll and antik libraries. These use his grid library for representing vectors, matrices, etc. It is unfortunate that at the time Liam was releasing grid, Tamas released his xarray.

Personally, I dislike fragmentation (grid vs xarray), but now nothing can be done about it. And some competition is good.

The interesting part of Liam's antik is that antik shadows several of CL's symobls: many math functions, and also aref and aref*. He reimplements these as generic functions, allowing extension for other data types (such as grids). In a way, Liam is on his way to creating what I would call Numerical Lisp: Extension of CL for numerical computing that may be incompatible with CL. But the extension is a relatively thin layer on top of CL, leaving all of CL accessible.

Futhermore, Numerical-Lisp (NL) can unify grid and xarray using the following three-layered structure:

• Application layer, such as CL-Statistics
• Numerical-Lisp layer
• Raw libraries (grid/xarray)

The numerical lisp layer would consist of Liam's redefinitions of CL mathematics related functions via generic functions. The raw libraries would hook into this via methods. In addition to CL mathematics functions, NL can define interface to higher mathematics functions also using generic functions. Then the raw libraries can plug in using packages such as GSL, LAPACK, etc.

What needs to be done is

• creation of the Numerical-Lisp library
• hooks in grid and xarray
• write the application layer to use Numerical-Lisp interface.

It seems to me that much of this should be relatively straightforward. I would be glad to help in testing (sbcl, clisp) and documenting.

Finally, what does this have to do with CL-statistics? Well, I am suggesting that the code be organized in a way to make it easy to transition from interfacing to CL & xarray into NL.

Best,

Mirko

PS - and in the end, when all is said and done, someone writes a great book: Practical Numerical Lisp