antik-devel

antik-devel@common-lisp.net

35 discussions

[Antik-devel] ann: numerical-lisp on github.com
by Mirko Vukovic 22 Oct '12

22 Oct '12

Based on a recent discussion<https://groups.google.com/d/topic/lisp-stat/sYBndICy_HA/discussion>on the Common Lisp Statistics group, I decided to start a github project numerical-lisp <https://github.com/mirkov/numerical-lisp>. The purpose of the project is to serve as an ordered (as much as possible) collection of thoughts on what would make a useful and powerful system for numerical programming, analysis and visualization based on common-lisp. I suspect the discussion will be taking place on the common lisp statistics group, antik, and maybe others. I don't think that we should form yet another discussion forum. Coding would start in the next few months. The project would depend a great deal on Antik, gsll, cl-num-utils, and other numerical libraries. The goal is not to start yet another incompatible numerical library. Instead, the goal is to provide a flexible framework into which one can plug in existing libraries with relatively little modification. When it comes to coding, there will be relatively little intense numerical or graphics coding. Instead, we should use as many of already existing resources and libraries already available, and allow enough flexibility so as not to be locked to a particular library. We should also look at other systems (R, numpy, Matlab, mathematica, Macsyma), and incorporate their best features, albeit in a lispy way. The project may boil down to software engineering, and good language design (maybe we should study Fortress for hints) Right now the project is quite empty, and I will add to it over the coming weeks. Once I figure out how github operates for collaborative work, I would welcome contributions from others as well. I hope this to be a community effort. Mirko

4 6

Re: [Antik-devel] antik on clisp - some corrections
by Liam Healy 20 Oct '12

20 Oct '12

Mirko, If you know texinfo you can patch the doc source where the links are bad, otherwise I'll try to fix it. If you can create a git patch to the head of cffi-libffi I will take a look at it. I think I understand the reason for the conflict on 'second (the unit of time vs. the function that produces the second element of a sequence) but I don't know why it's not seen on other implementations. I don't understand the other conflicts off the top of my head. Liam On Sun, Oct 7, 2012 at 12:14 PM, Mirko Vukovic <mirko.vukovic(a)gmail.com>wrote: > Liam, > > I installed antik on clisp on cygwin+windows. (I also have it running on > sbcl & linux). > > To make it go I had to change a few things: > > - minor syntax tweak in the asd file (use :asdf-system-connections instead > of #:asdf-system-connections) > - shadow a few more cl symbols in package.lisp > > #+clisp #:second #+clisp #:fifth #+clisp #:tenth > #+clisp #:rem > > > Also, in a few files I had to specify `cl:second' instead of just `second' > > Now all but two tests are passing. > > Two questions/points: > - Maybe I missed something. Could that be the reason for the symbol > conflicts. > - If you want to accept these changes, what is the best way to submit them? > > Also, there are faulty links on the html documentation. Chapter five > links5 and 5.1 both lead to Section 4.5. > > Thank you for the antik. It is quite something. > > I will now proceed to adapt map-grid-utils to the version of grid that is > built into antik. > > Mirko > > >

2 1

Re: [Antik-devel] First post: common-lisp -> numerical-lisp->CL-statistics
by Liam Healy 14 Oct '12

14 Oct '12

Hi Mirko, Tamas, Tony, I am happy to read of these developments; they align very closely with what I have in mind in building grid and Antik. A little bit of background: I do numerical computation as part of my "day jobs," and have used Lisp for 25 years now, starting with the Symbolics. Over the decades, I developed for personal use only a rather (ahem) hackish set of numerical libraries and utilities. When GSL came out in the late 90s, I saw it as an opportunity to gain some numerical utilities without having to write them myself, but it took until CFFI's release in 2005 to spur me to start GSLL. I am of the mind that any correct library is potentially useful and should be available in CL; while I prefer to code in CL myself, if the library is already written, then I will try to use it. I realize others may differ. My goal in writing GSLL was to make the non-lispness of it as invisible as possible, so that it can be used with all the power of lisp and none of the hassles of C. One of the hacks I had in my old personal library was that of generic arrays. With some GSL functions needing arrays, I was forced to reinvent that. I did a lot of research of existing CL libraries, and settled on Tamas's affine library (which predates xarray, I believe) as a basis, and built on that. Eventually, this was split out from GSLL into the grid package in Antik, which also includes other numerical utilities. I envision that grids could address any kind of array-like structure (including, say SQL tables, structured files like FITS, HDF, etc.) because these are the kinds of things I use in my work. They should be as transparent as possible, so that if for example there is a function in GSL that does the same thing as another library (written in C, or CL, or whatever), I can send the exact same inputs and compare the outputs. There is a lot of work to be done to get Antik and grids to that point, and I welcome ideas, contributions, and patches. Unfortunately, my time on this has been very limited recently, but I intend to move this forward as best as I can. Mirko has sent me emails and I've been slow to respond. I think the antik-devel<http://lists.common-lisp.net/cgi-bin/mailman/listinfo/antik-devel>list is a good place to discuss any ideas, and its email archive may be useful in the future. It is my intent to be inclusive of different approaches as possible, so that even working entirely within CL using arrays and displaced arrays should be supported (currently, displaced arrays are not there). I look at languages like Fortran 90, Python, and Matlab, and I think there is opportunity to borrow idioms and features, and maybe have interfaces to some of those libraries. I don't know R but suspect it too has something to offer. As Mirko noted, I use (shadowed) CL symbols where possible. One special note about the current status: all development for the last several months or so (and there hasn't been much) is on the "cffi-libffi" branches. Over last fall and winter, working with Luís Oliveira, I have replaced FSBV with the system cffi-libffi, currently distributed on the master branch of CFFI. This is a significant improvement on FSBV because calling structures by value require nothing more than CFFI calls and having libffi (also, things like recursive structures should work). While this is in the master branch of CFFI, it is not yet in a released version of CFFI, so I have held off merging my cffi-libffi branches into master because Quicklisp only pulls from released versions of software, where such a concept is defined (it isn't for Antik and GSLL). However, I don't maintain the FSBV-based master branches any more. (I am copying antik-devel so that other Antik users will see.) Liam On Fri, Oct 12, 2012 at 7:45 AM, A.J. Rossini <blindglobe(a)gmail.com> wrote: > Hi Mirko - > > The other possibility is that depending on the complexity of Antik (sounds > like it does a good deal, whether it should all be in the same place is > another matter, which is just a question that I shouldn't ask until I read > the code later this weekend, Sorry for posing it prematureley!), we just > replace xarray with antik. Some of this might be pretty simple. > > > > On Friday, October 12, 2012 2:02:06 AM UTC+2, Mirko Vukovic wrote: >> >> This is a bit long-winded, but it may have some relevance to the >> architecture of CL-Statistics >> >> The other day, I was writing this as part of a sequence generating >> library documentation that I hope to release soon: >> >> The Intoruction to R<http://cran.r-project.org/doc/manuals/R-intro.html#Top> describes >> vectors<http://cran.r-project.org/doc/manuals/R-intro.html#Simple-manipulations-num…>. >> Of interest are also sections on factors<http://cran.r-project.org/doc/manuals/R-intro.html#Factors>, >> arrays and matrices<http://cran.r-project.org/doc/manuals/R-intro.html#Arrays-and-matrices>, >> and lists and data frames<http://cran.r-project.org/doc/manuals/R-intro.html#Lists-and-data-frames>. >> Finally, reading data<http://cran.r-project.org/doc/manuals/R-intro.html#Reading-data-from-files>is of interest. At this point, this is starting to sound like Rossini's >> project common-lisp-stat. However note that this project uses Tamas >> Papp's xarray. >> >> At that point, I went to google what Rossini was up to, and came upon >> this list. It was up for only a day or so when I found it. >> >> The reason for this post is that I am using Liam Healy's gsll and antik >> libraries. These use his grid library for representing vectors, matrices, >> etc. It is unfortunate that at the time Liam was releasing grid, Tamas >> released his xarray. >> >> Personally, I dislike fragmentation (grid vs xarray), but now nothing can >> be done about it. And some competition is good. >> >> The interesting part of Liam's antik is that antik shadows several of >> CL's symobls: many math functions, and also aref and aref*. He >> reimplements these as generic functions, allowing extension for other data >> types (such as grids). In a way, Liam is on his way to creating what I >> would call Numerical Lisp: Extension of CL for numerical computing that may >> be incompatible with CL. But the extension is a relatively thin layer on >> top of CL, leaving all of CL accessible. >> >> Futhermore, Numerical-Lisp (NL) can unify grid and xarray using the >> following three-layered structure: >> - Application layer, such as CL-Statistics >> - Numerical-Lisp layer >> - Raw libraries (grid/xarray) >> >> The numerical lisp layer would consist of Liam's redefinitions of CL >> mathematics related functions via generic functions. The raw libraries >> would hook into this via methods. In addition to CL mathematics functions, >> NL can define interface to higher mathematics functions also using generic >> functions. Then the raw libraries can plug in using packages such as GSL, >> LAPACK, etc. >> >> What needs to be done is >> - creation of the Numerical-Lisp library >> - hooks in grid and xarray >> - write the application layer to use Numerical-Lisp interface. >> >> It seems to me that much of this should be relatively straightforward. I >> would be glad to help in testing (sbcl, clisp) and documenting. >> >> Finally, what does this have to do with CL-statistics? Well, I am >> suggesting that the code be organized in a way to make it easy to >> transition from interfacing to CL & xarray into NL. >> >> Best, >> >> Mirko >> >> PS - and in the end, when all is said and done, someone writes a great >> book: Practical Numerical Lisp >> >

1 0

[Antik-devel] Sharp-m reader macro ignores *read-suppress* (+ patch)
by Martin Schmauder 28 Apr '12

28 Apr '12

Hi, I have encountered a problem with the sharp-m reader macro function. The sharp-m reader macro function (lambda in grid/functions.lisp, l. 202) uses the function cl:read-delimited-list to read the macro arguments. However, when cl:*read-suppress* is true, read-delimited-list always returns nil (See *read-suppress* in CLHS.) That value is then (wrongly) passed to grid:check-initial-contents, which (rightly) signals the error "Initial-contents specified as NIL." The following code can be used to observe the behaviour. No matter what (valid or invalid) code is inside the sharp-m parentheses, the error will be signaled. (let ((*read-suppress* t)) (with-input-from-string (stream "#m(1 2 3)") (read stream))) The solution is to call read-delimited-list and then check if we have to ignore its return value with "(unless *read-suppress* ...)". Patch attached. By the way, how I encountered this issue: When I compile a file with C-c C-k and the compilation yields warnings or notes, the Slime/SBCL combination seems to re-read the file with *read-suppress*. Any sharp-m macro that is then read triggers the error. Try with: #m(1 2 3) (defun foo (bar)) ; unused variable style-warning Comments? Martin

2 1

[Antik-devel] Sharp-m reader macro function ignores *read-suppress* (+ patch)
by Martin Schmauder 25 Apr '12

25 Apr '12

Hi, the sharp-m reader macro function (grid/functions.lisp, l. 202) ignores cl:*read-suppress*. When *read-suppress* is true, the function unconditionally signals the error "Initial-contents specified as NIL." The following code can be used to observe the behaviour. No matter what (valid or invalid) code is inside the sharp-m parentheses, the error will be signaled. (let ((*read-suppress* t)) (with-input-from-string (stream "#m(1 2 3)") (read stream))) I encountered this in Emacs: When I compile a file with C-c C-k and the compilation yields warnings or notes, the Slime/SBCL combination seems to re-read the file with *read-suppress*. Any sharp-m macro that is then read triggers the error. Try with: #m(1 2 3) (defun foo (bar)) ; provoke unused variable style-warning My solution is to put a simple "(unless *read-suppress* ...)" at the appropriate place: if *read-suppress* is true, the macro contents will still be read but then ignored. A patch is attached. The patch did not break any tests for me. Does this seem sane? Comments? Martin

1 0

[Antik-devel] Setting values in an array using loop and setf, and :initial-contents
by Sumant S. R. Oemrawsingh 28 Mar '12

28 Mar '12

Hi Liam, I quickly did the following tests on my laptop, to verify where the speed problems I've been having come from. See below for the forms and results. The outcome is that grid:aref is slow (perhaps due to consing?) and :initial-contents in make-foreign-array is sloooooooow. A quick summary: Setfing the values of an array inside a loop: * With a cl-array, grid:aref is an order of magnitude slower than aref. Using grid:aref seems to cons about 4 times as much, and non-GC time is about a factor of 20 slower than just using cl aref. * Grid:aref is more than two times as slow with a foreign array than with a cl-array. There is about 4 times as much consing as well. Setting the values of an array using :initial-contents and a list: * With make-array, this is blazingly fast (no time, no consing); probably because the list values are already stored properly in memory. * With make-foreign array, this takes 50 minutes and a whole lot of consing (for a 1024x1024 array). Don't try this at home. Basically, when using antik/grid, don't use :initial-contents. Still, setfing values in a loop is an order of magnitude slower (when no optimizations are done). I assume that if optimizations are applied, where it is known at compile-time you have a cl-array, grid:aref should be as fast as aref, and I will test this next. The thing is, though, for a generic library of data processing functions, you can't optimize for the specific array that you will use. A minor hit in performance would still be acceptable, but an order of magnitude difference for a cl-array when changing from aref to grid:aref is a bit crazy imo. Sumant ;;;; loop-setf-test.lisp (defun make-lists-list (rows columns) (loop for row below rows collect (loop for column below columns collect (* row column 1.0d0)))) (defun aref-loop-setf (rows columns) (let ((array (make-array (list rows columns) :element-type 'double-float))) (loop for row below rows do (loop for column below columns do (setf (aref array row column) (* row column 1.0d0)))) array)) (defun grid-aref-loop-setf (rows columns) (let ((array (make-array (list rows columns) :element-type 'double-float))) (loop for row below rows do (loop for column below columns do (setf (grid:aref array row column) (* row column 1.0d0)))) array)) (defun grid-aref-loop-setf-foreign (rows columns) (let ((array (grid:make-foreign-array 'double-float :dimensions (list rows columns)))) (loop for row below rows do (loop for column below columns do (setf (grid:aref array row column) (* row column 1.0d0)))) array)) #| Results on my laptop are as follows. CL-USER> (time (progn (aref-loop-setf 1024 10240) (values))) Evaluation took: 0.234 seconds of real time 0.233963 seconds of total run time (0.192970 user, 0.040993 system) [ Run times consist of 0.057 seconds GC time, and 0.177 seconds non-GC time. ] 100.00% CPU 654,633,933 processor cycles 251,658,256 bytes consed ; No value CL-USER> (time (progn (grid-aref-loop-setf 1024 10240) (values))) Evaluation took: 3.094 seconds of real time 3.084532 seconds of total run time (3.054536 user, 0.029996 system) [ Run times consist of 0.103 seconds GC time, and 2.982 seconds non-GC time. ] 99.71% CPU 8,632,811,366 processor cycles 922,738,912 bytes consed ; No value CL-USER> (time (progn (grid-aref-loop-setf-foreign 1024 10240) (values))) Evaluation took: 7.608 seconds of real time 7.591846 seconds of total run time (7.552852 user, 0.038994 system) [ Run times consist of 0.506 seconds GC time, and 7.086 seconds non-GC time. ] 99.79% CPU 21,235,314,846 processor cycles 4,026,564,512 bytes consed ; No value CL-USER> (progn (defvar *initial-contents* (make-lists-list 1024 1024)) (values)) ; No value CL-USER> (time (progn (make-array (list 1024 1024) :element-type 'double-float :initial-contents *initial-contents*) (values))) Evaluation took: 0.000 seconds of real time 0.000000 seconds of total run time (0.000000 user, 0.000000 system) 100.00% CPU 1,456 processor cycles 0 bytes consed ; No value CL-USER> (progn (defvar *initial-contents* (make-lists-list 1024 1024)) (values)) ; No value CL-USER> (time (progn (make-array (list 1024 1024) :element-type 'double-float :initial-contents *initial-contents*) (values))) Evaluation took: 0.000 seconds of real time 0.000000 seconds of total run time (0.000000 user, 0.000000 system) 100.00% CPU 1,002 processor cycles 0 bytes consed ; No value CL-USER> (time (progn (grid:make-foreign-array 'double-float :dimensions (list 1024 1024) :initial-contents *initial-contents*) (values))) Evaluation took: 2934.652 seconds of real time 2919.775127 seconds of total run time (2919.301199 user, 0.473928 system) [ Run times consist of 0.199 seconds GC time, and 2919.577 seconds non-GC time. ] 99.49% CPU 8,190,419,850,119 processor cycles 368,726,592 bytes consed ; No value CL-USER> |# -- Sumant Oemrawsingh

1 0

[Antik-devel] error loading gsll with quicklisp
by harven＠free.fr 22 Mar '12

22 Mar '12

Hi, As far as I understand, gsll is being merged with antik, so I am posting here. I am trying to install gsll using quicklisp and I run into an error, as reported in the attachment below. This seems to be related to cffi-grovel, which is installed. $ ls ~/.quicklisp/dists/quicklisp/software/cffi_0.10.6/grovel/asdf.lisp common.h grovel.lisp invoke.lisp package.lisp CL-USER> (find-package 'cffi-grovel) #<PACKAGE "CFFI-GROVEL"> I am on debian squeeze, using sbcl from the repos. I have installed the following packages related to gsl libgsl0-dev, libgsl0ldbl, libffi-dev, gsl-doc-info, gsl-ref-html I also tried to load gsll with asdf, but I run into the same error CL-USER> (asdf:load-system "gsll") Error while trying to load definition for system gsll from pathname /home/harven/.quicklisp/dists/quicklisp/software/gsll-20111105-git/gsll.asd: The variable #:CFFI-GROVEL is unbound. [Condition of type ASDF:LOAD-SYSTEM-DEFINITION-ERROR] Any suggestion ? Regards, -- Harven

2 1

[Antik-devel] Reader macro for sharp-m returns literal foreign-array (patch included)
by James Wright 06 Mar '12

06 Mar '12

Hi, I have encountered the following problem a couple of times before, although not as reproducibly as the memory corruption issue from my previous message: If I use the #m reader macro in a function somewhere, dump an SBCL core image, load the image, and call the function, I get memory corruption. I have mostly dealt with it up until now by not using the #m() notation in anything that I plan to dump to a binary. I believe that the cause is that the #m macro returns a foreign-array object directly (when foreign-array is the default grid type). If it returned a `make-grid' form instead, it would operate identically on e.g. the REPL. It would have the advantage, however, of being usable in a dumped image, since the array would be created at run-time instead of at read-time, and hence no foreign-pointers would be saved into the image. I've attached a patch that changes the #m reader macro to return a creation form instead of a literal. I hope you'll consider adding it. I've pasted a repl session below that demonstrates the difference. Thanks, James ;;; Before the patch GRID> (read-from-string "#m(1d0 2d0)") #m(1.000000000000000d0 2.000000000000000d0) 11 GRID> #m(1d0 2d0) #m(1.000000000000000d0 2.000000000000000d0) ;;; After the patch GRID> (read-from-string "#m(1d0 2d0)") (MAKE-GRID '((FOREIGN-ARRAY) DOUBLE-FLOAT) :INITIAL-CONTENTS '(1.0 2.0)) 11 GRID> #m(1d0 2d0) #m(1.000000000000000d0 2.000000000000000d0)

2 5

[Antik-devel] Memory corruption when loading an SBCL core containing antik (patch included)
by James Wright 06 Mar '12

06 Mar '12

Hi, When I dump an SBCL core from an image that has antik loaded, I get a memory corruption error when loading the core: *** glibc detected *** sbcl: free(): invalid pointer: 0x0808d588 *** This happens even with a minimal project that does nothing but load antik; I've attached a script to reproduce the problem. I did some investigation, and it looks like the problem is the `*formatting-test-grid*' variable in `format-grid.lisp'. It contains a list of grids, including some foreign arrays. Presumably the foreign pointers in these arrays will be pointing to random/unowned memory when the core is loaded. I've run into similar problems before in my own code (attempting to dump a global `*rng*' variable that contained a GSLL random number generator). I've attached a patch that replaces the problem `defparameter' form with a function instead. I can't find any references to this variable in the source tree, so I assume it's just there for manual testing. Thanks, James

2 4

[Antik-devel] documentation: typo in aref example?
by Mirko Vukovic 19 Aug '11

19 Aug '11

Typo on http://www.common-lisp.net/project/antik/antik/The-foreign_002darray-class.… Is there a missing `(something' below: ANTIK-USER> (aref (cl-array *my-fa*) 1) 22.2d0) Mirko

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

antik-devel