Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv5313

Modified Files: CREDITS INSTALL NEWS README TODO ele-bdb.asd elephant.asd Removed Files: Makefile config.lisp Log Message:

Removed config.lisp and all Makefile related build options. The build is now entirely driven from the elephant.asd and ele-bdb.asd files with a simple user customization file, config.sexp, that should be copied to my-config.sexp and customized to the user's environment. Thanks to pinterface for the initial patch supporting this approach. Foreign library loading is also controlled from the asd files now at system load time rather than at compile time.

--- /project/elephant/cvsroot/elephant/CREDITS 2006/02/14 15:28:32 1.7 +++ /project/elephant/cvsroot/elephant/CREDITS 2006/11/11 06:27:37 1.8 @@ -2,12 +2,11 @@ Authors: Andrew Blumberg and Ben Lee ablumberg@common-lisp.net and blee@common-lisp.net

-Current maintainer: Robert L. Read -read@robertlread.net +Current maintainers: Robert L. Read read@robertlread.net + Ian S. Eslick eslick@media.mit.edu

http://www.common-lisp.net/project/elephant

- The CL-SQL based backend was written by Robert L. Read.

Thanks to: @@ -24,7 +23,7 @@

Rafal Strzalinski for the Makefile and package patch

-Bill Clementson for Win32 help and publicity +Bill Clementson for Win32 help and for publicity

The common-lisp.net people for hosting

@@ -41,15 +40,20 @@ Paul Foley for his berkeley-db package (which we didn't use, once we settled on UFFI)

-Various other people whom I'm forgetting who answered my -many idiotic questions - Dan Knapp fixed the fact that nil's were indistinguishable from unbound slots, and proved the system works with SQLite3.

Tayssir John Gabbour has found two bugs on Feb. 14, 2006.

-Ian Eslick wrote src/indexing.lisp, which added major -convenience features for automatically indexing the a slot -in a class. +Ian Eslick wrote src/indexing.lisp, which added major convenience +features for automatically indexing on slot values in a class. He is +also responsible for the refactoring and most of the bug fixes in the +0.6.0 release. + +Vladimir Sedach for the upgrade to Berkeley DB 4.4. + +pinterface@gmail.com provided the basic support in the .asd +files for removing the Makefile from the build process. + +Marco Baringer provided a partial patch for 64-bit support.

--- /project/elephant/cvsroot/elephant/INSTALL 2006/02/22 20:18:51 1.17 +++ /project/elephant/cvsroot/elephant/INSTALL 2006/11/11 06:27:37 1.18 @@ -46,11 +46,11 @@ Long Instructions -----------------

-I assume you have a supported lisp with asdf. +For SBCL, CMUCL, Allegro 7.0+, MCL and CLISP:

0) Unpack Elephant. I put mine in the directory

-/usr/local/share/common-lisp/elephant-0.3/ +/usr/local/share/common-lisp/elephant-0.6/

1) Install ASDF.

@@ -59,7 +59,7 @@

2) Install UFFI.

-3) Install a backend: Either Berkeley DB 4.3, PostGresql, or SQLite 3. +3) Install a backend: Either Berkeley DB 4.4, PostGresql, or SQLite 3.

------- SQL @@ -69,31 +69,27 @@ other the heading "SQL-BACK-END".

------------- -Berkeley 4.3: +Berkeley 4.4: -------------

+(Note: 0.6.0 users used 4.3; upgrade to 4.4 and run 0.6.1+ and + your existing DB will automatically upgrade when the DB is opened) + Under Un*x, you may actually already have this installed, though it may be compiled with funny options, so if things don't work you may want to try to start from scratch. FreeBSD has a port for this, as I'm sure do other BSDs (including Darwin/Fink.) Take note of where libdb.so and db.h are installed, usually:

- /usr/local/BerekleyDB.4.3/lib/libdb.so and - /usr/local/BerekleyDB.4.3/include/db.h, or - - /usr/local/lib/db42/libdb.so and - /usr/local/include/db42/db.h.) - -a) Edit Makefile variable DB43DIR and DB43INC, DB43LIB if necessary - -This makes sure that the build process can find your files. -You can test that the build works by calling: + /usr/local/BerekleyDB.4.4/lib/libdb.so and + /usr/local/BerekleyDB.4.4/include/db.h, or

-'make bdb'. + /usr/local/lib/db44/libdb.so and + /usr/local/include/db44/db.h.)

-b) Also edit the variable *sleepycat-foreign-library-path* in +a) Site specific configuration

- config.lisp + config.sexp

to point to your local distribution of the Berkeley DB libraries

--- /project/elephant/cvsroot/elephant/NEWS 2005/11/23 18:17:52 1.8 +++ /project/elephant/cvsroot/elephant/NEWS 2006/11/11 06:27:37 1.9 @@ -1,3 +1,9 @@ +April, 2006 - Elephant 0.6.0 released by +Robert Read and Ian Eslick. Supports class slot +indexing and benefits from a clean refactoring +of backends and a host of other small changes. +This is a solid BETA release. + November 30, 2005 - Elephant 0.3.0 released by the new maintainer, Robert L. Read, providing support for relational database backends, repository --- /project/elephant/cvsroot/elephant/README 2006/11/10 01:48:49 1.5 +++ /project/elephant/cvsroot/elephant/README 2006/11/11 06:27:37 1.6 @@ -67,11 +67,14 @@

HTML docs and texinfo sources can be found in the docs/ directory.

------- -Design ------- +------------------------ +Design and Development +------------------------

-See NOTES. +See NOTES for some design internals. + +If you want to contribute, see TODO for current feature release plans +and other things that need to be done.

------- Authors --- /project/elephant/cvsroot/elephant/TODO 2006/11/10 01:48:49 1.26 +++ /project/elephant/cvsroot/elephant/TODO 2006/11/11 06:27:37 1.27 @@ -1,5 +1,5 @@

-September 1st, 2006 +Last updated: November 11, 2006

Ongoing release plan notes:

@@ -7,10 +7,9 @@ -----------------------------------------------------------

Bugs or Observations: -- New build interface - 64-bit support (from Marco) -- pthreads issue? -- MCL compatibility issues +? MCL compatibility issues +- Windows support for asdf-based library builds?

Stability: - Review all the NOTE comments in the code @@ -35,6 +34,7 @@ Multi-threading operation: - Make elephant threads appropriately bind dynamic variables - Verify that operations such as indexing are thread safe +- Verify that serialization is thread safe

BDB Features: ~ Automatically run db_deadlock when opening a bdb backend? Requires path to @@ -57,8 +57,6 @@ - Reclaim table storage on index drop? It's nice to be able to reconnect sometimes! Perhaps an API command that allows explicit dropping of tables for a class and a policy parameter that determines if this is the default? -- Should we delete slot-values in the db when redefining classes, currently those values - stay around - probably indefinitely unless we GC

Performance: - Metering and understanding locking issues. Large transactions seem @@ -84,6 +82,7 @@

0.6.1 - Features COMPLETED to date ---------------------------------- +x New build interface; all-lisp compilation (sans win32) x Ensure serialization is multi-threaded and efficient x Determine how to detect deadlock conditions as an optional run-safe mode? x BDB overwrite of values makes DB grow @@ -93,6 +92,9 @@ [Ditto above] x Update to support BDB 4.4 x Add ability from within lisp to reclaim DB space after deleting btree key-value pairs +x Should we delete slot-values in the db when redefining classes, currently those values + stay around - probably indefinitely unless we GC (no, we'll resolve this with a + stop-and-copy GC - need to make migration bookkeeping more efficient)

0.6.2 - Advanded work, low-hanging fruit (Fall '06) -------------------------------------------------- --- /project/elephant/cvsroot/elephant/ele-bdb.asd 2006/06/01 12:55:43 1.10 +++ /project/elephant/cvsroot/elephant/ele-bdb.asd 2006/11/11 06:27:37 1.11 @@ -19,65 +19,47 @@ (in-package :cl-user)

(defpackage ele-bdb-system - (:use :cl :asdf)) + (:use :cl :asdf :elephant-system))

(in-package :ele-bdb-system)

-(eval-when (:compile-toplevel :load-toplevel :execute) - ;; We need this dependency satisfied to compute what to do for C files - (unless (find-package 'uffi) - (asdf:operate 'asdf:load-op 'uffi))) - -(defclass bdb-c-source (c-source-file) - ()) - -(defparameter *root-dir* (pathname-directory *load-truename*)) - -(defparameter *library-file-dir* (append (pathname-directory *load-truename*) - (list "src" "db-bdb"))) - -;; Compile foreign library on non-win32 platforms - -(defmethod output-files ((o compile-op) (c bdb-c-source)) - (let ((library-file-type - (funcall (intern (symbol-name '#:default-foreign-library-type) - (symbol-name '#:uffi))))) - (list (make-pathname :name (component-name c) - :type library-file-type - :directory *library-file-dir*)))) - -(defmethod perform ((o compile-op) (c bdb-c-source)) - (unless (operation-done-p o c) - #-(or win32 windows) - (unless (zerop (uffi:run-shell-command - (format nil - #-freebsd "cd ~A; make bdb" - #+freebsd "cd ~A; gmake bdb" - (make-pathname :directory *root-dir*)))) - (format t "Couldn't build library from libsleepycat.c via 'make bdb'~%") - (error 'operation-error :component c :operation o)))) - -(defmethod operation-done-p ((o compile-op) (c bdb-c-source)) - (or (let ((lib (make-pathname :defaults (component-pathname c) - :type (uffi:default-foreign-library-type)))) - (and (probe-file lib) (probe-file (component-pathname c)) - (> (file-write-date lib) (file-write-date (component-pathname c))))))) - -;; Load op - ensure that foreign library is loaded - -(defmethod perform ((o load-op) (c bdb-c-source)) - "Nothing to do!" - t) - -(defmethod operation-done-p ((o load-op) (c bdb-c-source)) - "Operation is done when the foreign library is loaded which should - happen when we compile the interface lisp file" - (and (and (find-package '#:sleepycat) - (ignore-errors - (symbol-function (intern (symbol-name '#:%db-strerror) - (find-package '#:sleepycat))))) - t)) - +;; +;; User parameters (bdb root and pthread, if necessary) +;; + +(defparameter *bdb-config* nil) + +(defun get-config-option (option component) + (unless *bdb-config* + (with-open-file (config (make-pathname :defaults (asdf:component-pathname + (asdf:component-system component)) + :name "my-config" + :type "sexp")) + (setf *bdb-config* (read config)))) + (cdr (assoc option *bdb-config*))) + +;; +;; Compile bdb lib and load libraries +;; + +(defclass bdb-c-source (elephant-c-source) ()) + +(defmethod compiler-options ((compiler (eql :gcc)) (c bdb-c-source) &key &allow-other-keys) + (let* ((include (merge-pathnames (get-config-option :sleepycat-root c) "include")) + (lib (merge-pathnames (get-config-option :sleepycat-root c) "lib"))) + (append (list (format nil "-L~A" lib) (format nil "-I~A" include)) + (call-next-method) + (list "-ldb")))) + +(defmethod foreign-libraries-to-load-first ((c bdb-c-source)) + (remove-if #'(lambda (x) (null (car x))) + (list + (cons (get-config-option :pthread-lib c) "pthread") + (cons (get-config-option :sleepycat-lib c) "sleepycat")))) + +;; +;; System definition +;;

(defsystem ele-bdb :name "elephant" @@ -101,4 +83,3 @@ :serial t)))) :depends-on (:uffi :elephant))

- --- /project/elephant/cvsroot/elephant/elephant.asd 2006/11/10 01:48:49 1.19 +++ /project/elephant/cvsroot/elephant/elephant.asd 2006/11/11 06:27:37 1.20 @@ -19,63 +19,113 @@ (in-package :cl-user)

(defpackage elephant-system - (:use :cl :asdf)) + (:use :cl :asdf) + (:export :elephant-c-source :compiler-options :foreign-libraries-to-load-first))

(in-package :elephant-system)

-(eval-when (:compile-toplevel :load-toplevel :execute) - ;; We need this dependency satisfied to compute what to do for C files - (unless (find-package 'uffi) - (asdf:operate 'asdf:load-op 'uffi))) - -(defclass elephant-util-c-source (c-source-file) - ()) - -(defparameter *root-dir* (pathname-directory *load-truename*)) - -(defparameter *library-file-dir* (append (pathname-directory *load-truename*) - (list "src" "memutil"))) - -;; Compile foreign library on non-win32 platforms - -(defmethod output-files ((o compile-op) (c elephant-util-c-source)) - (let* ((library-file-type - (funcall (intern (symbol-name '#:default-foreign-library-type) - (symbol-name '#:uffi))))) - (list (make-pathname :name (component-name c) - :type library-file-type - :directory *library-file-dir*)))) - -(defmethod perform ((o compile-op) (c elephant-util-c-source)) - (unless (operation-done-p o c) - #-(or win32 windows) - (unless (zerop (uffi:run-shell-command - (format nil - #-freebsd "cd ~A; make" - #+freebsd "cd ~A; gmake" - (make-pathname :directory *root-dir*)))) - (error 'operation-error :component c :operation o)))) - -(defmethod operation-done-p ((o compile-op) (c elephant-util-c-source)) - (or (let ((lib (make-pathname :defaults (component-pathname c) - :type (uffi:default-foreign-library-type)))) - (and (probe-file lib) (probe-file (component-pathname c)) - (> (file-write-date lib) (file-write-date (component-pathname c))))))) - -;; Load op - ensure that foreign library is loaded - -(defmethod perform ((o load-op) (c elephant-util-c-source)) - "Nothing to do!" - t) - -(defmethod operation-done-p ((o load-op) (c elephant-util-c-source)) - "Operation is done when the foreign library is loaded which should - happen when we compile the interface lisp file" - (and (find-package "ELEPHANT-MEMUTIL") - (ignore-errors - (symbol-function (intern "COPY-BUFS" - (find-package "ELEPHANT-MEMUTIL")))) - t)) +;; +;; Simple lisp/asdf-based make utility for elephant c files +;; + +(defvar *c-compilers* + '((:gcc . "/usr/bin/gcc") + (:msvc . "")) + "Associate compilers with platforms for compiling libmemutil/libsleepycat") + +(defvar *compiler* + #-(or win32 windows) :gcc + #+(or win32 windows) :msvc) + +(defgeneric compiler-options (compiler c-source-file &key input-file output-file) + (:documentation "Returns a list of options to pass to <compiler>")) + +(defgeneric foreign-libraries-to-load-first (c-source-file) + (:documentation "Provides an alist of foreign-libraries to load and the modules to load them into. Similar to (input-files load-op), but much more specific")) + + +(defun uffi-funcall (fn &rest args) + "Simplify uffi funcall, first ensure uffi is loaded" + (unless (find-package :uffi) + (asdf:operate 'asdf:load-op :uffi)) + (apply (find-symbol (symbol-name fn) (symbol-name :uffi)) args)) + +;; +;; Basic utilities for elephant c files +;; + +(defclass elephant-c-source (c-source-file) ()) + +;; COMPILE + +(defmethod output-files ((o compile-op) (c elephant-c-source)) + "Compute the output files (for dependency tracking), here we assume + a library with the same name and a platform dependant extension" + (list (make-pathname :name (component-name c) + :type (uffi-funcall :default-foreign-library-type) + :defaults (component-pathname c)))) + +(defmethod perform ((o compile-op) (c elephant-c-source)) + "Run the appropriate compiler for this platform on the source, getting + the specific options from 'compiler-options method. Default options + can be overridden or augmented by subclass methods" + (unless (zerop (run-shell-command + "~A ~{~A ~}" + (cdr (assoc *compiler* *c-compilers*)) + (compiler-options + *compiler* + c + :input-file (namestring (component-pathname c)) + :output-file (namestring (first (output-files o c)))))) + (error 'operation-error :component c :operation o))) + +(defmethod operation-done-p ((o compile-op) (c elephant-c-source)) + "Is the first generated library more recent than the source file?" + (let ((lib (first (output-files o c)))) + (and (probe-file (component-pathname c)) + (probe-file lib) + (> (file-write-date lib) (file-write-date (component-pathname c)))))) + +(defmethod compiler-options ((compiler (eql :gcc)) (c elephant-c-source) &key input-file output-file) + "Default compile and link options to create a library; no -L or -I options included; math lib as default" + (unless (and input-file output-file) + (error "Must specify both input and output files")) + (list + #-(or darwin macosx) "-shared" + #+(or darwin macosx) "-bundle" + "-Wall" + "-fPIC" + "-O3" + "-o" output-file + input-file + "-lm")) + +(defmethod compiler-options ((compiler (eql :msvc)) (c elephant-c-source) &key input-file output-file) + (error "MSVC compiler option not supported yet")) + +;; LOAD + +(defmethod perform ((o load-op) (c elephant-c-source)) + ;; Load any required external libraries + (dolist (file+module (foreign-libraries-to-load-first c)) + (destructuring-bind (file . module) file+module + (or (uffi-funcall :load-foreign-library file :module module) + (error "Could not load ~A into ~A" file module)))) + ;; Load the compiled libraries + (dolist (file (output-files (make-instance 'compile-op) c)) + (format t "~A" file) + (or (uffi-funcall :load-foreign-library file :module (component-name c)) + (error "Could not load ~A" file)))) + +(defmethod operation-done-p ((o load-op) (c elephant-c-source)) + nil) + +(defmethod foreign-libraries-to-load-first ((c elephant-c-source)) + nil) + +;; +;; System definition +;;

(defsystem elephant :name "elephant" @@ -90,13 +140,12 @@ :components ((:module memutil :components - ((:elephant-util-c-source "libmemutil") + ((:elephant-c-source "libmemutil") (:file "memutil")) :serial t) (:module elephant :components ((:file "package") - (:file "config" :pathname "../../config.lisp") (:file "variables") #+cmu (:file "cmu-mop-patches") #+openmcl (:file "openmcl-mop-patches") @@ -113,6 +162,5 @@ (:file "backend")) :serial t :depends-on (memutil))))) - :depends-on (:uffi :closer-mop)) - + :depends-on (:uffi))