April 2007 - elephant-cvs@common-lisp.net

[elephant-cvs] CVS elephant/doc
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv575 Modified Files: elephant-design.texinfo Log Message: Few tweaks to docs to build them; updated docs on website to match latest checkins --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/27 03:14:55 1.5 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/30 02:33:17 1.6 @@ -13,7 +13,7 @@ The architecture has been carefully modularized: -(a)image{ArchDiagram,,4.5in,[Architecture Diagram],.png} +(a)image{ArchDiagram,,4.5in,[Architecture Diagram],png} To get a feeling for what is happening inside elephant, it is probably best to walk through the various major protocols to see how these @@ -176,7 +176,7 @@ The BTree protocol almost requires an actual BTree implementation to be at all efficient. Keys and values need to be accessible via the cursor API, which means they need to be walked linearly in the sort -order of the keys (described in @xref{Persistent BTrees}). +order of the keys (described in @ref{Persistent BTrees}). An indexed BTree automatically maintains a hash table of the indices defined on it so that users can access them by mapping or

1 0

[elephant-cvs] CVS elephant
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv11649 Modified Files: elephant.asd Log Message: Typo --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:38:29 1.47 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:54:03 1.48 @@ -256,7 +256,7 @@ (error "Could not load ~A" file)) (format t "Loaded ~A~%" file)) (let* ((root-dir (asdf:component-pathname (asdf:component-system c))) - (root-library-file (merge-pathnames (file-namestring file) root-path))) + (root-library-file (merge-pathnames (file-namestring file) root-dir))) (or (uffi-funcall :load-foreign-library root-library-file :module (component-name c))

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv8074/src/elephant Modified Files: variables.lisp Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/25 02:28:02 1.17 +++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/30 00:38:31 1.18 @@ -42,7 +42,7 @@ ;;;; General support for user configurable parameters (defvar *user-configurable-parameters* - '((:map-using-degree2 *map-using-degree2*) + '((:berkeley-db-map-degree2 *map-using-degree2*) (:berkeley-db-cachesize *berkeley-db-cachesize*))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

1 0

[elephant-cvs] CVS elephant
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv8074 Modified Files: config.sexp ele-clsql.asd elephant.asd Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/config.sexp 2007/04/25 02:27:56 1.11 +++ /project/elephant/cvsroot/elephant/config.sexp 2007/04/30 00:38:29 1.12 @@ -1,48 +1,63 @@ -#+(or sbcl allegro) -((:berkeley-db-include-dir . "/opt/local/include/db45/") +#+(and (or sbcl allegro) (not mswindows) (not windows)) +((:compiler . :gcc) + (:berkeley-db-include-dir . "/opt/local/include/db45/") (:berkeley-db-lib-dir . "/opt/local/lib/db45/") (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) (:berkeley-db-cachesize . 20971520) - (:clsql-lib . nil) - (:compiler . :gcc)) + (:berkeley-db-map-degree2 . t) + (:clsql-lib-paths . nil) + (:prebuilt-libraries . nil)) + #+openmcl -((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") +((:compiler . :gcc) + (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.dylib") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :gcc)) + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:prebuilt-libraries . nil) + (:clsql-lib-paths . nil)) #+(and lispworks (not mswindows)) -((:berkeley-db-include-dir . "/opt/local/include/db45/") - (:berkeley-db-lib-dir . "/opt/local/lib/db45/") - (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") - (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :gcc)) +((:compiler . :gcc) + (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.dylib") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:prebuilt-libraries . nil) + (:clsql-lib-paths . nil)) #+(or mswindows windows) -((:berkeley-db-include-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/include/") +((:compiler . :cygwin) + (:berkeley-db-include-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/include/") (:berkeley-db-lib-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/") (:berkeley-db-lib . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/libdb45.dll") (:berkeley-db-deadlock . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/db_deadlock.exe") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :cygwin)) + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:clsql-lib-paths . nil) + (:prebuilt-libraries . t)) ;; Berkeley 4.5 is required, each system will have different settings for ;; these directories, use this as an indication of what each key means ;; -;; Typical pthread settings are: /lib/tls/libpthread.so.0 -;; -;; nil means that the library in question is not loaded -;; -;; NOTE: The latest SBCL (0.9.17+) on linux no longer needs the pthread library, -;; it is statically linked against it now with the new thread support +;; :prebuilt-libraries is true by default for windows machines. It causes +;; the library loader to look in the elephant root directory for the shared +;; libraries. (nil or t) +;; +;; :clsql-lib-paths tell clsql where to look for which ever SQL distribution +;; library files you need it to look for. For example... +;; +;; (:clsql-lib-paths . ("/Users/me/Work/SQlite3/" "/Users/me/Work/Postgresql/")) +;; +;; :pthread-lib is deprecated, for old versions of sbcl prior to 0.9.17 that +;; did not have pthreads compiled in. If you are using an old version, we +;; recommend that you upgrade! +;; Typical pthread settings were: /lib/tls/libpthread.so.0 ;; ;; :compiler options are ;; :gcc (default: for unix platforms with /usr/bin/gcc) @@ -52,16 +67,18 @@ ;; Additional supported parameters include: ;; ;; -;; :berkeley-db-cachesize - an integer indicating the number of bytes -;; for the page cache, default 10MB which is -;; about enough storage for 10k-30k indexed -;; persistent objects -;; -;; :map-using-degree2 - Boolean parameter that indicates whether map -;; operations lock down the btree for the entire -;; transaction or whether they allow other -;; transactions to add/delete/modify values -;; before the map operation is completed. The -;; map operation remains stable and any writes -;; are kept transactional, see user manual as -;; well as berkeley DB docs for more details +;; :berkeley-db-cachesize +;; An integer indicating the number of bytes +;; for the page cache, default 20MB which is +;; about enough storage for 10k-30k indexed +;; persistent objects +;; +;; :berkeley-db-map-degree2 +;; Boolean parameter that indicates whether map +;; operations lock down the btree for the entire +;; transaction or whether they allow other +;; transactions to add/delete/modify values +;; before the map operation is completed. The +;; map operation remains stable and any writes +;; are kept transactional, see user manual as +;; well as berkeley DB docs for more details. --- /project/elephant/cvsroot/elephant/ele-clsql.asd 2007/03/23 16:31:52 1.10 +++ /project/elephant/cvsroot/elephant/ele-clsql.asd 2007/04/30 00:38:29 1.11 @@ -16,29 +16,33 @@ ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. -;; (eval-when (:compile-toplevel :load-toplevel :execute) -;; (unless (find-package 'clsql) -;; (asdf:operate 'asdf:load-op 'clsql))) - -;; ;; clsql needs to know how to find libaries to various databases.... -;; (defvar *clsql-foreign-lib-path* "/usr/lib") -;; (defvar *elephant-lib-path* "/usr/local/share/common-lisp/elephant/") -;; (clsql:push-library-path *clsql-foreign-lib-path*) -;; (clsql:push-library-path *elephant-lib-path*) -;; ) - -;;(defparameter *clsql-foreign-lib-path* #p"/usr/lib") -;;(defparameter *elephant-lib-path* #p"/usr/local/share/common-lisp/elephant/") - -;;(defmethod asdf:perform :after ((o asdf:load-op) -; (c (eql (asdf:find-system 'clsql)))) -;; (let ((plp (find-symbol (symbol-name '#:push-library-path) -;; (find-package 'clsql)))) -;; (funcall plp -;; *clsql-foreign-lib-path*) -;; (funcall plp -;; *elephant-lib-path*) -;;)) +(in-package :cl-user) + +;; Forward definition +(defpackage elephant-system + (:use :cl :asdf) + (:export :elephant-c-source :compiler-options :foreign-libraries-to-load-first :get-config-option)) + +(defpackage ele-clsql-system + (:use :cl :asdf :elephant-system)) + +(in-package :ele-clsql-system) + +;; +;; Make sure asdf is loaded prior to evaluating the rest of the file (tacky) +;; +(eval-when (:compile-toplevel :load-toplevel :execute) + (unless (find-package 'clsql) + (asdf:operate 'asdf:load-op 'clsql))) + +(defmethod asdf:perform :after ((o asdf:load-op) + (c (eql (asdf:find-system 'clsql)))) + (let ((paths (get-config-option :clsql-lib-paths (find-system :elephant))) + (plp (find-symbol (symbol-name '#:push-library-path) + (find-package 'clsql)))) + (loop for path in paths do + (format t "Pushing ~A onto clsql::*FOREIGN-LIBRARY-SEARCH-PATHS*~%" path) + (funcall plp (pathname path))))) (defsystem ele-clsql :name "elephant" --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/29 02:10:54 1.46 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:38:29 1.47 @@ -94,6 +94,7 @@ "Run the appropriate compiler for this platform on the source, getting the specific options from 'compiler-options method. Default options can be overridden or augmented by subclass methods" + (unless (get-config-option :prebuilt-libraries c) #+(or mswindows windows) (progn (let* ((pathname (component-pathname c)) @@ -144,7 +145,7 @@ (compiler-options (c-compiler c) c :input-file (namestring (component-pathname c)) :output-file (namestring (first (output-files o c)))))) - (error 'operation-error :component c :operation o))) + (error 'operation-error :component c :operation o)))) #| (defmethod perform ((o compile-op) (c elephant-c-source)) @@ -247,9 +248,21 @@ (error "Could not load ~A into ~A" file module))))) ;; Load the compiled libraries (dolist (file (output-files (make-instance 'compile-op) c)) - (format t "Loading ~A~%" file) - (or (uffi-funcall :load-foreign-library file :module (component-name c)) - (error "Could not load ~A" file)))) + (format t "Attempting to load ~A...~%" (file-namestring file)) + (if (and (probe-file file) + (not (get-config-option :prebuilt-libraries c))) + (progn + (or (uffi-funcall :load-foreign-library file :module (component-name c)) + (error "Could not load ~A" file)) + (format t "Loaded ~A~%" file)) + (let* ((root-dir (asdf:component-pathname (asdf:component-system c))) + (root-library-file (merge-pathnames (file-namestring file) root-path))) + (or (uffi-funcall :load-foreign-library + root-library-file + :module (component-name c)) + (error "Output file ~A not found in elephant root" (namestring root-library-file))) + (format t "Loaded ~A~%" root-library-file))))) + (defmethod operation-done-p ((o load-op) (c elephant-c-source)) nil) @@ -264,7 +277,7 @@ (defsystem elephant :name "elephant" :author "Ben Lee <blee(a)common-lisp.net>" - :version "0.6.1" + :version "0.9" :maintainer "Robert Read <read(a)robertlread.net>" :licence "LLGPL" :description "Object database for Common Lisp"

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv8074/doc Modified Files: installation.texinfo user-guide.texinfo Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/29 21:54:51 1.14 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/30 00:38:31 1.15 @@ -246,6 +246,10 @@ @comment node-name, next, previous, up @section Setting up Berkeley DB +We recommend that you download and build a distribution from Oracle. +Some problems have been reported with linking to Debian, Cygwin or +other packages. This is especially true for Windows users. + Beyond ensuring that the file ``my-config.sexp'' points to your BDB installation directories and files, nothing else should be required to configure the example that uses a local ``testdb'' directory as a @@ -428,13 +432,20 @@ @comment node-name, next, previous, up @section Elephant on Windows -The build process on Windows currently only works with GCC under Cygwin. -The process can be a bit tricky, so if it doesn't work out of the box, -we recommend that you download the DLLs from the Elephant website -download page (@url{http://www.common-lisp.net/project/elephant/downloads.html'}). - -Unpack the .zip file into the elephant root directory. Then run the batch file -``copy-dll.bat'' to move them into their proper directory. +The build process on Windows currently only works with GCC under +Cygwin. The process can be a bit tricky, so if it doesn't work out of +the box or you don't want to install cygwin, we recommend that you +download the DLLs from the Elephant website download page +(@url{http://www.common-lisp.net/project/elephant/downloads.html'}). + +Unpack the .zip file into the elephant root directory. Ensure that +your @code{my-config.sexp} file configuration for Windows has +@code{:prebuilt-binaries} set to ``t'' so it will know to look in +the elephant root during the asdf loading process. + +For Berkeley DB users we recommend downloading the Windows binary +distribution of Berkeley DB 4.5 to minimize any potential linking +issues. @node Test Suites @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 14:24:57 1.21 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/30 00:38:31 1.22 @@ -1555,7 +1555,7 @@ described in config.sexp. They are: @itemize -@item @code{:map-using-degree2} - Improve the efficiency of cursor traversals +@item @code{:berkeley-db-map-degree2} - Improve the efficiency of cursor traversals in the various mapping functions. Defaults to true, meaning a value you just read while mapping may change before the traversal is done. So if you operate only on the current cursor location, you are

1 0

[elephant-cvs] CVS elephant/src/utils
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/utils In directory clnet:/tmp/cvs-serv3086/src/utils Modified Files: os.lisp Log Message: Fix for lispworks deadlock detect (silently fails instead of asserting an error) --- /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/29 02:10:56 1.4 +++ /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/29 21:54:51 1.5 @@ -52,7 +52,7 @@ (in-directory (directory) (ccl:run-program program args :wait nil)) #+lispworks - (apply #'sys::call-system + (funcall #'sys::call-system (format nil "~a~{ '~a'~} &" program args) :current-directory directory :wait nil)

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv3086/doc Modified Files: installation.texinfo Log Message: Fix for lispworks deadlock detect (silently fails instead of asserting an error) --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 14:24:57 1.13 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/29 21:54:51 1.14 @@ -36,8 +36,10 @@ Elephant supports SBCL, Allegro, Lispworks, OpenMCL and CMUCL. Each lisp is supported on each of the platforms it runs on: Mac OS X, Linux and Windows. As of release 0.6.1, both 32-bit and 64-bit systems -should be supported. Elephant has a small developer base and as of -the writing of this manual, there are: +should be supported. + +Due to the small number of developers and the large number of configurations +providing full test coverage is problematic. There are: @enumerate @item Five lisp environments @@ -46,16 +48,14 @@ @item Three data store configurations: Berkeley DB, SQLite3 and Postgresql @end enumerate -This means that the total number of combinations that should be tested -comes to: +which means that the total number of combinations to be tested could be as +much as: @math{lisps * os * radix * dstore = 5 * 3 * 2 * 3 = 90 configurations} -Of course not all of these combinations are valid, but the -implications of these combinatorics is that not every combination will -be tested in any given release. The developers and active user base -currently cover all three data store configurations on the following -platforms: +Not all of these combinations are valid, but the implication is that +not every combination will be tested in any given release. The +developers and user base regularly use the following platforms @itemize @item 32/64-bit SBCL on Linux and Mac OS X @@ -63,13 +63,15 @@ @item 32-bit Allegro on Mac OS X @end itemize -The developers will do their best to accomodate users who are keen to -test other combinations, but practically these configurations will be -the most stable and reliable. Elephant is becoming quite stable in -general, so don't be afriad to try an unemphasized combination - -chances are it is just a little more work to bring it up. In particular, -Elephant can probably work with MySQL or Oracle with just a little work, -but nobody has asked for this yet. +The CLSQL backend is used predominantly under SBCL on Linux and Mac OS +X at the time of writing. The developers will do their best to +accomodate users who are keen to test other combinations, but the +above configurations will be the most stable and reliable. + +Elephant is now quite stable in general, so don't be afraid to try an +unemphasized combination - chances are it is just a little more work +to bring it up. In particular, Elephant can probably work with MySQL +or Oracle with just a little work, but nobody has asked for this yet. @subsection Library dependencies @@ -77,7 +79,7 @@ @enumerate @item asdf -- @uref{http://www.cliki.net/asdf} -@item uffi -- version 1.5.17 or later, @uref{http://uffi.b9.com/} or @uref{http://www.cliki.net/UFFI} +@item uffi -- requires version 1.5.18 or later, @uref{http://uffi.b9.com/} or @uref{http://www.cliki.net/UFFI} @item cl-base64 -- @uref{http://www.cliki.net/cl-base64} @item gcc -- Your system needs GCC (or Cygwin) to build the Elephant C-based serializer library. (Precompiled DLL's are available for Windows platforms on the @uref{http://www.common-lisp.net/project/elephant/downloads.html, download page}. @item rt -- The RT regression test sytem is required to run the test suite: @uref{http://www.cliki.net/RT} @@ -86,7 +88,8 @@ Follow the instructions at these URLs to download and setup the libraries. (Note: uffi and cl-base64 are @uref{http://www.cliki.net/ASDF-Install, asdf-installable} for those -of you with asdf-install on your system) +of you with asdf-install on your system). Elephant, however, is not +asdf-installable today. In addition to these libraries, each data store has their own dependencies as discussed in @ref{Berkeley DB} and @@ -97,10 +100,11 @@ @section Configuring Elephant Before you can load the elephant packages into your running lisp, you -need to setup the configuration file. First, copy the reference file -config.sexp from the root directory to my-config.sexp. my-config.sexp -contains a lisp reader-formatted list of key-value pairs that tells -elephant where to find various libraries, how to build libraries, etc. +need to setup the configuration file. Copy the reference file +config.sexp from the root directory to my-config.sexp in the root +directory. my-config.sexp contains a lisp reader-formatted list of +key-value pairs that tells elephant where to find various libraries +and how to build them. For example: @@ -110,8 +114,6 @@ (:berkeley-db-lib-dir . "/opt/local/lib/db45/") (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) - (:clsql-lib . nil) (:compiler . :gcc)) @end lisp @@ -132,9 +134,9 @@ The config.sexp file contains a set of example configurations to start from, but you will most likely need to modify it for your system. -Elephant has one small C library that it uses for binary serialization -which means that you need to have gcc in your path (@pxref{Elephant on -Windows} for exceptions on the Windows platform). +Elephant has one small C library that it uses for binary +serialization. This means that you need to have gcc in your path +(@pxref{Elephant on Windows} for exceptions on the Windows platform). @node Loading Elephant @comment node-name, next, previous, up @@ -152,12 +154,11 @@ This will load the cl-base64 and uffi libraries. It will also automatically compile and load the C library. The build process no -longer depends on a Makefile. This build process has been verified on -most platforms, but if you have a problem please report it, and any -output you can capture, to the developers at -@email{elephant-devel@@common-lisp.net}. We will update the FAQ at -@uref{http://trac.common-lisp.net/elephant} with common problems users -run into. +longer depends on a Makefile and has been verified on most platforms, +but if you have a problem please report it, and any output you can +capture, to the developers at @email{elephant-devel@@common-lisp.net}. +We will update the FAQ at @uref{http://trac.common-lisp.net/elephant} +with common problems users run into. @subsection Two-Phase Load Process @@ -170,8 +171,7 @@ (NOTE: There are some good reasons and not so good reasons for this process. One reason you cannot load ele-bdb.asd directly as it depends on lisp code defined in elephant.asd. We decided not to fix -this in this release although later releases may avoid the oddity of -the two phase loading) +this in the 0.9 release although later releases may improve on this). @subsection Packages @@ -199,7 +199,8 @@ (:use :common-lisp :elephant)) @end lisp -The symbols that are imported are captured in @ref{User API Reference} +The imported symbols are all that is needed to control Elephant +databases and are documented in detail in @ref{User API Reference} @subsection Opening a Store @@ -224,19 +225,18 @@ The first time you load a specific data store, Elephant will call ASDF to load all the specified data store's dependencies, connect to a database and return the @code{store-controller} subclass instance for -that backend. +that data store. @node Berkeley DB @comment node-name, next, previous, up @section Berkeley DB -The Berkeley DB Data Store started out as a very simple data dictionary in the -Berkeley Unix operating system. There are many ``Xdb'' systems that use the -same API, or a similarly one. A commercial version of the BDB was provided by -Sleepycat Software, and this product has since been purchased by Oracle corporation, -and is currently distributed under a similar licensing arrangement, with commercial -support also available. Please follow the download and installation procedures -defined here: +The Berkeley DB Data Store started out as a very simple data +dictionary in the Berkeley Unix operating system. There are many +``Xdb'' systems that use the same API, or a similarly one. A free for +non-commercial use version of Berkeley DB is provided by Oracle +corporation with commercial licenses available. Please follow the +download and installation procedures defined here: http://www.oracle.com/technology/products/berkeley-db/db/index.html @@ -246,68 +246,71 @@ @comment node-name, next, previous, up @section Setting up Berkeley DB -Beyond ensuring that the file ``my-config.sexp'' points correctly -to your BDB installation, nothing else should be required to configure -the example that uses a local ``testdb'' directory as a dabase (under ``tests'') -in the top-level Elephant directory. +Beyond ensuring that the file ``my-config.sexp'' points to your BDB +installation directories and files, nothing else should be required to +configure the example that uses a local ``testdb'' directory as a +dabase (under ``tests'') in the top-level Elephant directory. On one Fedora based system, the ``my-config.sexp'' file looked like this: @lisp ((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include") - (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") - (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") - (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") (:pthread-lib . nil) - (:clsql-lib . "/usr/local/share/common-lisp/") - (:compiler . :gcc)) + (:clsql-lib . "/usr/local/share/common-lisp/") + (:compiler . :gcc)) @end lisp The @ref{Test Suites} give a nice example of using BDB by running the test using the specification: @lisp -'(:BDB "/home/me/db/testdb/")) +'(:BDB "<elephant-root>/tests/testdb/") @end lisp -Once you start working on an application, you will want to change this path -to a directory that make sense for you application, and use that as -the specification passed to -@lisp open-store -@end lisp -in your application. +Once you start working on an application, you will want to change the +path to a directory that is appropriate for your application, and use +that as the specification passed to @code{ open-store } on application +startup. @node Upgrading Berkeley DB Databases @comment node-name, next, previous, up @section Upgrading Berkeley DB Databases -At regular intervals, Elephant will require a specific version of Berkeley DB -because it does not parse the header files which tend to change with each release. -The patches are usually minor, but sometimes Elephant also depends on new -features of Berkeley DB. In this case, you have to update both the Berkeley DB -database files as well as any data Elephant has built within it. That creates -some special constraints for upgrading databases. +When there is a new release of Elephant, it will depend on a new +version of Berkeley DB. If so, you must upgrade your BDB databases to +use the new version Elephant. This forced upgrade is a consequence of +Elephant not parsing the BDB header files which tend to change various +important constants with each release. These patches are usually +minor. Upgrading also happens because Elephant tries to leverage new +features of Berkeley DB. + +The rest of this section talks about how to upgrade your existing +Berkeley DB databases, opening them in the new Elephant version and +migrating them to a newly created Elephant database. @subsection Upgrading to 0.9 -This section outlines how to upgrade from Elephant code base version 0.6.0 -using Berkeley DB 4.3. +This section outlines how to upgrade from Elephant version 0.6.0 and +Berkeley DB 4.3. @enumerate @item Install BDB 4.5 (keep 4.3 around for now) -@item Update my-config.sexp to point to the appropriate BDB 4.5 directories -@item Upgrade your database directory to 4.5 +@item Setup my-config.sexp to point to the appropriate BDB 4.5 directories +@item Upgrade your existing database directory to 4.5 @itemize @item Run db43_recover in your 0.6 database @item Optional: run db43_archive -d to remove all logs not part of a checkpoint This will make catastrophic recovery impossible, but reduces the amount of data you have to backup. @item Backup your db files and remaining logs - @item Run db45_checkpoint -1 in your 0.6 database directory + @item Run db45_checkpoint -1 in the database directory @end itemize -@item Migrate 0.6 data to a new 0.6.1 database +@item Upgrade 0.6 data to a fresh 0.9 database @itemize @item Open your old database: @code{(setf sc (open-store '(:BDB "/Users/me/db/ele060/")))} - @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele061/"))} + @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele090/"))} @end itemize @item Test your new application and report any bugs that arise to @email{elephant-devel@@common-lisp.net} @end enumerate @@ -315,14 +318,15 @@ @emph{(NOTE: close-store may fail when closing the old 0.6 database, this is OK.)} @emph{(NOTE: 64-bit lisps will not successfully upgrade 32-bit 0.6 databases. Use a 32-bit - version of your lisp to update to 0.6.1 and then open that database in your 64-bit + version of your lisp to update to 0.9 and then open that database in your 64-bit lisp. There should be no compatibility problems. Best to test your application on a 32-bit lisp if you can, just to be sure.)} @subsection Upgrade from Elephant 0.5 -Follow the upgrade procedures outlined in Elephant release 0.6.0 to migrate your database -from 0.5 to 0.6. Then follow the above procedures for upgrading from an 0.6 database. +Follow the upgrade procedures outlined in the Elephant 0.6.0 INSTALL +file to upgrade your database from 0.5 to 0.6.0. Then follow the above +procedures for upgrading to 0.9. @emph{(NOTE: It may not take much work to make 0.9 upgrade directly from 0.5 However there are so few (none?) 0.5 users that it wasn't deemed worth @@ -333,55 +337,50 @@ @section CL-SQL Although originally designed as an interface to the BerkeleyDB system, -the original Elephant system has been experimenetally extended to -support the use of relational database management systems as the -implementation of the persistent store. This relies on Kevin Rosenberg's -CL-SQL interface to a large number of relational systems. - -Although the BerkeleyDB system is an ideal object store for LISP objects, -one might prefer the licensing of a different system. For example, at -the time of this writing, it is my interpretation that one cannot use -the BerkeleyDB system behind a public website +the original Elephant system has been extended to support the use of +relational database management systems as the implementation of the +persistent store. This relies on Kevin Rosenberg's CL-SQL interface, +which provides access to a large number of relational systems. + +A major motivation of this extension is that one one might prefer the +licensing of a different system. For example, at the time of this +writing, it is our interpretation that one cannot use the BerkeleyDB +system behind a public website http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source. Neither the PostGres DBMS nor SQLite 3, nor Elephant itself, imposes -any such restriction. (Elephant is released under the LLGPL -(http://opensource.franz.com/preamble.html .) Older versions were -released under the GPL.) +any such restriction. Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application needs to use the truly relational aspects of those systems, preference for the tools associated with those systems, etc. -The SQL back-end extention of Elephant provides a function for migrating -data seamlessly between repositories. One can quite easily move -data from a BerkeleyDB repository to a PostGres repository, and vice versa. -This offers at least the possiblity than once can develop using one backend, -for example BerkeleyDB, and then later move to Postgres, or vice versa. -One could even operate simultaneously out of multiple repositories, if -there were a good reason to do so. - -At the time of this writing, the basic strategy for the SQL implementation -is quite simple. The same serializer used for the Sleepycat implementation -is employed, the byte-string is base64 encoded, and placed in a single -table which is managed by Elephant. - -All functionality except nested transaction support and cursor-puts -supported by the BerkeleyDB backend is supported by the CL-SQL -back-end. Concurrency and transaction atomicity have not been stress -tested well for the CL-SQL based system. - -Additionally, it is NOT the case that the Elephant system currently provides -transaction support across multiple repositories; it provides the transaction -support provided by the underlying repository to the user in a per-repository -basis. - -The PostGres backend is as currently employed is about 5 times slower than -the BerkeleyDB backend. This could probably change with continued development. - -CL-SQL supports a lot of DBMS systems, but only PostGres and SqlLite 3 have been tested. +Elephant provides functions for migrating data seamlessly between data +stores. One can quite easily move data from a BerkeleyDB repository +to a PostGres repository, and vice versa. This offers at least the +possibility than one can develop using one data store, for example +BerkeleyDB, and then later move to Postgres. One could even operate +simultaneously out of multiple repositories, if there were a good +reason to do so. + +The SQL implementation shares the serializer with the BDB data store, but +base64 encodes the resulting binary stream. This data is placed into +a single table in the SQL data store. + +All functionality except for nested transaction support and +cursor-puts supported by the BerkeleyDB data store is supported by the +CL-SQL data store. CL-SQL transaction integrity under concurrent +operation has not been extensively stress tested. + +Additionally, it is NOT the case that the Elephant system currently +provides transaction support across multiple repositories; it provides +transaction support on a per-repository basis. + +The PostGres backend is currently about 5 times slower than the +BerkeleyDB backend. As of the time of this writing, only PostGres and +SqlLite 3 have been tested as CL-SQL backends. @node CL-SQL Example @comment node-name, next, previous, up @@ -391,20 +390,20 @@ @enumerate @item Install postgres and make sure postmaster is running. -Postgres may be installed on your system; you may be able to use a package manager to -install it, or you can install it quite easily from the PostgresSQL site directly -(http://www.postgresql.org/). +Postgres may be installed on your system; you may be able to use a [104 lines skipped]

1 0

[elephant-cvs] CVS elephant/src/db-clsql
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv6510/src/db-clsql Modified Files: sql-collections.lisp Log Message: Oops, cleanup debug stmts --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:28 1.14 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:54 1.15 @@ -518,12 +518,11 @@ (cursor-prev-x cursor) ) (defmethod cursor-prev-x ((cursor sql-secondary-cursor) &key (returnpk nil)) -;; (declare (optimize (speed 3))) + (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (progn (let ((prior-pk (get-current-key cursor))) (decf (sql-crsr-ck cursor)) - (format t "prior-key: ~A current-key: ~A%" prior-pk (get-current-key cursor)) (if (eq prior-pk (get-current-key cursor)) (setf (dp-nmbr cursor) (max 0 (- (dp-nmbr cursor) 1))) (setf (dp-nmbr cursor)

1 0

[elephant-cvs] CVS elephant/src/db-clsql
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv6425/src/db-clsql Modified Files: sql-collections.lisp Log Message: Fixed fencepost error in cursor-prev-x in clsql data store --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/12 02:47:24 1.13 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:28 1.14 @@ -518,18 +518,19 @@ (cursor-prev-x cursor) ) (defmethod cursor-prev-x ((cursor sql-secondary-cursor) &key (returnpk nil)) - (declare (optimize (speed 3))) +;; (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (progn - (let ((cur-pk (get-current-key cursor))) + (let ((prior-pk (get-current-key cursor))) (decf (sql-crsr-ck cursor)) - (if (equal cur-pk (get-current-key cursor)) + (format t "prior-key: ~A current-key: ~A%" prior-pk (get-current-key cursor)) + (if (eq prior-pk (get-current-key cursor)) (setf (dp-nmbr cursor) (max 0 (- (dp-nmbr cursor) 1))) - (setf (dp-nmbr cursor) - (sql-get-from-clcn-cnt (cursor-oid cursor) - (get-current-key cursor) - (get-con (cursor-btree cursor)) - )))) + (setf (dp-nmbr cursor) + (1- (sql-get-from-clcn-cnt (cursor-oid cursor) + (get-current-key cursor) + (get-con (cursor-btree cursor)))) + ))) (has-key-value-scnd cursor :returnpk returnpk)) (cursor-last-x cursor :returnpk returnpk)))

1 0

[elephant-cvs] CVS elephant/tests
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv12328/tests Modified Files: testmigration.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/tests/testmigration.lisp 2007/03/24 10:51:45 1.20 +++ /project/elephant/cvsroot/elephant/tests/testmigration.lisp 2007/04/29 02:11:02 1.21 @@ -34,7 +34,7 @@ (progn (format t "~%Single store mode: ignoring") t) - (let* ((*store-controller*) + (let* (*store-controller* (sc1 (open-store *test-spec-primary* :recover t :deadlock-detect t)) (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect t))) (unwind-protect

1 0

[elephant-cvs] CVS elephant/src/utils
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/utils In directory clnet:/tmp/cvs-serv12328/src/utils Modified Files: os.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/12 02:47:51 1.3 +++ /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/29 02:10:56 1.4 @@ -53,7 +53,7 @@ (ccl:run-program program args :wait nil)) #+lispworks (apply #'sys::call-system - (format nil "~a~{ '~a'~}~@[ &~]" prog args) + (format nil "~a~{ '~a'~} &" program args) :current-directory directory :wait nil) )

1 0

[elephant-cvs] CVS elephant/src/memutil
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/memutil In directory clnet:/tmp/cvs-serv12328/src/memutil Modified Files: memutil.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/src/memutil/memutil.lisp 2007/03/21 14:29:31 1.27 +++ /project/elephant/cvsroot/elephant/src/memutil/memutil.lisp 2007/04/29 02:10:55 1.28 @@ -471,12 +471,12 @@ (simple-string (loop for i fixnum from 0 below length do - (setf (deref-array dest 'array-or-pointer-char (+ i dest-offset)) + (setf (deref-array dest '(:array :unsigned-char) (+ i dest-offset)) (char-code (schar src (+ i src-offset)))))) (string (loop for i fixnum from 0 below length do - (setf (deref-array dest 'array-or-pointer-char (+ i dest-offset)) + (setf (deref-array dest '(:array :unsigned-char) (+ i dest-offset)) (char-code (char src (+ i src-offset))))))))

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv12328/src/elephant Modified Files: classes.lisp classindex.lisp collections.lisp unicode2.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/27 13:32:16 1.33 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/29 02:10:54 1.34 @@ -273,18 +273,21 @@ "Ensures that object can be written as a reference into store sc" (eq (dbcn-spc-pst object) (controller-spec sc))) -(define-condition cross-reference-error () +(define-condition cross-reference-error (error) ((object :accessor cross-reference-error-object :initarg :object) (home-controller :accessor cross-reference-error-home-controller :initarg :home-ctrl) (foreign-controller :accessor cross-reference-error-foreign-controller :initarg :foreign-ctrl)) (:documentation "An error condition raised when an object is being written into a data store other - than its home store")) + than its home store") + (:report (lambda (condition stream) + (format stream "Attempted to write object ~A with home store ~A into store ~A" + (cross-reference-error-object condition) + (cross-reference-error-home-controller condition) + (cross-reference-error-foreign-controller condition))))) (defun signal-cross-reference-error (object sc) (cerror "Proceed to write incorrect reference" 'cross-reference-error - :format-control "Attempted to write object ~A with home store ~A into store ~A" - :format-arguments (list object (get-con object) sc) :object object :home-ctrl (get-con object) :foreign-ctrl sc)) --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/28 03:07:38 1.41 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/29 02:10:54 1.42 @@ -84,18 +84,17 @@ (let ((class (find-class class-name nil))) (when class (indexed class)))) -(define-condition persistent-class-not-indexed (error) - ((class-obj :initarg :class :initarg nil :reader unindexed-class-obj))) - +(define-condition persistent-class-not-indexed () + ((class-obj :initarg :class :initarg nil :reader unindexed-class-obj)) + (:report (lambda (condition stream) + (format stream "Class ~A is not enabled for indexing" + (class-name (unindexed-class-obj condition)))))) + (defun signal-class-not-indexed (class) (cerror "Ignore and continue?" 'persistent-class-not-indexed - :format-control "Class ~A is not enabled for indexing" - :format-arguments (list (class-name class)) :class class)) -;; (define-condition - (defmethod find-class-index ((class persistent-metaclass) &key (sc *store-controller*) (errorp t)) (ensure-finalized class) (if (not (indexed class)) --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/28 17:18:33 1.29 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/29 02:10:54 1.30 @@ -74,13 +74,6 @@ (:documentation "Delete all key-value pairs from the btree and render it an invalid object in the data store")) -(defmethod drop-btree ((bt btree)) - (ensure-transaction (:store-controller *store-controller*) - (with-btree-cursor (cur bt) - (loop for (exists? key) = (multiple-value-list (cursor-first cur)) - then (multiple-value-list (cursor-next cur)) - while exists? - do (remove-kv key bt))))) ;; ;; Btrees that support secondary indices @@ -380,7 +373,6 @@ different key.) Returns has-tuple / secondary key / value / primary key.")) - (defmacro with-btree-cursor ((var bt) &body body) "Macro which opens a named cursor on a BTree (primary or not), evaluates the forms, then closes the cursor." @@ -389,6 +381,14 @@ (progn ,@body) (cursor-close ,var)))) +(defmethod drop-btree ((bt btree)) + (ensure-transaction (:store-controller *store-controller*) + (with-btree-cursor (cur bt) + (loop for (exists? key) = (multiple-value-list (cursor-first cur)) + then (multiple-value-list (cursor-next cur)) + while exists? + do (remove-kv key bt))))) + ;; ======================================= ;; Generic Mapping Functions ;; ======================================= --- /project/elephant/cvsroot/elephant/src/elephant/unicode2.lisp 2007/04/12 02:47:33 1.8 +++ /project/elephant/cvsroot/elephant/src/elephant/unicode2.lisp 2007/04/29 02:10:55 1.9 @@ -83,13 +83,13 @@ (let ((code (char-code (schar string i)))) (declare (type fixnum code)) (when (> code #xFF) (fail)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ i size)) code)))) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ i size)) code)))) (string (loop for i fixnum from 0 below characters do (let ((code (char-code (char string i)))) (declare (type fixnum code)) (when (> code #xFF) (fail)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ i size)) code))))) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ i size)) code))))) (setf (buffer-stream-size bstream) needed) (succeed)))))) @@ -120,20 +120,20 @@ (loop for i fixnum from 0 below characters do (let ((code (char-code (schar string i)))) (when (> code #xFFFF) (fail)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 2) size)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size)) ;; (coerce (ldb (byte 8 8) code) '(signed 8))) (ldb (byte 8 8) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 2) size 1)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size 1)) ;; (coerce (ldb (byte 8 0) code) '(signed 8)))))) (ldb (byte 8 0) code))))) (string (loop for i fixnum from 0 below characters do (let ((code (char-code (schar string i)))) (when (> code #xFFFF) (fail)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 2) size)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size)) ;; (coerce (ldb (byte 8 8) code) '(signed 8))) (ldb (byte 8 8) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 2) size 1)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size 1)) ;; (coerce (ldb (byte 8 0) code) '(signed 8))))))) (ldb (byte 8 0) code)))))) (incf size (* characters 2)) @@ -158,25 +158,25 @@ (loop for i fixnum from 0 below characters do (let ((code (char-code (schar string i)))) (when (> code #x10FFFF) (error "Invalid unicode code type")) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 0)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 0)) (ldb (byte 8 24) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 1)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 1)) (ldb (byte 8 16) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 2)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 2)) (ldb (byte 8 8) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 3)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 3)) (ldb (byte 8 0) code))))) (string (loop for i fixnum from 0 below characters do (let ((code (char-code (schar string i)))) (when (> code #x10FFFF) (error "Invalid unicode code type")) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 0)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 0)) (ldb (byte 8 24) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 1)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 1)) (ldb (byte 8 16) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 2)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 2)) (ldb (byte 8 8) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 3)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 3)) (ldb (byte 8 0) code)))))) (incf size (* characters 4)) t))))

1 0

[elephant-cvs] CVS elephant
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv12328 Modified Files: elephant.asd Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/21 17:22:35 1.45 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/29 02:10:54 1.46 @@ -305,7 +305,7 @@ (:file "serializer2") ;; 0.6.1 db's (:file "unicode2") (:file "migrate") - (:file "query") +;; (:file "query") (:file "data-store-api")) :serial t :depends-on (memutil utils)))))

1 0

[elephant-cvs] CVS elephant/tests
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv32598/tests Modified Files: elephant-tests.lisp testconditions.lisp Log Message: test conditions fix, new run-regression test interface; remove doc strings to cleanup manual --- /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/04/28 02:31:31 1.32 +++ /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/04/28 17:18:34 1.33 @@ -86,6 +86,9 @@ (defvar *testpg-spec* '(:clsql (:postgresql "localhost.localdomain" "test" "postgres" ""))) +(defvar *testpg-spec2* + '(:clsql (:postgresql "localhost.localdomain" "test2" "postgres" ""))) + (defvar *testsqlite3-spec* `(:clsql (:sqlite3 ,(namestring @@ -116,14 +119,36 @@ ;; -;; GUIDE TO TESTING +;; COMPREHENSIVE TESTING +;; + +(defun run-regression-tests (type) + (let (sc1 sc2 oid) + (case type + (:BDB + (setf sc1 *testbdb-spec* + sc2 *testbdb-spec2* + oid *testbdb-spec-oid*)) + (:SQLITE + (setf sc1 *testsqlite3-spec* + sc2 *testsqlite3-spec2*)) + (:POSTGRES + (setf sc1 *testpg-spec* + sc2 *testpg-spec*)) + (t (error "Unrecognized data store type: ~A" type))) + (let ((*test-spec-primary* sc1) + (*test-spec-secondary* sc2)) + (declare (special *test-spec-secondary* *test-spec-primary*)) + (do-backend-tests sc1)))) + +;; +;; GUIDE TO SUBTESTS ;; -;; 1) Set *default-spec* to the above spec of your choice -;; 2) Call (do-backend-tests) to test the standard API -;; 3) To test migration: (do-migration-tests *default-spec* <second-spec>) inserting a second +;; 1) Call (do-backend-tests <specname>) to test the standard API +;; 3) To test migration: (do-migration-tests <first-spec> <second-spec>) inserting a second ;; spec, typically a bdb spec or create another instance of a sql db depending on ;; your configuration -;; 4) A backend is green if it passes do-backend-tests and can succesfully be +;; 4) A data store is green if it passes do-backend-tests and can succesfully be ;; used as spec1 or spec2 argument in the migration test ;; @@ -146,8 +171,7 @@ "For easy interactive running of single tests while debugging" (when spec (with-open-store (spec) - (let ((*auto-commit* nil)) - (do-test testname))))) + (do-test testname)))) (defun do-migration-tests (spec1 spec2 &optional oid-spec) "Interface to do explicit migration tests between backends" --- /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 03:07:39 1.4 +++ /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 17:18:34 1.5 @@ -19,9 +19,9 @@ (progn (format t "~%Second store spec missing: ignoring") (values t t t t)) - (let (*store-controller* - (sc1 *store-controller*) - (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect nil))) + (let ((sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect nil)) + (*store-controller* *store-controller*) + (sc1 *store-controller*)) (unwind-protect (let ((inst1 (make-instance 'pfoo :slot1 100 :sc sc1)) (inst2 (make-instance 'pfoo :slot1 200 :sc sc2))) @@ -36,10 +36,7 @@ t t t t) (deftest unindexed-class-condition - (let* ((sc (open-store *test-spec-primary* :recover t :deadlock-detect nil)) - (inst (make-instance 'pfoo :slot1 1 :sc sc))) - (unwind-protect - (signals-specific-condition (persistent-class-not-indexed) - (find-class-index 'pfoo :sc sc)) - (close-store sc))) + (let ((inst (make-instance 'pfoo :slot1 1))) + (signals-specific-condition (persistent-class-not-indexed) + (find-class-index 'pfoo))) t) \ No newline at end of file

1 0

[elephant-cvs] CVS elephant
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv32598 Modified Files: ele-bdb.asd Log Message: test conditions fix, new run-regression test interface; remove doc strings to cleanup manual --- /project/elephant/cvsroot/elephant/ele-bdb.asd 2007/04/19 05:24:37 1.26 +++ /project/elephant/cvsroot/elephant/ele-bdb.asd 2007/04/28 17:18:32 1.27 @@ -18,6 +18,11 @@ (in-package :cl-user) +;; Forward def +(defpackage elephant-system + (:use :cl :asdf) + (:export :elephant-c-source :compiler-options :foreign-libraries-to-load-first :get-config-option)) + (defpackage ele-bdb-system (:use :cl :asdf :elephant-system)) @@ -40,6 +45,9 @@ (setf result (concatenate 'string "/cygdrive" result)) (substitute #\/ #\\ result))) +;; Forward def +(defclass elephant-c-source (c-source-file) ()) + (defclass bdb-c-source (elephant-c-source) ()) (defmethod compiler-options ((compiler (eql :gcc)) (c bdb-c-source) &key &allow-other-keys)

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv32598/src/elephant Modified Files: collections.lisp Log Message: test conditions fix, new run-regression test interface; remove doc strings to cleanup manual --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/28 02:31:15 1.28 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/28 17:18:33 1.29 @@ -340,8 +340,8 @@ "Move to the previous duplicate element (with the same key.) Returns has-pair key value.")) +;; Default implementation. Plan is to update both backends when BDB 4.6 comes out (defmethod cursor-prev-dup ((cur cursor)) - "Default implementation. Plan is to update both backends when BDB 4.6 comes out" (when (cursor-initialized-p cur) (multiple-value-bind (exists? skey-cur) (cursor-current cur) @@ -362,8 +362,8 @@ "Move to the previous duplicate element (with the same key.) Returns has-tuple / secondary key / value / primary key.")) +;; Default implementation. Plan is to update both backends when BDB 4.6 comes out (defmethod cursor-pprev-dup ((cur cursor)) - "Default implementation. Plan is to update both backends when BDB 4.6 comes out" (when (cursor-initialized-p cur) (multiple-value-bind (exists? skey-cur) (cursor-current cur)

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv26611 Modified Files: Makefile data-store-reference.texinfo elephant.texinfo installation.texinfo intro.texinfo reference.texinfo user-guide.texinfo Log Message: Final doc cleanup; added contents to PDF (finally) --- /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/01 14:33:24 1.6 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/28 14:24:57 1.7 @@ -9,4 +9,5 @@ makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo pdf: includes-stuff - texi2dvi --pdf elephant.texinfo + texi2pdf --texinfo=@setcontentsaftertitlepage --texinfo=@setshortcontentsaftertitlepage \ + elephant.texinfo --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/28 02:31:07 1.8 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/28 14:24:57 1.9 @@ -28,12 +28,12 @@ * Registration: DSR Registration. Register the data store for the appropriate controller specifications. * Store Controllers: DSR Store Controllers. Subclassing the store controller. * Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects. -* Memory Utilities: DSR Memory Utilities. Writing primitive C types. * Persistent Objects and Slot access: DSR Persistent Objects and Slot Access. Support for metaprotocol slot access. * Collections: DSR Collections. BTrees and indices. * Cursors: DSR Cursors. Traversing BTrees. * Transactions: DSR Transactions. Transaction implementation. * Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations. +@c * Memory Utilities: DSR Memory Utilities. Writing primitive C types. @c * Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries @end menu @@ -241,7 +241,7 @@ @node DSR Handling Serialization @comment node-name, next, previous, up @section Handling Serialization -@cindex Serialization +@cindex Serializer Data stores must initialize @ref{Class elephant:store-controller} with internal serializer functions. Packages @code{elephant-serializer1} --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/28 02:31:07 1.11 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/28 14:24:57 1.12 @@ -2,9 +2,10 @@ @c %**start of header @setfilename elephant.info @settitle Elephant User Manual -@setchapternewpage odd @c %**end of header +@setchapternewpage odd + @copying @quotation Elephant System @* @@ -44,7 +45,6 @@ @top Copyright @insertcopying -@end ifnottex @ifhtml @menu @@ -74,13 +74,14 @@ * Variable Index:: * Colophon:: @end menu - @end ifhtml @node Table of Contents @unnumbered @comment node-name, next, previous, up -@contents +@contents + +@end ifnottex @include intro.texinfo @include tutorial.texinfo @@ -131,4 +132,8 @@ @uref{http://www.common-lisp.net/project/elephant/}) and submit a patch. +@iftex +@shortcontents +@contents +@end iftex @bye --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 02:31:07 1.12 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 14:24:57 1.13 @@ -11,7 +11,7 @@ * Loading Elephant:: Loading Elephant and the data store loading protocol. * Berkeley DB:: Installing support for the Berkeley DB data store. * Berkeley DB Example:: An example of installing and running the Berkeley DB data store. -* Upgrading Berkeley DB Database:: How to upgrade to a new version of Berkeley DB. +* Upgrading Berkeley DB Databases:: How to upgrade to a new version of Berkeley DB. * CL-SQL:: Install and connecting to the CL-SQL data store. * CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows @@ -309,8 +309,8 @@ @item Open your old database: @code{(setf sc (open-store '(:BDB "/Users/me/db/ele060/")))} @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele061/"))} @end itemize -@item Test your new application and report any bugs that arise to elephant-devel(a)common-lisp.net -@end itemize +@item Test your new application and report any bugs that arise to @email{elephant-devel@@common-lisp.net} +@end enumerate @emph{(NOTE: close-store may fail when closing the old 0.6 database, this is OK.)} --- /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/04/02 00:51:06 1.6 +++ /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/04/28 14:24:57 1.7 @@ -3,7 +3,9 @@ @node Introduction @comment node-name, next, previous, up @chapter Introduction -@cindex Introduction +@cindex History +@cindex Goals + Elephant is a persistent object protocol and database for Common Lisp. The persistent protocol component of elephant overrides class --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/28 02:31:07 1.14 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/28 14:24:57 1.15 @@ -7,22 +7,22 @@ @cindex API Reference @menu -* Store Controllers:: Connecting to a data store. -* Persistent Object API:: Defining persistent classes and creating and manipulating persistent indices. -* Persistent Object Indexing API:: Convenient indexing of persistent classes. -* Persistent Set API:: Maintaining persistent collections the easy way. +* Store Controllers: Store Controller API. Connecting to a data store. +* Persistent Objects: Persistent Object API. Defining persistent classes and creating and manipulating persistent indices. +* Persistent Object Indexing: Persistent Object Indexing API. Convenient indexing of persistent classes. +* Persistent Sets: Persistent Set API. Maintaining persistent collections the easy way. @c * Query Interfaces:: Finding instances. -* BTree API:: BTrees and indices, a low level persistent data structure. -* Cursor API:: Traversing BTrees. -* Index Cursor API:: Traversing BTree Indicies. -* Transaction API:: Transaction functions. -* Migration and Upgrading API:: Migration and upgrading. -* Miscellaneous API:: Other functions and data store specific functions +* BTrees: BTree API. BTrees and indices, a low level persistent data structure. +* BTree Cursors: BTree Cursor API. Traversing BTrees. +* Index Cursors: Index Cursor API. Traversing BTree Indicies. +* Transactions: Transaction API. Transaction functions. +* Migration and Upgrading: Migration and Upgrading API. Migration and upgrading. +@c * Miscellaneous API:: Other functions and data store specific functions @end menu @node Store Controller API @comment node-name, next, previous, up -@section Store Controller API +@section Store Controllers @cindex Store Controller @cindex API @@ -49,7 +49,7 @@ @node Persistent Object API @comment node-name, next, previous, up -@section Persistent Object API +@section Persistent Objects @cindex Persistent Objects @cindex API @@ -63,7 +63,7 @@ @node Persistent Object Indexing API @comment node-name, next, previous, up -@section Persistent Object Indexing API +@section Persistent Object Indexing @cindex Persistent Object Indexing @cindex API @@ -100,8 +100,8 @@ @node Persistent Set API @comment node-name, next, previous, up -@section Persistent Set API -@cindex Persistent Set API +@section Persistent Sets +@cindex Persistent Sets Persistent sets are a simple persistent collection abstraction. They maintain an unordered collection of objects. Unlike the normal @@ -132,7 +132,7 @@ @node BTree API @comment node-name, next, previous, up -@section BTree API +@section BTrees @cindex BTree @cindex API @@ -167,9 +167,9 @@ @include includes/fun-elephant-map-index.texinfo -@node Cursor API +@node BTree Cursor API @comment node-name, next, previous, up -@section Cursor API +@section Btree Cursors @cindex Cursors @cindex API @@ -200,7 +200,7 @@ @node Index Cursor API @comment node-name, next, previous, up -@section Index Cursor API +@section Index Cursors @cindex Cursor @cindex Index @cindex Indices @@ -239,7 +239,7 @@ @node Transaction API @comment node-name, next, previous, up -@section Transaction API +@section Transactions @cindex Transaction API @cindex API @@ -256,7 +256,7 @@ @node Migration and Upgrading API @comment node-name, next, previous, up -@section Migration and Upgrading API +@section Migration and Upgrading @cindex Migration and Upgrading @cindex API --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 02:31:11 1.20 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 14:24:57 1.21 @@ -882,7 +882,7 @@ setting a second cursor to the same location. Cursors can be in two states: initialized and uninitialized. -@ref{Cursor API} for details. +@ref{BTree Cursor API} for details. To initialize a cursor, you have to use one of the initializing functions to select a key-value pair in the btree. @@ -1068,6 +1068,7 @@ that the cursor is now uninitialized (@code{cursor-initialized-p}) and needs to be reset by a set or set both call. +@xref{Index Cursor API} for further details. @node Multi-threaded Applications @comment node-name, next, previous, up

1 0

[elephant-cvs] CVS elephant/doc/oldfiles
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/doc/oldfiles In directory clnet:/tmp/cvs-serv26611/oldfiles Modified Files: TODO Log Message: Final doc cleanup; added contents to PDF (finally) --- /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 02:31:14 1.1 +++ /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 14:24:57 1.2 @@ -12,24 +12,9 @@ TASKS TO GET TO FINAL RELEASE: Bugs: -- Fix awkward serializer API -- Support for asdf-install? - - problem with other .asd files in root - -Lisp Support: -- Validate OpenMCL pre-1.1 on Mac OS X -- Validate OpenMCL 1.1 and/or 64-bit on Mac OS X? -- Verify db_deadlock for other lisps (launch and kill background program I/F) - -Documentation: -- Update install, build and test procedures -- Upgrade, migration and other system level issues -- Performance and design issues - - More notes about transaction performance - - Serious discussion of threading implications - - Add notes about with/ensure-transaction usage (abort & commit behavior on exit) - - Add notes about deadlock-detect - - Add notes about checkpoint (null in SQL?) +- Validate OpenMCL 1.0 on Mac OS X +- Lispworks on Linux +- Verify install, build and test documentation and procedures 0.6.1 - Features COMPLETED to date ----------------------------------

1 0

[elephant-cvs] CVS elephant/tests
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv23936/tests Modified Files: testconditions.lisp Log Message: Fixed test bug; cleaned up get-instances-by-xxx fns to use new map operators --- /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 02:31:31 1.3 +++ /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 03:07:39 1.4 @@ -20,7 +20,7 @@ (format t "~%Second store spec missing: ignoring") (values t t t t)) (let (*store-controller* - (sc1 (open-store *test-spec-primary* :recover t :deadlock-detect nil)) + (sc1 *store-controller*) (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect nil))) (unwind-protect (let ((inst1 (make-instance 'pfoo :slot1 100 :sc sc1)) @@ -32,13 +32,14 @@ (add-to-root 'inst1 inst1 :sc sc2)) (signals-specific-condition (cross-reference-error) (add-to-root 'inst2 inst2 :sc sc1)))) - (close-store sc1) (close-store sc2)))) t t t t) (deftest unindexed-class-condition (let* ((sc (open-store *test-spec-primary* :recover t :deadlock-detect nil)) (inst (make-instance 'pfoo :slot1 1 :sc sc))) - (signals-specific-condition (persistent-class-not-indexed) - (find-class-index 'pfoo :sc sc))) + (unwind-protect + (signals-specific-condition (persistent-class-not-indexed) + (find-class-index 'pfoo :sc sc)) + (close-store sc))) t) \ No newline at end of file

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv23936/src/elephant Modified Files: classindex.lisp controller.lisp Log Message: Fixed test bug; cleaned up get-instances-by-xxx fns to use new map operators --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/28 02:31:15 1.40 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/28 03:07:38 1.41 @@ -389,7 +389,7 @@ ;; USER MAPPING API ;; ====================== -(defun map-class (fn class) +(defun map-class (fn class &key collect) "Perform a map operation over all instances of class. Takes a function of one argument, a class instance" (let* ((class (if (symbolp class) @@ -400,9 +400,9 @@ (declare (ignore k)) (funcall fn v))) (declare (dynamic-extent map-fn)) - (map-btree #'map-fn class-idx)))) + (map-btree #'map-fn class-idx :collect collect)))) -(defun map-inverted-index (fn class index &rest args &key start end value from-end) +(defun map-inverted-index (fn class index &rest args &key start end (value nil value-p) from-end collect) "map-inverted-index maps a function of two variables, taking key and instance, over a subset of class instances in the order defined by the index. Specify the class and index by quoted @@ -431,7 +431,10 @@ (declare (ignore pkey)) (funcall fn key value))) (declare (dynamic-extent wrapper)) - (map-index #'wrapper index :start start :end end :value value :from-end from-end)))) + (if value-p + (map-index #'wrapper index :value value :collect collect) + (map-index #'wrapper index :start start :end end :from-end from-end :collect collect))))) + ;; ================= @@ -454,30 +457,27 @@ nil to start or end indicates, respectively, the lowest or highest value in the index")) + +(defun identity2 (k v) + (declare (ignore k)) + v) + +(defun identity3 (k v pk) + (declare (ignore k pk)) + v) + (defmethod get-instances-by-class ((class symbol)) (get-instances-by-class (find-class class))) (defmethod get-instances-by-class ((class persistent-metaclass)) - (let ((instances nil)) - (flet ((accum (c) - (declare (dynamic-extent c)) - (push c instances))) - (map-class #'accum class) - (nreverse instances)))) + (map-class #'identity class :collect t)) (defmethod get-instances-by-value ((class symbol) slot-name value) (get-instances-by-value (find-class class) slot-name value)) (defmethod get-instances-by-value ((class persistent-metaclass) slot-name value) (declare (type (or string symbol) slot-name)) - (let ((instances nil)) - (declare (type list instances)) - (flet ((collector (k v pk) - (declare (ignore k pk)) - (push v instances))) - (declare (dynamic-extent collector)) - (map-index #'collector (find-inverted-index class slot-name) :value value)) - (nreverse instances))) + (map-inverted-index #'identity2 class slot-name :value value :collect t)) (defmethod get-instance-by-value ((class symbol) slot-name value) (let ((list (get-instances-by-value (find-class class) slot-name value))) @@ -495,15 +495,7 @@ (defmethod get-instances-by-range ((class persistent-metaclass) idx-name start end) (declare (type (or fixnum null) start end) (type symbol idx-name)) - (let ((instances nil)) - (declare (type list instances)) - (flet ((collector (k v pk) - (declare (ignore k pk)) - (push v instances))) - (declare (dynamic-extent collector)) - (map-index #'collector (find-inverted-index class idx-name) - :start start :end end)) - (nreverse instances))) + (map-inverted-index #'identity2 class idx-name :start start :end end :collect t)) (defun drop-instances (instances &key (sc *store-controller*)) "Removes a list of persistent objects from all class indices --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/28 02:31:15 1.51 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/28 03:07:38 1.52 @@ -220,7 +220,7 @@ "Reset the instance cache (flush object lookups). Useful for testing. Does not reclaim existing objects so there will be duplicate instances with identical functionality" - (ele-with-lock ((instance-cache-lock sc)) + (ele-with-fast-lock ((instance-cache-lock sc)) (setf (instance-cache sc) (make-cache-table :test 'eql))))

1 0

[elephant-cvs] CVS elephant/tests
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv16753/tests Modified Files: elephant-tests.lisp testconditions.lisp Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/03/30 14:34:35 1.31 +++ /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/04/28 02:31:31 1.32 @@ -300,9 +300,15 @@ (defmacro signals-condition (&body body) `(handler-case - (progn ,@body) - (condition () t) - (:no-error (&rest rest) (declare (ignore rest)) nil))) + (progn ,@body) + (condition () t) + (:no-error (&rest rest) (declare (ignore rest)) nil))) + +(defmacro signals-specific-condition ((condition) &body body) + `(handler-case + (progn ,@body) + (,condition () t) + (:no-error (&rest rest) (declare (ignore rest)) nil))) (defmacro signals-error (&body body) `(handler-case --- /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/22 03:02:35 1.2 +++ /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 02:31:31 1.3 @@ -28,8 +28,17 @@ (values (is-not-null (add-to-root 'inst1 inst1 :sc sc1)) (is-not-null (add-to-root 'inst2 inst2 :sc sc2)) - (signals-condition (add-to-root 'inst1 inst1 :sc2)) - (signals-condition (add-to-root 'inst2 inst2 :sc1)))) + (signals-specific-condition (cross-reference-error) + (add-to-root 'inst1 inst1 :sc sc2)) + (signals-specific-condition (cross-reference-error) + (add-to-root 'inst2 inst2 :sc sc1)))) (close-store sc1) (close-store sc2)))) - t t t t) \ No newline at end of file + t t t t) + +(deftest unindexed-class-condition + (let* ((sc (open-store *test-spec-primary* :recover t :deadlock-detect nil)) + (inst (make-instance 'pfoo :slot1 1 :sc sc))) + (signals-specific-condition (persistent-class-not-indexed) + (find-class-index 'pfoo :sc sc))) + t) \ No newline at end of file

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv16753/src/elephant Modified Files: classindex.lisp collections.lisp controller.lisp package.lisp transactions.lisp Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/24 12:58:10 1.39 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/28 02:31:15 1.40 @@ -84,11 +84,23 @@ (let ((class (find-class class-name nil))) (when class (indexed class)))) +(define-condition persistent-class-not-indexed (error) + ((class-obj :initarg :class :initarg nil :reader unindexed-class-obj))) + +(defun signal-class-not-indexed (class) + (cerror "Ignore and continue?" + 'persistent-class-not-indexed + :format-control "Class ~A is not enabled for indexing" + :format-arguments (list (class-name class)) + :class class)) + +;; (define-condition + (defmethod find-class-index ((class persistent-metaclass) &key (sc *store-controller*) (errorp t)) (ensure-finalized class) (if (not (indexed class)) (when errorp - (error "Class ~A is not an indexed class" class)) + (signal-class-not-indexed class)) (if (class-index-cached? class) (%index-cache class) ;; we've got a cached reference, just return it (multiple-value-bind (btree found) @@ -110,31 +122,26 @@ (synchronize-class-to-store class :sc sc :method method) btree)) -(define-condition persistent-class-not-indexed (error) - ((class-obj :initarg :class :initarg nil :reader unindexed-class-obj))) - (defun cache-new-class-index (class sc) "If not cached or persistent then this is a new class, make the new index" (if (indexed class) (enable-class-indexing class (indexing-record-slots (indexed-record class)) :sc sc) - (signal 'persistent-class-not-indexed - :class class - :format-control "Class ~A is not enabled for indexing" - :format-arguments (list (class-name class))))) + (signal-class-not-indexed class))) (defmethod find-inverted-index ((class symbol) slot &key (null-on-fail nil)) (find-inverted-index (find-class class) slot :null-on-fail null-on-fail)) (defmethod find-inverted-index ((class persistent-metaclass) slot &key (null-on-fail nil)) - (let* ((cidx (find-class-index class)) + (let* ((cidx (find-class-index class :errorp (not null-on-fail))) (idx (or (get-index cidx slot) (get-index cidx (make-derived-name slot))))) (if idx idx (if null-on-fail nil - (error "Inverted index ~A not found for class ~A with - persistent slots: ~A" slot (class-name class) (car (%persistent-slots class))))))) + (cerror "Ignore and continue?" + "Inverted index ~A not found for class ~A with persistent slots: ~A" + slot (class-name class) (car (%persistent-slots class))))))) (defmethod find-inverted-index-names ((class persistent-metaclass)) (let ((names nil)) --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/27 13:32:17 1.27 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/28 02:31:15 1.28 @@ -312,6 +312,7 @@ primary key greater or equal to the pkey argument. Returns has-tuple / secondary key / value / primary key.")) + (defgeneric cursor-next-dup (cursor) (:documentation "Move to the next duplicate element (with the same key.) @@ -322,11 +323,6 @@ "Move to the next non-duplicate element (with different key.) Returns has-pair key value.")) -(defgeneric cursor-prev-nodup (cursor) - (:documentation - "Move to the previous non-duplicate element (with -different key.) Returns has-pair key value.")) - (defgeneric cursor-pnext-dup (cursor) (:documentation "Move to the next duplicate element (with the same key.) @@ -338,12 +334,53 @@ key.) Returns has-tuple / secondary key / value / primary key.")) + +(defgeneric cursor-prev-dup (cursor) + (:documentation + "Move to the previous duplicate element (with the same key.) +Returns has-pair key value.")) + +(defmethod cursor-prev-dup ((cur cursor)) + "Default implementation. Plan is to update both backends when BDB 4.6 comes out" + (when (cursor-initialized-p cur) + (multiple-value-bind (exists? skey-cur) + (cursor-current cur) + (declare (ignore exists?)) + (multiple-value-bind (exists? skey value) + (cursor-prev cur) + (if (lisp-compare-equal skey-cur skey) + (values exists? skey value) + (setf (cursor-initialized-p cur) nil)))))) + +(defgeneric cursor-prev-nodup (cursor) + (:documentation + "Move to the previous non-duplicate element (with +different key.) Returns has-pair key value.")) + +(defgeneric cursor-pprev-dup (cursor) + (:documentation + "Move to the previous duplicate element (with the same key.) +Returns has-tuple / secondary key / value / primary key.")) + +(defmethod cursor-pprev-dup ((cur cursor)) + "Default implementation. Plan is to update both backends when BDB 4.6 comes out" + (when (cursor-initialized-p cur) + (multiple-value-bind (exists? skey-cur) + (cursor-current cur) + (declare (ignore exists?)) + (multiple-value-bind (exists? skey value pkey) + (cursor-pprev cur) + (if (lisp-compare-equal skey-cur skey) + (values exists? skey value pkey) + (setf (cursor-initialized-p cur) nil)))))) + (defgeneric cursor-pprev-nodup (cursor) (:documentation "Move to the previous non-duplicate element (with different key.) Returns has-tuple / secondary key / value / primary key.")) + (defmacro with-btree-cursor ((var bt) &body body) "Macro which opens a named cursor on a BTree (primary or not), evaluates the forms, then closes the cursor." @@ -439,6 +476,7 @@ start end)) (let ((sc (get-con index)) (end (or value end)) + (from-end (and from-end (not value-set-p))) (results nil)) (flet ((collector (k v pk) (push (funcall fn k v pk) results))) @@ -454,11 +492,12 @@ (lisp-compare-equal key start)) (lisp-compare<= key end)))) (value-increment () - ;; Step to the next key value + ;; Step to the next key value; + ;; from-end duplicate cursor is already there (if from-end - (pprev-hack cur) + (cursor-current cur) (cursor-pnext-nodup cur))) - (next-value () + (map-values () ;; Handle the next key value (multiple-value-bind (exists? skey val pkey) (value-increment) @@ -468,18 +507,23 @@ (map-duplicates skey)) (return-from map-index (nreverse results))))) + (next-duplicate (key) + (if from-end + (pprev-dup-hack cur key) + (cursor-pnext-dup cur))) (map-duplicates (key) ;; Map all duplicates for key value (multiple-value-bind (exists? skey val pkey) - (cursor-pnext-dup cur) + (next-duplicate key) (if exists? (progn (funcall fn skey val pkey) (map-duplicates key)) (progn - (cursor-pset-range cur key) - (next-value)))))) - (declare (dynamic-extent (function next-value) (function next-value-increment) + (unless from-end + (cursor-pset cur key)) + (map-values)))))) + (declare (dynamic-extent (function map-values) (function next-duplicate) (function continue-p) (function map-duplicates))) (multiple-value-bind (exists? skey val pkey) (cond (value-set-p @@ -487,9 +531,9 @@ ((and (not from-end) (null start)) (cursor-pfirst cur)) ((and from-end (null end)) - (cursor-last-range-hack cur)) + (cursor-plast cur)) (t (if from-end - (cursor-pset-range cur end) + (pset-range-for-descending cur end) (cursor-pset-range cur start)))) (if (and exists? (continue-p skey)) (progn @@ -497,23 +541,24 @@ (map-duplicates skey)) nil))))))))) -(defun pprev-hack (cur) - "Get the first duplicate instance of the prior value off the current cursor" - (let ((e? (cursor-pprev-nodup cur))) - (when e? - (let ((e? (cursor-pprev-nodup cur))) - (if e? - (cursor-pnext cur) - (cursor-pfirst cur)))))) - -(defun cursor-last-range-hack (cur) - "Get the first duplicate instance of the last value of the cursor's index" - (let ((e? (cursor-plast cur))) - (when e? - (let ((e? (cursor-pprev-nodup cur))) - (if e? - (cursor-pnext cur) - (cursor-pfirst cur)))))) +(defun pset-range-for-descending (cur end) + (if (cursor-pset cur end) + (progn + (cursor-next-nodup cur) + (cursor-pprev cur)) + (progn + (cursor-pset-range cur end) + (cursor-pprev cur)))) + +(defun pprev-dup-hack (cur key) + "Go back one step in a duplicate set, returns nil + if previous element is a different key. More efficient than + the current default implementation of cursor-pprev-dup" + (multiple-value-bind (exists? skey value pkey) + (cursor-pprev cur) + (when (lisp-compare-equal key skey) + (values exists? key value pkey)))) + ;; =============================== ;; Some generic utility functions --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/25 02:28:01 1.50 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/28 02:31:15 1.51 @@ -136,7 +136,7 @@ (defun initialize-user-parameters () (loop for (keyword variable) in *user-configurable-parameters* do (awhen (get-user-configuration-parameter keyword) - (setf variable it)))) + (setq variable it)))) ;; ;; COMMON STORE CONTROLLER FUNCTIONALITY @@ -165,7 +165,7 @@ "This is an instance cache and part of the metaclass protocol. Data stores should not override the default behavior.") - (instance-cache-lock :accessor instance-cache-lock :initform (ele-make-lock) + (instance-cache-lock :accessor instance-cache-lock :initform (ele-make-fast-lock) :documentation "Protection for updates to the cache from multiple threads. Do not override.") @@ -202,14 +202,16 @@ (defun cache-instance (sc obj) "Cache a persistent object with the controller." (declare (type store-controller sc)) - (ele-with-lock ((instance-cache-lock sc)) + (ele-with-fast-lock ((instance-cache-lock sc)) (setf (get-cache (oid obj) (instance-cache sc)) obj))) (defun get-cached-instance (sc oid class-name) "Get a cached instance, or instantiate!" (declare (type store-controller sc) (type fixnum oid)) - (let ((obj (get-cache oid (instance-cache sc)))) + (let ((obj + (ele-with-fast-lock ((instance-cache-lock sc)) + (get-cache oid (instance-cache sc))))) (if obj obj ;; Should get cached since make-instance calls cache-instance (make-instance class-name :from-oid oid :sc sc)))) --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/27 13:32:17 1.34 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/28 02:31:16 1.35 @@ -222,12 +222,12 @@ #:cursor #:secondary-cursor #:make-cursor #:make-simple-cursor #:cursor-close #:cursor-duplicate #:cursor-current #:cursor-first #:cursor-last #:cursor-next #:cursor-next-dup - #:cursor-next-nodup #:cursor-prev #:cursor-prev-nodup + #:cursor-next-nodup #:cursor-prev #:cursor-prev-nodup #:cursor-prev-dup #:cursor-set #:cursor-set-range #:cursor-get-both #:cursor-get-both-range #:cursor-delete #:cursor-put #:cursor-pcurrent #:cursor-pfirst #:cursor-plast #:cursor-pnext #:cursor-pnext-dup #:cursor-pnext-nodup - #:cursor-pprev #:cursor-pprev-nodup #:cursor-pset + #:cursor-pprev #:cursor-pprev-dup #:cursor-pprev-nodup #:cursor-pset #:cursor-pset-range #:cursor-pget-both #:cursor-pget-both-range #:cursor-initialized-p @@ -267,6 +267,7 @@ ;; Various error conditions #:cross-reference-error #:controller-lost-error + #:persistent-class-not-indexed #:map-class-query #:get-query-instances --- /project/elephant/cvsroot/elephant/src/elephant/transactions.lisp 2007/04/27 13:32:17 1.12 +++ /project/elephant/cvsroot/elephant/src/elephant/transactions.lisp 2007/04/28 02:31:16 1.13 @@ -120,7 +120,7 @@ If nested, the backend must support nested transactions." (let ((sc (gensym))) `(let ((,sc ,store-controller)) - (funcall #'execute-transaction ,store-controller + (funcall #'execute-transaction ,sc (lambda () ,@body) :parent (if (owned-txn-p ,sc ,parent) (transaction-object ,parent) @@ -146,10 +146,9 @@ (,sc ,store-controller)) (if (owned-txn-p ,sc ,parent) (funcall ,txn-fn) - (funcall #'execute-transaction ,store-controller + (funcall #'execute-transaction ,sc ,txn-fn :parent nil - :transaction nil :retries ,retries ,@(remove-keywords '(:store-controller :parent :transaction :retries) keyargs))))))

1 0

[elephant-cvs] CVS elephant/doc/oldfiles
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/doc/oldfiles In directory clnet:/tmp/cvs-serv16753/doc/oldfiles Added Files: INSTALL NEWS NOTES TODO TUTORIAL Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/doc/oldfiles/INSTALL 2007/04/28 02:31:14 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/INSTALL 2007/04/28 02:31:14 1.1 ------------ Requirements ------------ Supported Lisps: CMUCL 19a Linux SBCL 0.9.17/1.0+ Linux / Mac OSX Allegro CL 7.0/8.0 Linux / Mac OSX OpenMCL 0.14.2 LispWorks (port in-progress) Lisp libraries: ASDF - http://www.cliki.net/asdf UFFI 1.5.4+ - http://uffi.b9.com/ A Backend Database: 1) Oracle Berkeley DB 4.5 - http://www.oracle.com/database/berkeley-db.html 2) CLSQL - http://clsql.b9.com/ with an appropriate SQL installation. Tested with SQlite3 and Postgresql so far A C compiler, probably gcc or Visual Studio. Presumably you have this if you installed ------------------ Short Instructions ------------------ The new build system should work out of the box on most Un*x platforms that have asdf, uffi and either clsql or Berkeley DB installed in the usual places. Try: (asdf:operate 'asdf:load-op :elephant) Then: (open-store '(<backend> <spec>)) Where <backend> = { :bdb | :clsql } <spec> = { "fresh directory for BDB files" | '(:sqlite3 "db path") | '(:postgresql "db path") This should load all files, including compiling libraries, on most systems. For Win32, see the instructions below. (We'll improve the build process for Win32 if there is demand) ----------------- Long Instructions ----------------- For SBCL, CMUCL, Allegro 8.0+, MCL and CLISP: 0) Unpack Elephant. I put mine in the directory /usr/local/share/common-lisp/elephant-0.6.x/ 1) Install ASDF. Ensure that you have a recent version of ASDF installed as the load process now depends upon it. 2) Install UFFI 3) Install a backend: Either Berkeley DB 4.5, PostGresql, or SQLite 3. ------- SQL ------- For relational database systems, refering the formal documentation other the heading "SQL-BACK-END". ------------- Berkeley 4.5: ------------- (Note: 0.6.0 required BDB 4.3; to upgrade 0.6.0 to 0.6.1, upgrade BDB to 4.5, modify my-config.sexp appropriately then run 0.6.1+; your underlying Berekely DB files will automatically upgrade when the DB is opened. To use 0.6.1, you will have to manually migrate your 0.6.0 database to a fresh database created in 0.6.1) Under Un*x, you may actually already have this installed, though it may be compiled with funny options, so if things don't work you may want to try to start from scratch. FreeBSD has a port for this, as I'm sure do other BSDs (including DarwinPorts/Fink.) Take note of where libdb.so and db.h are installed, usually: /usr/local/BerkeleyDB.4.5/lib/libdb.so and /usr/local/BerkeleyDB.4.5/include/db.h, or /usr/local/lib/db45/libdb.so and /usr/local/include/db45/db.h.) a) Site specific configuration config.sexp Which contains an alist providing string paths pointing to the root of the Berkeley DB distribution :berkeley-db-root, the library to load :berkeley-db-lib and the pthreads library if you're running linux :pthread-lib. For Win32 (directions courtesy of Bill Clementson): --------------------------------------------------- Create an MSVC dll project and add src/db-bdb/libberkeley-db.c, src/db-bdb/libberkeley-db.def and the Berkeley DB libdb43.lib files to the project (should be in the build_win32/release folder) Add the Berkeley DB dbinc include files directory and the build_win32/release directory (where the Berkeley DB install instructions builds the Berkeley DB objects by default) to the build directories for the project Build the Elephant DLL file Since you've statically included libdb43.lib inside libberkeley-db.c, it may or may not be necessary to load libdb43.dll into Lisp (see below.) 4) Compile and load Elephant: The new backend load process should work automatically on Un*x systems but if there are probolems with loading foreign libraries, then you can test your C tools setup with 'make' in the elephant root directory. This will build the common memutils library in src/memutil/libmemutil.so/dylib that all backends require. There is a new two-phase load process. The first requires that you use asdf to load the main elephant front-end: (asdf:operate 'asdf:load-op :elephant) This will load and compile Elephant. This will also automatically load UFFI. When you call (open-store <spec>) inside lisp it will automatically load the remaining dependencies for the specified backend via ASDF. To test the load process explicitly the following asdf files are provided: if you are using Berkeley DB, type: (asdf:operate 'asdf:load-op :ele-bdb) if you are using CL-SQL, type: (asdf:operate 'asdf:load-op :ele-clsql) if you are using SQLite3, type: (asdf:operate 'asdf:load-op :ele-sqlite3) 5) Make the documentation: Execute: make In the doc directory should be build the HTML version of the texinfo files. ------- Testing ------- Elephant uses RT for regression testing, available at: http://www.cliki.net/RT Once RT is installed (asdf:operate 'asdf:load-op :elephant-tests) (in-package :ele-tests) (setf *default-spec* <backend>) Where <backend> = { *testsqlite3-spec* | *testpg-spec* | *testbdb-spec* } (do-backend-tests) This will test the standalone API for your backend. Currently all tests are passing on 0.6.0. There will be a set of migration tests that will be 'ignored' but the final message should indicate no failing tests. This should take less than 5 minutes on decent hardware. The tests are not idempotent, so if you run the tests a second time, they are likely to fail. To avoid this, for example if you are debugging tests, just run the script delscript.sh (or do the equivalent on Win32) in the elephant/tests directory. Elephant allows migration between repositories. To test this: (do-migration-tests *default-spec* <backend>) where <backend> is a different *testXXXXX-spec* variable to test migration to that backend. This should take less than 2 minutes on decent hardware. A backend is considered "green" if it can pass both the backend tests and the migration tests. --- /project/elephant/cvsroot/elephant/doc/oldfiles/NEWS 2007/04/28 02:31:14 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/NEWS 2007/04/28 02:31:14 1.1 April, 2006 - Elephant 0.6.0 released by Robert Read and Ian Eslick. Supports class slot indexing and benefits from a clean refactoring of backends and a host of other small changes. This is a solid BETA release. November 30, 2005 - Elephant 0.3.0 released by the new maintainer, Robert L. Read, providing support for relational database backends, repository migration, and multi-repository operation. As of this release, the documentation provides a lot of information about installation and getting things working; I wouldn't at all claim that it is complete, smooth, or well organized. The more notes I get about the use of Elephant, the more inclined I will be to invest time in improving the documentation. October 7, 2004 - Elephant 0.2.1 released. Thanks to Bill Clementson, Elephant should compile on Win32 now. Also, a few minor fixups. September 19, 2004 - Elephant 0.2 released. This is an BETA release. New features: - Secondary indices and cursors - PPC Darwin OpenMCL / SBCL - Doc strings and improved documentation - An RT-based test suite - many bugfixes This release has been tested on CMUCL 19a, SBCL 0.8.14 and Allegro 6.2 on x86 Linux and FreeBSD, and OpenMCL 0.14.2-p1 and SBCL 0.8.14 on PPC Darwin. September 2, 2004 - The bad news: there was a bug in 0.1 which made OID generation inside of manual transactions deadlock. The good news: this is fixed, and I've added OpenMCL support. So I'm releasing 0.1-p1. August 30, 2004 - Elephant 0.1 was released August 30th, 2004. This is an ALPHA quality release, so claims about correctness, performance and safety should be taken with a grain of salt. This release has been tested on CMUCL 19a, SBCL 0.8.13 and Allegro 6.2 on x86 Linux and FreeBSD. OpenMCL and Lispworks versions will come soon. As a proof of concept I've compiled and run CL-IRC http:://www.common-lisp.net/project/cl-irc making all objects and slots persistent, except for the socket-streams. It runs, and saves everything except for the socket-streams. --- /project/elephant/cvsroot/elephant/doc/oldfiles/NOTES 2007/04/28 02:31:15 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/NOTES 2007/04/28 02:31:15 1.1 ------- GENERAL ------- this has been optimized for use with CMUCL / SBCL / Allegro. OpenMCL has been minimally supported. Lispworks is a target as well but less so as the developers don't have access to it. Theoretically one can port this to any lisp with a decent FFI and MOP. However since those are two of the less standardized bits of Lisp, in practice this might be difficult. >From top to bottom, here are the implementation layers: ELEPHANT package persistent meta-object, persistent collections controller serializer memutils package UFFI / implementation specific stuff libsleepycat.so Sleepycat 4.2/3 While I loath specials, since you can't change the signature of slot accessors, in order to pass parameters to the database / serializer, specials are needed. Also specials will probably play nice with threaded lisps. ----------------------- CLASSES AND METACLASSES ----------------------- Persistent classes which the user defines are declared and instrumented by using the persistent-metaclass. Ideally creating persistent versions of class, slot-defintion, et al would be enough, but in reality various implementations do things in different ways. CMUCL / SBCL: their's a bit of work to make class slot allocation and reader / writer / slot-boundp work right. Allegro: is using slot-boundp instead of slot-boundp-using-class inside of shared-initialize, which necessitates some work. CMUCL doesn't do non-standard allocation types correctly, so we've created our own slot definition keyword :transient. In the future this will change. Andrew will add some notes here in the future. ----------- COLLECTIONS ----------- While we support serializing and persisting a wide class of Lisp data types, there are problems with persisting aggregate types (conses, lists, arrays, objects, hash-tables...) 1) not automatic: there's no way for elephant to know when you've changed a value in an aggregate object, so you have to manually restore it back into the slot to get it saved. example 1: you put a cons into the database. you change it's car. this is not saved unless you resave the cons into the database. example 2: slot-1 of obj A (saved in the database) contains a cons. you change the car of the cons. this is not reflected into the database unless you resave A. 2) merge-conflicts: changing one value and saving an aggregate will write out the whole aggregate, possibly blowing away changes other threads have made behind your back. this is not protected by transactions! 3) consing, non-lazy and expensive (de)serialization: you have to serialize/deserialize the entire aggregate every time you save it or restore it. This is pretty fast all things considered, but it's probably better to use persistent collections. 4) you have to store the entire collection in memory, whereas one of the points of the database to store large collections of objects..... For these and other reasons, we provide a hash-table-like interface to Berkeley BTrees. These have many advantages over ordinary hash-tables from the point of view of persistence. There is a separate table for BTrees. This is because we use a hand coded C function for sorting, which understands a little of the serialized data. It can handle numbers (up to 64-bit bignums -- they are approximated by floats) and strings (case-insensitive for 8-bit, code-point-order for 16-bit Unicode.) It should be fast but we don't want a performance penalty on objects. Secondary indices are mostly handled on the lisp side, because of our weird table layout (see below) and to avoid crossing FFI boundaries. Some unscientific microbenchmarks indicated that there was no performance benefit on CMUCL / SBCL, and only minor benefit (asymptotically nil) on OpenMCL. They have a separate table. Actually two handles are opened on this table: one which is plain, and one which is associated to the primary btree table by a no-op indexing function. Since we maintain the secondary keys ourselves, the associated handle is good for gets / cursor traversals. We use the unassociated handle for updates. ---------- CONTROLLER ---------- The controller is accessed through the special *store-controller*. The controller keeps track of 1) the environment handle 2) the DB handle(s) 3) the instance cache 4) the root object The environment handle and DB handle currently aren't really exposed. Eventually they should be, so that tuning flags can be set on them. OIDs are generated by a bit of C code, which isn't great, nor that safe (to get acceptable performance i use DB_TXN_NOSYNC.) Waiting for Sleepycat 4.3. The instance cache is implemented as a values-weak hash-table. This is a hash-table where the values can be collected, and when they are, the entire key-value entry is deleted. There are implementations of this on the various platforms. The instance cache is there to make deserialization of persistant objects faster. Since we [249 lines skipped] --- /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 02:31:15 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 02:31:15 1.1 [498 lines skipped] --- /project/elephant/cvsroot/elephant/doc/oldfiles/TUTORIAL 2007/04/28 02:31:15 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/TUTORIAL 2007/04/28 02:31:15 1.1 [923 lines skipped]

1 0

[elephant-cvs] CVS elephant/src/db-bdb
by ieslick 28 Apr '07

28 Apr '07

Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv16753/src/db-bdb Modified Files: bdb-transactions.lisp Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/27 13:32:14 1.15 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/28 02:31:15 1.16 @@ -31,15 +31,13 @@ for count fixnum from 1 to retries for success of-type boolean = nil do - (let ((txn - (if transaction transaction - (db-transaction-begin env + (let ((txn (db-transaction-begin env :parent (if parent parent +NULL-VOID+) :degree-2 degree-2 :read-uncommitted read-uncommitted :txn-nosync txn-nosync :txn-nowait txn-nowait - :txn-sync txn-sync)))) + :txn-sync txn-sync))) (declare (type pointer-void txn)) (let (result) (let ((*current-transaction* (make-transaction-record sc txn))

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

elephant-cvs April 2007