elephant-cvs April 2007

elephant-cvs@common-lisp.net

2 participants
99 discussions

[elephant-cvs] CVS elephant/doc
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv575 Modified Files: elephant-design.texinfo Log Message: Few tweaks to docs to build them; updated docs on website to match latest checkins --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/27 03:14:55 1.5 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/30 02:33:17 1.6 @@ -13,7 +13,7 @@ The architecture has been carefully modularized: -(a)image{ArchDiagram,,4.5in,[Architecture Diagram],.png} +(a)image{ArchDiagram,,4.5in,[Architecture Diagram],png} To get a feeling for what is happening inside elephant, it is probably best to walk through the various major protocols to see how these @@ -176,7 +176,7 @@ The BTree protocol almost requires an actual BTree implementation to be at all efficient. Keys and values need to be accessible via the cursor API, which means they need to be walked linearly in the sort -order of the keys (described in @xref{Persistent BTrees}). +order of the keys (described in @ref{Persistent BTrees}). An indexed BTree automatically maintains a hash table of the indices defined on it so that users can access them by mapping or

1 0

[elephant-cvs] CVS elephant
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv11649 Modified Files: elephant.asd Log Message: Typo --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:38:29 1.47 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:54:03 1.48 @@ -256,7 +256,7 @@ (error "Could not load ~A" file)) (format t "Loaded ~A~%" file)) (let* ((root-dir (asdf:component-pathname (asdf:component-system c))) - (root-library-file (merge-pathnames (file-namestring file) root-path))) + (root-library-file (merge-pathnames (file-namestring file) root-dir))) (or (uffi-funcall :load-foreign-library root-library-file :module (component-name c))

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv8074/src/elephant Modified Files: variables.lisp Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/25 02:28:02 1.17 +++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/30 00:38:31 1.18 @@ -42,7 +42,7 @@ ;;;; General support for user configurable parameters (defvar *user-configurable-parameters* - '((:map-using-degree2 *map-using-degree2*) + '((:berkeley-db-map-degree2 *map-using-degree2*) (:berkeley-db-cachesize *berkeley-db-cachesize*))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

1 0

[elephant-cvs] CVS elephant
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv8074 Modified Files: config.sexp ele-clsql.asd elephant.asd Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/config.sexp 2007/04/25 02:27:56 1.11 +++ /project/elephant/cvsroot/elephant/config.sexp 2007/04/30 00:38:29 1.12 @@ -1,48 +1,63 @@ -#+(or sbcl allegro) -((:berkeley-db-include-dir . "/opt/local/include/db45/") +#+(and (or sbcl allegro) (not mswindows) (not windows)) +((:compiler . :gcc) + (:berkeley-db-include-dir . "/opt/local/include/db45/") (:berkeley-db-lib-dir . "/opt/local/lib/db45/") (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) (:berkeley-db-cachesize . 20971520) - (:clsql-lib . nil) - (:compiler . :gcc)) + (:berkeley-db-map-degree2 . t) + (:clsql-lib-paths . nil) + (:prebuilt-libraries . nil)) + #+openmcl -((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") +((:compiler . :gcc) + (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.dylib") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :gcc)) + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:prebuilt-libraries . nil) + (:clsql-lib-paths . nil)) #+(and lispworks (not mswindows)) -((:berkeley-db-include-dir . "/opt/local/include/db45/") - (:berkeley-db-lib-dir . "/opt/local/lib/db45/") - (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") - (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :gcc)) +((:compiler . :gcc) + (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.dylib") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:prebuilt-libraries . nil) + (:clsql-lib-paths . nil)) #+(or mswindows windows) -((:berkeley-db-include-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/include/") +((:compiler . :cygwin) + (:berkeley-db-include-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/include/") (:berkeley-db-lib-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/") (:berkeley-db-lib . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/libdb45.dll") (:berkeley-db-deadlock . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/db_deadlock.exe") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :cygwin)) + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:clsql-lib-paths . nil) + (:prebuilt-libraries . t)) ;; Berkeley 4.5 is required, each system will have different settings for ;; these directories, use this as an indication of what each key means ;; -;; Typical pthread settings are: /lib/tls/libpthread.so.0 -;; -;; nil means that the library in question is not loaded -;; -;; NOTE: The latest SBCL (0.9.17+) on linux no longer needs the pthread library, -;; it is statically linked against it now with the new thread support +;; :prebuilt-libraries is true by default for windows machines. It causes +;; the library loader to look in the elephant root directory for the shared +;; libraries. (nil or t) +;; +;; :clsql-lib-paths tell clsql where to look for which ever SQL distribution +;; library files you need it to look for. For example... +;; +;; (:clsql-lib-paths . ("/Users/me/Work/SQlite3/" "/Users/me/Work/Postgresql/")) +;; +;; :pthread-lib is deprecated, for old versions of sbcl prior to 0.9.17 that +;; did not have pthreads compiled in. If you are using an old version, we +;; recommend that you upgrade! +;; Typical pthread settings were: /lib/tls/libpthread.so.0 ;; ;; :compiler options are ;; :gcc (default: for unix platforms with /usr/bin/gcc) @@ -52,16 +67,18 @@ ;; Additional supported parameters include: ;; ;; -;; :berkeley-db-cachesize - an integer indicating the number of bytes -;; for the page cache, default 10MB which is -;; about enough storage for 10k-30k indexed -;; persistent objects -;; -;; :map-using-degree2 - Boolean parameter that indicates whether map -;; operations lock down the btree for the entire -;; transaction or whether they allow other -;; transactions to add/delete/modify values -;; before the map operation is completed. The -;; map operation remains stable and any writes -;; are kept transactional, see user manual as -;; well as berkeley DB docs for more details +;; :berkeley-db-cachesize +;; An integer indicating the number of bytes +;; for the page cache, default 20MB which is +;; about enough storage for 10k-30k indexed +;; persistent objects +;; +;; :berkeley-db-map-degree2 +;; Boolean parameter that indicates whether map +;; operations lock down the btree for the entire +;; transaction or whether they allow other +;; transactions to add/delete/modify values +;; before the map operation is completed. The +;; map operation remains stable and any writes +;; are kept transactional, see user manual as +;; well as berkeley DB docs for more details. --- /project/elephant/cvsroot/elephant/ele-clsql.asd 2007/03/23 16:31:52 1.10 +++ /project/elephant/cvsroot/elephant/ele-clsql.asd 2007/04/30 00:38:29 1.11 @@ -16,29 +16,33 @@ ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. -;; (eval-when (:compile-toplevel :load-toplevel :execute) -;; (unless (find-package 'clsql) -;; (asdf:operate 'asdf:load-op 'clsql))) - -;; ;; clsql needs to know how to find libaries to various databases.... -;; (defvar *clsql-foreign-lib-path* "/usr/lib") -;; (defvar *elephant-lib-path* "/usr/local/share/common-lisp/elephant/") -;; (clsql:push-library-path *clsql-foreign-lib-path*) -;; (clsql:push-library-path *elephant-lib-path*) -;; ) - -;;(defparameter *clsql-foreign-lib-path* #p"/usr/lib") -;;(defparameter *elephant-lib-path* #p"/usr/local/share/common-lisp/elephant/") - -;;(defmethod asdf:perform :after ((o asdf:load-op) -; (c (eql (asdf:find-system 'clsql)))) -;; (let ((plp (find-symbol (symbol-name '#:push-library-path) -;; (find-package 'clsql)))) -;; (funcall plp -;; *clsql-foreign-lib-path*) -;; (funcall plp -;; *elephant-lib-path*) -;;)) +(in-package :cl-user) + +;; Forward definition +(defpackage elephant-system + (:use :cl :asdf) + (:export :elephant-c-source :compiler-options :foreign-libraries-to-load-first :get-config-option)) + +(defpackage ele-clsql-system + (:use :cl :asdf :elephant-system)) + +(in-package :ele-clsql-system) + +;; +;; Make sure asdf is loaded prior to evaluating the rest of the file (tacky) +;; +(eval-when (:compile-toplevel :load-toplevel :execute) + (unless (find-package 'clsql) + (asdf:operate 'asdf:load-op 'clsql))) + +(defmethod asdf:perform :after ((o asdf:load-op) + (c (eql (asdf:find-system 'clsql)))) + (let ((paths (get-config-option :clsql-lib-paths (find-system :elephant))) + (plp (find-symbol (symbol-name '#:push-library-path) + (find-package 'clsql)))) + (loop for path in paths do + (format t "Pushing ~A onto clsql::*FOREIGN-LIBRARY-SEARCH-PATHS*~%" path) + (funcall plp (pathname path))))) (defsystem ele-clsql :name "elephant" --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/29 02:10:54 1.46 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:38:29 1.47 @@ -94,6 +94,7 @@ "Run the appropriate compiler for this platform on the source, getting the specific options from 'compiler-options method. Default options can be overridden or augmented by subclass methods" + (unless (get-config-option :prebuilt-libraries c) #+(or mswindows windows) (progn (let* ((pathname (component-pathname c)) @@ -144,7 +145,7 @@ (compiler-options (c-compiler c) c :input-file (namestring (component-pathname c)) :output-file (namestring (first (output-files o c)))))) - (error 'operation-error :component c :operation o))) + (error 'operation-error :component c :operation o)))) #| (defmethod perform ((o compile-op) (c elephant-c-source)) @@ -247,9 +248,21 @@ (error "Could not load ~A into ~A" file module))))) ;; Load the compiled libraries (dolist (file (output-files (make-instance 'compile-op) c)) - (format t "Loading ~A~%" file) - (or (uffi-funcall :load-foreign-library file :module (component-name c)) - (error "Could not load ~A" file)))) + (format t "Attempting to load ~A...~%" (file-namestring file)) + (if (and (probe-file file) + (not (get-config-option :prebuilt-libraries c))) + (progn + (or (uffi-funcall :load-foreign-library file :module (component-name c)) + (error "Could not load ~A" file)) + (format t "Loaded ~A~%" file)) + (let* ((root-dir (asdf:component-pathname (asdf:component-system c))) + (root-library-file (merge-pathnames (file-namestring file) root-path))) + (or (uffi-funcall :load-foreign-library + root-library-file + :module (component-name c)) + (error "Output file ~A not found in elephant root" (namestring root-library-file))) + (format t "Loaded ~A~%" root-library-file))))) + (defmethod operation-done-p ((o load-op) (c elephant-c-source)) nil) @@ -264,7 +277,7 @@ (defsystem elephant :name "elephant" :author "Ben Lee <blee(a)common-lisp.net>" - :version "0.6.1" + :version "0.9" :maintainer "Robert Read <read(a)robertlread.net>" :licence "LLGPL" :description "Object database for Common Lisp"

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 30 Apr '07

30 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv8074/doc Modified Files: installation.texinfo user-guide.texinfo Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/29 21:54:51 1.14 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/30 00:38:31 1.15 @@ -246,6 +246,10 @@ @comment node-name, next, previous, up @section Setting up Berkeley DB +We recommend that you download and build a distribution from Oracle. +Some problems have been reported with linking to Debian, Cygwin or +other packages. This is especially true for Windows users. + Beyond ensuring that the file ``my-config.sexp'' points to your BDB installation directories and files, nothing else should be required to configure the example that uses a local ``testdb'' directory as a @@ -428,13 +432,20 @@ @comment node-name, next, previous, up @section Elephant on Windows -The build process on Windows currently only works with GCC under Cygwin. -The process can be a bit tricky, so if it doesn't work out of the box, -we recommend that you download the DLLs from the Elephant website -download page (@url{http://www.common-lisp.net/project/elephant/downloads.html'}). - -Unpack the .zip file into the elephant root directory. Then run the batch file -``copy-dll.bat'' to move them into their proper directory. +The build process on Windows currently only works with GCC under +Cygwin. The process can be a bit tricky, so if it doesn't work out of +the box or you don't want to install cygwin, we recommend that you +download the DLLs from the Elephant website download page +(@url{http://www.common-lisp.net/project/elephant/downloads.html'}). + +Unpack the .zip file into the elephant root directory. Ensure that +your @code{my-config.sexp} file configuration for Windows has +@code{:prebuilt-binaries} set to ``t'' so it will know to look in +the elephant root during the asdf loading process. + +For Berkeley DB users we recommend downloading the Windows binary +distribution of Berkeley DB 4.5 to minimize any potential linking +issues. @node Test Suites @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 14:24:57 1.21 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/30 00:38:31 1.22 @@ -1555,7 +1555,7 @@ described in config.sexp. They are: @itemize -@item @code{:map-using-degree2} - Improve the efficiency of cursor traversals +@item @code{:berkeley-db-map-degree2} - Improve the efficiency of cursor traversals in the various mapping functions. Defaults to true, meaning a value you just read while mapping may change before the traversal is done. So if you operate only on the current cursor location, you are

1 0

[elephant-cvs] CVS elephant/src/utils
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/utils In directory clnet:/tmp/cvs-serv3086/src/utils Modified Files: os.lisp Log Message: Fix for lispworks deadlock detect (silently fails instead of asserting an error) --- /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/29 02:10:56 1.4 +++ /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/29 21:54:51 1.5 @@ -52,7 +52,7 @@ (in-directory (directory) (ccl:run-program program args :wait nil)) #+lispworks - (apply #'sys::call-system + (funcall #'sys::call-system (format nil "~a~{ '~a'~} &" program args) :current-directory directory :wait nil)

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv3086/doc Modified Files: installation.texinfo Log Message: Fix for lispworks deadlock detect (silently fails instead of asserting an error) --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 14:24:57 1.13 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/29 21:54:51 1.14 @@ -36,8 +36,10 @@ Elephant supports SBCL, Allegro, Lispworks, OpenMCL and CMUCL. Each lisp is supported on each of the platforms it runs on: Mac OS X, Linux and Windows. As of release 0.6.1, both 32-bit and 64-bit systems -should be supported. Elephant has a small developer base and as of -the writing of this manual, there are: +should be supported. + +Due to the small number of developers and the large number of configurations +providing full test coverage is problematic. There are: @enumerate @item Five lisp environments @@ -46,16 +48,14 @@ @item Three data store configurations: Berkeley DB, SQLite3 and Postgresql @end enumerate -This means that the total number of combinations that should be tested -comes to: +which means that the total number of combinations to be tested could be as +much as: @math{lisps * os * radix * dstore = 5 * 3 * 2 * 3 = 90 configurations} -Of course not all of these combinations are valid, but the -implications of these combinatorics is that not every combination will -be tested in any given release. The developers and active user base -currently cover all three data store configurations on the following -platforms: +Not all of these combinations are valid, but the implication is that +not every combination will be tested in any given release. The +developers and user base regularly use the following platforms @itemize @item 32/64-bit SBCL on Linux and Mac OS X @@ -63,13 +63,15 @@ @item 32-bit Allegro on Mac OS X @end itemize -The developers will do their best to accomodate users who are keen to -test other combinations, but practically these configurations will be -the most stable and reliable. Elephant is becoming quite stable in -general, so don't be afriad to try an unemphasized combination - -chances are it is just a little more work to bring it up. In particular, -Elephant can probably work with MySQL or Oracle with just a little work, -but nobody has asked for this yet. +The CLSQL backend is used predominantly under SBCL on Linux and Mac OS +X at the time of writing. The developers will do their best to +accomodate users who are keen to test other combinations, but the +above configurations will be the most stable and reliable. + +Elephant is now quite stable in general, so don't be afraid to try an +unemphasized combination - chances are it is just a little more work +to bring it up. In particular, Elephant can probably work with MySQL +or Oracle with just a little work, but nobody has asked for this yet. @subsection Library dependencies @@ -77,7 +79,7 @@ @enumerate @item asdf -- @uref{http://www.cliki.net/asdf} -@item uffi -- version 1.5.17 or later, @uref{http://uffi.b9.com/} or @uref{http://www.cliki.net/UFFI} +@item uffi -- requires version 1.5.18 or later, @uref{http://uffi.b9.com/} or @uref{http://www.cliki.net/UFFI} @item cl-base64 -- @uref{http://www.cliki.net/cl-base64} @item gcc -- Your system needs GCC (or Cygwin) to build the Elephant C-based serializer library. (Precompiled DLL's are available for Windows platforms on the @uref{http://www.common-lisp.net/project/elephant/downloads.html, download page}. @item rt -- The RT regression test sytem is required to run the test suite: @uref{http://www.cliki.net/RT} @@ -86,7 +88,8 @@ Follow the instructions at these URLs to download and setup the libraries. (Note: uffi and cl-base64 are @uref{http://www.cliki.net/ASDF-Install, asdf-installable} for those -of you with asdf-install on your system) +of you with asdf-install on your system). Elephant, however, is not +asdf-installable today. In addition to these libraries, each data store has their own dependencies as discussed in @ref{Berkeley DB} and @@ -97,10 +100,11 @@ @section Configuring Elephant Before you can load the elephant packages into your running lisp, you -need to setup the configuration file. First, copy the reference file -config.sexp from the root directory to my-config.sexp. my-config.sexp -contains a lisp reader-formatted list of key-value pairs that tells -elephant where to find various libraries, how to build libraries, etc. +need to setup the configuration file. Copy the reference file +config.sexp from the root directory to my-config.sexp in the root +directory. my-config.sexp contains a lisp reader-formatted list of +key-value pairs that tells elephant where to find various libraries +and how to build them. For example: @@ -110,8 +114,6 @@ (:berkeley-db-lib-dir . "/opt/local/lib/db45/") (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) - (:clsql-lib . nil) (:compiler . :gcc)) @end lisp @@ -132,9 +134,9 @@ The config.sexp file contains a set of example configurations to start from, but you will most likely need to modify it for your system. -Elephant has one small C library that it uses for binary serialization -which means that you need to have gcc in your path (@pxref{Elephant on -Windows} for exceptions on the Windows platform). +Elephant has one small C library that it uses for binary +serialization. This means that you need to have gcc in your path +(@pxref{Elephant on Windows} for exceptions on the Windows platform). @node Loading Elephant @comment node-name, next, previous, up @@ -152,12 +154,11 @@ This will load the cl-base64 and uffi libraries. It will also automatically compile and load the C library. The build process no -longer depends on a Makefile. This build process has been verified on -most platforms, but if you have a problem please report it, and any -output you can capture, to the developers at -@email{elephant-devel@@common-lisp.net}. We will update the FAQ at -@uref{http://trac.common-lisp.net/elephant} with common problems users -run into. +longer depends on a Makefile and has been verified on most platforms, +but if you have a problem please report it, and any output you can +capture, to the developers at @email{elephant-devel@@common-lisp.net}. +We will update the FAQ at @uref{http://trac.common-lisp.net/elephant} +with common problems users run into. @subsection Two-Phase Load Process @@ -170,8 +171,7 @@ (NOTE: There are some good reasons and not so good reasons for this process. One reason you cannot load ele-bdb.asd directly as it depends on lisp code defined in elephant.asd. We decided not to fix -this in this release although later releases may avoid the oddity of -the two phase loading) +this in the 0.9 release although later releases may improve on this). @subsection Packages @@ -199,7 +199,8 @@ (:use :common-lisp :elephant)) @end lisp -The symbols that are imported are captured in @ref{User API Reference} +The imported symbols are all that is needed to control Elephant +databases and are documented in detail in @ref{User API Reference} @subsection Opening a Store @@ -224,19 +225,18 @@ The first time you load a specific data store, Elephant will call ASDF to load all the specified data store's dependencies, connect to a database and return the @code{store-controller} subclass instance for -that backend. +that data store. @node Berkeley DB @comment node-name, next, previous, up @section Berkeley DB -The Berkeley DB Data Store started out as a very simple data dictionary in the -Berkeley Unix operating system. There are many ``Xdb'' systems that use the -same API, or a similarly one. A commercial version of the BDB was provided by -Sleepycat Software, and this product has since been purchased by Oracle corporation, -and is currently distributed under a similar licensing arrangement, with commercial -support also available. Please follow the download and installation procedures -defined here: +The Berkeley DB Data Store started out as a very simple data +dictionary in the Berkeley Unix operating system. There are many +``Xdb'' systems that use the same API, or a similarly one. A free for +non-commercial use version of Berkeley DB is provided by Oracle +corporation with commercial licenses available. Please follow the +download and installation procedures defined here: http://www.oracle.com/technology/products/berkeley-db/db/index.html @@ -246,68 +246,71 @@ @comment node-name, next, previous, up @section Setting up Berkeley DB -Beyond ensuring that the file ``my-config.sexp'' points correctly -to your BDB installation, nothing else should be required to configure -the example that uses a local ``testdb'' directory as a dabase (under ``tests'') -in the top-level Elephant directory. +Beyond ensuring that the file ``my-config.sexp'' points to your BDB +installation directories and files, nothing else should be required to +configure the example that uses a local ``testdb'' directory as a +dabase (under ``tests'') in the top-level Elephant directory. On one Fedora based system, the ``my-config.sexp'' file looked like this: @lisp ((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include") - (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") - (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") - (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") (:pthread-lib . nil) - (:clsql-lib . "/usr/local/share/common-lisp/") - (:compiler . :gcc)) + (:clsql-lib . "/usr/local/share/common-lisp/") + (:compiler . :gcc)) @end lisp The @ref{Test Suites} give a nice example of using BDB by running the test using the specification: @lisp -'(:BDB "/home/me/db/testdb/")) +'(:BDB "<elephant-root>/tests/testdb/") @end lisp -Once you start working on an application, you will want to change this path -to a directory that make sense for you application, and use that as -the specification passed to -@lisp open-store -@end lisp -in your application. +Once you start working on an application, you will want to change the +path to a directory that is appropriate for your application, and use +that as the specification passed to @code{ open-store } on application +startup. @node Upgrading Berkeley DB Databases @comment node-name, next, previous, up @section Upgrading Berkeley DB Databases -At regular intervals, Elephant will require a specific version of Berkeley DB -because it does not parse the header files which tend to change with each release. -The patches are usually minor, but sometimes Elephant also depends on new -features of Berkeley DB. In this case, you have to update both the Berkeley DB -database files as well as any data Elephant has built within it. That creates -some special constraints for upgrading databases. +When there is a new release of Elephant, it will depend on a new +version of Berkeley DB. If so, you must upgrade your BDB databases to +use the new version Elephant. This forced upgrade is a consequence of +Elephant not parsing the BDB header files which tend to change various +important constants with each release. These patches are usually +minor. Upgrading also happens because Elephant tries to leverage new +features of Berkeley DB. + +The rest of this section talks about how to upgrade your existing +Berkeley DB databases, opening them in the new Elephant version and +migrating them to a newly created Elephant database. @subsection Upgrading to 0.9 -This section outlines how to upgrade from Elephant code base version 0.6.0 -using Berkeley DB 4.3. +This section outlines how to upgrade from Elephant version 0.6.0 and +Berkeley DB 4.3. @enumerate @item Install BDB 4.5 (keep 4.3 around for now) -@item Update my-config.sexp to point to the appropriate BDB 4.5 directories -@item Upgrade your database directory to 4.5 +@item Setup my-config.sexp to point to the appropriate BDB 4.5 directories +@item Upgrade your existing database directory to 4.5 @itemize @item Run db43_recover in your 0.6 database @item Optional: run db43_archive -d to remove all logs not part of a checkpoint This will make catastrophic recovery impossible, but reduces the amount of data you have to backup. @item Backup your db files and remaining logs - @item Run db45_checkpoint -1 in your 0.6 database directory + @item Run db45_checkpoint -1 in the database directory @end itemize -@item Migrate 0.6 data to a new 0.6.1 database +@item Upgrade 0.6 data to a fresh 0.9 database @itemize @item Open your old database: @code{(setf sc (open-store '(:BDB "/Users/me/db/ele060/")))} - @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele061/"))} + @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele090/"))} @end itemize @item Test your new application and report any bugs that arise to @email{elephant-devel@@common-lisp.net} @end enumerate @@ -315,14 +318,15 @@ @emph{(NOTE: close-store may fail when closing the old 0.6 database, this is OK.)} @emph{(NOTE: 64-bit lisps will not successfully upgrade 32-bit 0.6 databases. Use a 32-bit - version of your lisp to update to 0.6.1 and then open that database in your 64-bit + version of your lisp to update to 0.9 and then open that database in your 64-bit lisp. There should be no compatibility problems. Best to test your application on a 32-bit lisp if you can, just to be sure.)} @subsection Upgrade from Elephant 0.5 -Follow the upgrade procedures outlined in Elephant release 0.6.0 to migrate your database -from 0.5 to 0.6. Then follow the above procedures for upgrading from an 0.6 database. +Follow the upgrade procedures outlined in the Elephant 0.6.0 INSTALL +file to upgrade your database from 0.5 to 0.6.0. Then follow the above +procedures for upgrading to 0.9. @emph{(NOTE: It may not take much work to make 0.9 upgrade directly from 0.5 However there are so few (none?) 0.5 users that it wasn't deemed worth @@ -333,55 +337,50 @@ @section CL-SQL Although originally designed as an interface to the BerkeleyDB system, -the original Elephant system has been experimenetally extended to -support the use of relational database management systems as the -implementation of the persistent store. This relies on Kevin Rosenberg's -CL-SQL interface to a large number of relational systems. - -Although the BerkeleyDB system is an ideal object store for LISP objects, -one might prefer the licensing of a different system. For example, at -the time of this writing, it is my interpretation that one cannot use -the BerkeleyDB system behind a public website +the original Elephant system has been extended to support the use of +relational database management systems as the implementation of the +persistent store. This relies on Kevin Rosenberg's CL-SQL interface, +which provides access to a large number of relational systems. + +A major motivation of this extension is that one one might prefer the +licensing of a different system. For example, at the time of this +writing, it is our interpretation that one cannot use the BerkeleyDB +system behind a public website http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source. Neither the PostGres DBMS nor SQLite 3, nor Elephant itself, imposes -any such restriction. (Elephant is released under the LLGPL -(http://opensource.franz.com/preamble.html .) Older versions were -released under the GPL.) +any such restriction. Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application needs to use the truly relational aspects of those systems, preference for the tools associated with those systems, etc. -The SQL back-end extention of Elephant provides a function for migrating -data seamlessly between repositories. One can quite easily move -data from a BerkeleyDB repository to a PostGres repository, and vice versa. -This offers at least the possiblity than once can develop using one backend, -for example BerkeleyDB, and then later move to Postgres, or vice versa. -One could even operate simultaneously out of multiple repositories, if -there were a good reason to do so. - -At the time of this writing, the basic strategy for the SQL implementation -is quite simple. The same serializer used for the Sleepycat implementation -is employed, the byte-string is base64 encoded, and placed in a single -table which is managed by Elephant. - -All functionality except nested transaction support and cursor-puts -supported by the BerkeleyDB backend is supported by the CL-SQL -back-end. Concurrency and transaction atomicity have not been stress -tested well for the CL-SQL based system. - -Additionally, it is NOT the case that the Elephant system currently provides -transaction support across multiple repositories; it provides the transaction -support provided by the underlying repository to the user in a per-repository -basis. - -The PostGres backend is as currently employed is about 5 times slower than -the BerkeleyDB backend. This could probably change with continued development. - -CL-SQL supports a lot of DBMS systems, but only PostGres and SqlLite 3 have been tested. +Elephant provides functions for migrating data seamlessly between data +stores. One can quite easily move data from a BerkeleyDB repository +to a PostGres repository, and vice versa. This offers at least the +possibility than one can develop using one data store, for example +BerkeleyDB, and then later move to Postgres. One could even operate +simultaneously out of multiple repositories, if there were a good +reason to do so. + +The SQL implementation shares the serializer with the BDB data store, but +base64 encodes the resulting binary stream. This data is placed into +a single table in the SQL data store. + +All functionality except for nested transaction support and +cursor-puts supported by the BerkeleyDB data store is supported by the +CL-SQL data store. CL-SQL transaction integrity under concurrent +operation has not been extensively stress tested. + +Additionally, it is NOT the case that the Elephant system currently +provides transaction support across multiple repositories; it provides +transaction support on a per-repository basis. + +The PostGres backend is currently about 5 times slower than the +BerkeleyDB backend. As of the time of this writing, only PostGres and +SqlLite 3 have been tested as CL-SQL backends. @node CL-SQL Example @comment node-name, next, previous, up @@ -391,20 +390,20 @@ @enumerate @item Install postgres and make sure postmaster is running. -Postgres may be installed on your system; you may be able to use a package manager to -install it, or you can install it quite easily from the PostgresSQL site directly -(http://www.postgresql.org/). +Postgres may be installed on your system; you may be able to use a [104 lines skipped]

1 0

[elephant-cvs] CVS elephant/src/db-clsql
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv6510/src/db-clsql Modified Files: sql-collections.lisp Log Message: Oops, cleanup debug stmts --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:28 1.14 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:54 1.15 @@ -518,12 +518,11 @@ (cursor-prev-x cursor) ) (defmethod cursor-prev-x ((cursor sql-secondary-cursor) &key (returnpk nil)) -;; (declare (optimize (speed 3))) + (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (progn (let ((prior-pk (get-current-key cursor))) (decf (sql-crsr-ck cursor)) - (format t "prior-key: ~A current-key: ~A%" prior-pk (get-current-key cursor)) (if (eq prior-pk (get-current-key cursor)) (setf (dp-nmbr cursor) (max 0 (- (dp-nmbr cursor) 1))) (setf (dp-nmbr cursor)

1 0

[elephant-cvs] CVS elephant/src/db-clsql
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv6425/src/db-clsql Modified Files: sql-collections.lisp Log Message: Fixed fencepost error in cursor-prev-x in clsql data store --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/12 02:47:24 1.13 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:28 1.14 @@ -518,18 +518,19 @@ (cursor-prev-x cursor) ) (defmethod cursor-prev-x ((cursor sql-secondary-cursor) &key (returnpk nil)) - (declare (optimize (speed 3))) +;; (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (progn - (let ((cur-pk (get-current-key cursor))) + (let ((prior-pk (get-current-key cursor))) (decf (sql-crsr-ck cursor)) - (if (equal cur-pk (get-current-key cursor)) + (format t "prior-key: ~A current-key: ~A%" prior-pk (get-current-key cursor)) + (if (eq prior-pk (get-current-key cursor)) (setf (dp-nmbr cursor) (max 0 (- (dp-nmbr cursor) 1))) - (setf (dp-nmbr cursor) - (sql-get-from-clcn-cnt (cursor-oid cursor) - (get-current-key cursor) - (get-con (cursor-btree cursor)) - )))) + (setf (dp-nmbr cursor) + (1- (sql-get-from-clcn-cnt (cursor-oid cursor) + (get-current-key cursor) + (get-con (cursor-btree cursor)))) + ))) (has-key-value-scnd cursor :returnpk returnpk)) (cursor-last-x cursor :returnpk returnpk)))

1 0

[elephant-cvs] CVS elephant/tests
by ieslick 29 Apr '07

29 Apr '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv12328/tests Modified Files: testmigration.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/tests/testmigration.lisp 2007/03/24 10:51:45 1.20 +++ /project/elephant/cvsroot/elephant/tests/testmigration.lisp 2007/04/29 02:11:02 1.21 @@ -34,7 +34,7 @@ (progn (format t "~%Single store mode: ignoring") t) - (let* ((*store-controller*) + (let* (*store-controller* (sc1 (open-store *test-spec-primary* :recover t :deadlock-detect t)) (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect t))) (unwind-protect

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

elephant-cvs April 2007