- elephant-cvs@lists.common-lisp.net - mailman3.common-lisp.net

[elephant-cvs] CVS elephant/tests
by ieslick 30 Mar '07

30 Mar '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv31032/tests Modified Files: elephant-tests.lisp Log Message: Significant documentation string and documentation edits towards 0.6.1 manual. Clean up packages so elephant exports user visible symbols and backend exports backend-relevant symbols. Change required fix in serializer packages also. Added :elephant-user package. --- /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/03/12 01:32:06 1.30 +++ /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/03/30 14:34:35 1.31 @@ -19,7 +19,7 @@ (defpackage elephant-tests (:nicknames :ele-tests) (:use :common-lisp :elephant :regression-test) - (:import-from :ele + (:import-from :elephant with-buffer-streams serialize deserialize)

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 30 Mar '07

30 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv31032/doc Modified Files: copying.texinfo data-store-reference.texinfo elephant-design.texinfo elephant.texinfo installation.texinfo make-ref.lisp reference.texinfo tutorial.texinfo user-guide.texinfo Log Message: Significant documentation string and documentation edits towards 0.6.1 manual. Clean up packages so elephant exports user visible symbols and backend exports backend-relevant symbols. Change required fix in serializer packages also. Added :elephant-user package. --- /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/03/24 12:16:02 1.3 +++ /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/03/30 14:34:34 1.4 @@ -1,38 +1,39 @@ @c -*-texinfo-*- -@node Copying +@node Copyright and License @comment node-name, next, previous, up -@chapter Copying -@cindex Copying +@chapter Copyright and License +@cindex Copyright and License @cindex License -@quotation -@b{Elephant}: an object-oriented database for Common Lisp. +@section Elephant Licensing + +@b{Elephant}: a persistent metaprotocol and object-oriented database +for Common Lisp. Homepage: @uref{http://www.common-lisp.net/project/elephant} +@quotation Elephant users are granted the rights to distribute and use this software as governed by the terms of the Lisp Lesser GNU Public License @uref{http://opensource.franz.com/preamble.html}, also known as the LLGPL. +@end quotation Copyrights include: +@quotation Copyright (c) 2004 by Andrew Blumberg and Ben Lee -Copyright (c) 2006-2007 by Ian Eslick - Copyright (c) 2005-2007 by Robert L. Read +Copyright (c) 2006-2007 by Ian Eslick +@end quotation -Portions of this program (namely the C unicode string -sorter) are derived from IBM's @b{ICU}: - -@uref{http://oss.software.ibm.com/icu/} - -whose copyright and license follows the GPL below. - - +Portions of this program (namely the C unicode string sorter) are +derived from IBM's @b{ICU}: @uref{http://oss.software.ibm.com/icu/, +ICU Website} whose copyright and license follows below. +@quotation ICU License - ICU 1.8.1 and later COPYRIGHT AND PERMISSION NOTICE @@ -72,5 +73,32 @@ All trademarks and registered trademarks mentioned herein are the property of their respective owners. - @end quotation + +@section Data Store Licensing Considerations + +The Berkeley DB data store is based on the Berkeley DB C library, now +owned by Oracle, but available as GPL'ed software. It is important to +understand that applications using Berkeley DB must also be GPL'ed +unless you negotiate a commercial license from Oracle. In most +interpretations of the license, this includes a requirement to make +code available for the entirety of any publicly visible website that +is based on Berkeley DB. See + +@uref{http://www.oracle.com/@/technology/@/software/@/products/@/berkeley-db/@/htdocs/bdboslicense.html}. + +The CL-SQL backend, depending on which SQL engine you use, may not +carry this restriction and you can easily migrate data between the +two. Since the Berkeley DB store is 4-5x faster than SQL, it may make +sense to develop under BDB and transition to SQL after you've tuned +the performance of the application. Licenses for various SQL engines +can be found at: + +@itemize +@item SQLite: Public Domain, see @uref{http://www.sqlite.org/copyright.html, the SQLite license page} +@item Postgresql: BSD License, see @uref{http://www.postgresql.org/about/licence, the Postgresql license page} +@item MySQL: Dual licensing (similar to BDB), see @uref{http://www.mysql.com/company/legal/licensing/, the MySQL license page} +@end itemize + + + --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 1.1 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 14:34:34 1.2 @@ -7,24 +7,34 @@ @cindex Data Store @cindex API Reference -These are the functions that need to be overridden to implement -support for a data store backend. Included are the exported elephant -functions that need methods defined on them. Some functions here are -utilities from the main elephant package that support store -implementations. Migration, class indices and query interfaces are +This reference includes functions that need to be overridden, classes +inherited from or other action taken to implement support for a new +data store backend. Included are the exported elephant functions that +need methods defined on them as well as the backend-only functions +exported in backends.lisp. Some functions here are utilities from the +main elephant package that support store implementations, but are not +required. Migration, class indices and query interfaces are implemented on top of the store API and require no special support by implementors. +Because the number of backend implementors is small, this is a minimal +documentation set intended to serve as an initial guide and a +reference. However, it is anticipated that some interaction will be +needed with the developers to properly harden a datastore for release. + +The sections each contain a short guide and a list of functions +relevant to them. + @menu -* Registration:: Register the backend to parse controller specifications +* Registration:: Register the backend to parse controller specifications. * Store Controllers:: Subclassing the store controller. +* Handling Serialization:: Available facilities for serializing objects. +* C Utilities:: Writing primitive C types. * Slot access:: Support for metaprotocol slot access. * Collections:: BTrees and indices. * Cursors:: Traversing BTrees. * Transactions:: Transaction implementation. * Multithreading:: Multithreading considerations. -* Serialization:: Facilities for serializing objects. -* C Utilities:: Writing primitive C types. * Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries @end menu @@ -33,8 +43,17 @@ @section Registration @cindex Registration -@include includes/fun-elephant-register-backend-con-init.texinfo -@include includes/fun-elephant-lookup-backend-con-init.texinfo +Elephant looks at the first element of the specification list to +determine which backend code base to use. The master table for this +information is @code{*elephant-backends*} in elephant/controller.lisp. +This will need to be augmented for every backend with the +specification keyword tag to be used (such as @code{:BDB} or +@code{:CLSQL}) and the required asdf dependencies. + +In addition, the backend source should use an eval-when statement to +call the following function: + +@include includes/fun-elephant-backend-register-backend-con-init.texinfo @node Store Controllers @comment node-name, next, previous, up @@ -44,25 +63,22 @@ Subclass store-controller and implement store and close controller which are called by open-store and close-store respectively. -@include includes/fun-elephant-store-controller.texinfo +@include includes/class-elephant-backend-store-controller.texinfo @include includes/fun-elephant-backend-open-controller.texinfo @include includes/fun-elephant-backend-close-controller.texinfo -The slots for these accessors must be initialized. + @include includes/fun-elephant-backend-database-version.texinfo -@include includes/fun-elephant-backend-controller-serialize.texinfo -@include includes/fun-elephant-backend-controller-deserialize.texinfo -@include includes/fun-elephant-backend-root.texinfo -@include includes/fun-elephant-backend-class-root.texinfo These functions are important utilities for implementing store-controllers. -@include includes/fun-elephant-backend-oid.texinfo @include includes/fun-elephant-backend-get-con.texinfo +@include includes/fun-elephant-backend-oid.texinfo @include includes/fun-elephant-backend-next-oid.texinfo @include includes/fun-elephant-backend-connection-is-indeed-open.texinfo +@include includes/fun-elephant-get-user-configuration-parameter.texinfo @node Slot Access @comment node-name, next, previous, up @@ -82,17 +98,37 @@ @section Collections @cindex Collections -@c #:btree #:btree-index #:indexed-btree -@c #:build-indexed-btree #:build-btree #:existsp -@c #:map-indices +To support collections, the data store must subclass the following +classes. +@include includes/class-elephant-btree.texinfo.texinfo +@include includes/class-elephant-btree-index.texinfo +@include includes/class-elephant-indexed-btree.texinfo + +To create the backend-appropriate type of btree, the backend +implements these methods aginst their store-controller. + +@include includes/fun-elephant-build-btree.texinfo +@include includes/fun-elephant-build-indexed-btree.texinfo + +And every btree needs accessors, these must be implemented for btree, +indexed-btree and btree-index. + +@include includes/fun-elephant-get-value.texinfo +@include includes/fun-elephant-setf-get-value.texinfo +@include includes/fun-elephant-existsp.texinfo +@include includes/fun-elephant-remove-kv.texinfo + +@include includes/fun-elephant-map-indices.texinfo +@include includes/fun-elephant-get-index.texinfo +@include includes/fun-elephant-remove-index.texinfo @node Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors -@c #:cursor +@include includes/class-cursor.texinfo @c #:cursor-btree @c #:cursor-oid @c #:cursor-initialized-p @@ -106,19 +142,23 @@ @c #:make-transaction-record @c #:transaction-store @c #:transaction-object + @c #:execute-transaction @c #:controller-start-transaction @c #:controller-commit-transaction @c #:controller-abort-transaction -@node Multithreading +@node Multithreading Considerations @comment node-name, next, previous, up -@section Multithreading +@section Multithreading Considerations @cindex Multithreading -@node Serialization +@c utils locks +@c utils thread-vars + +@node Handling Serialization @comment node-name, next, previous, up -@section Serialization +@section Handling Serialization @cindex Serialization @c #:deserialize #:serialize --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 1.1 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/30 14:34:34 1.2 @@ -1,5 +1,16 @@ -Debugger entered--Lisp error: (void-variable Design) - eval(Design) - eval-last-sexp-1(nil) - eval-last-sexp(nil) - call-interactively(eval-last-sexp) + +@node Elephant Design +@comment node-name, next, previous, up +@section Elephant Design +@cindex design + +When the main elephant @code{open-store} function is called, it calls +@code{get-controller} which grabs an existing store controller if the +spec is identical, or builds a new controller. Building the +controller requires loading any dependencies via asdf, calling a +backend initialization function (if it is the first instance of that +backend being created), and then calling an initialization function +that returns a @code{store-controller} subclass instance specific to +that backend. + +Elephant than calls open-controller --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/24 12:16:02 1.5 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/30 14:34:34 1.6 @@ -5,8 +5,9 @@ @c %**end of header @copying -Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. -Copyright @copyright{} 2006-2007 Robert L. Read and Ian Eslick +Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. +2006 SQL Backend, Copyright @copyright{} 2006 Robert L. Read. +2007 Rewrite, Copyright @copyright{} 2007 Ian Eslick. @quotation Permission is granted to copy, distribute and/or modify this document @@ -52,7 +53,7 @@ * User API Reference:: Function and class documentation of the user API. * Elephant Design:: An overview of elephant's internal architecture. * Data Store API Reference:: Function level documentation for data store implementors. -* Copying:: Your rights and freedoms. +* Copyright and License:: Your rights and freedoms. @end menu @chapheading Appendices @@ -66,6 +67,7 @@ @end menu @node Table of Contents +@unnumbered @comment node-name, next, previous, up @contents --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/24 12:16:02 1.4 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/30 14:34:34 1.5 @@ -6,157 +6,232 @@ @cindex Installation @menu -* Installation Basics:: Basic installation -* Test-Suites:: Running the test suites -* Berkeley DB Introduction:: The Berkeley DB backend -* SQL Data Store:: The design and status of the SQL back-end extension. -* Lisp Data Store:: A native lisp-based repository. -* Multi-repository Operation:: Specifying repositories. -* Setting up PostGres:: An example. +* Requirements:: Supported lisps and required libraries. +* Configuring Elephant:: Setting up Elephant and the configuration file. +* Loading Elephant:: Loading Elephant and the data store loading protocol. +* Berkeley DB Data Store:: Installing support for the Berkeley DB data store +* Berkeley DB Examples:: An example of installing and running the Berkeley DB data store. +* CL-SQL Data Store:: Install and connecting to the CL-SQL data store +* CL-SQL Examples:: An example of using the CL-SQL data store. +* Elephant on Windows:: More details about running Elephant on Windows +* Test Suites:: How to run and interpret the output of the regression test suite +* Documentation:: Building documentation from texinfo sources. @end menu -@node Installation Basics +@node Requirements @comment node-name, next, previous, up -@section Installation +@section Requirements -Please see the file ``INSTALL'' in the source distribution for -more precise information; this is an overview. +Elephant is a multi-platform, multi-lisp and multi-backend system. As +such there is a great deal of complexity in testing. The system has +tried to minimize external dependencies as much as possible to ease +installation, but it still requires some patience and care to bring +Elephant up on any given platform. This section attempts to simplify +this for new users as much as possible. Patches and suggestions will +be gladly accepted. + +@subsection Supported Lisp, Platform and Data store combinations + +Elephant supports SBCL, Allegro, Lispworks, OpenMCL and CMUCL. Each +lisp is supported on each of the platforms it runs on: Mac OS X, Linux +and Windows. As of release 0.6.1, both 32-bit and 64-bit systems +should be supported. Elephant has a small developer base and as of +the writing of this manual, there are: -Installation of Elephant itself is easy because of the asdf system. -Just execute: -@lisp -(asdf:operate 'asdf:load-op :elephant) -@end lisp +@enumerate +@item Five lisp environments +@item Three Operating System platforms +@item 32-bit or 64-bit OS/compilation configuration +@item Three data store configurations: Berkeley DB, SQLite3 and Postgresql +@end enumerate -However, Elephant cannot function without a back-end repository. -Elephant presents exactly the same API no matter what you choose -as a repository. In most cases Elephant will automatically load -the backend you refer to with your controller spec when you call -open store. However, you may have to use asdf to load the -code that interfaces to particular repository system. +This means that the total number of combinations that should be tested +comes to: -The basic choices are to use the BerkeleyDB system or -a SQL based system. You must perform one of these: -@lisp -(asdf:operate 'asdf:load-op :ele-clsql) -(asdf:operate 'asdf:load-op :ele-bdb) -@end lisp +@math{lisps * os * radix * dstore = 5 * 3 * 2 * 3 = 90 configurations} -If you choose a SQL based system, you may have to -load a specific package for that system, such as: +Of course not all of these combinations are valid, but the +implications of these combinatorics is that not every combination will +be tested in any given release. The developers and active user base +currently cover all three data store configurations on the following +platforms: + +@itemize +@item 32/64-bit SBCL on Linux and Mac OS X +@item 32-bit Lispworks on Windows and Mac OS X +@item 32-bit Allegro on Mac OS X +@end itemize + +The developers will do their best to accomodate users who are keen to +test other combinations, but practically these configurations will be +the most stable and reliable. Elephant is becoming quite stable in +general, so don't be afriad to try an unemphasized combination - +chances are it is just a little more work to bring it up. -@lisp -(asdf:operate 'asdf:load-op :ele-sqlite3) -@end lisp -or, for Postgres, -@lisp -(asdf:oos 'asdf:load-op :clsql-postgresql-socket) -@end lisp +@subsection Library dependencies -You will have to have the CL-SQL package installed. Following the -documentation for CL-SQL under the section ``How CLSQL finds and loads foreign -libraries'' you may need to do something like: -@lisp -(clsql:push-library-path "/usr/lib/") -@end lisp +The Elephant core system requires: -before doing -@lisp -(asdf:oos 'asdf:load-op :clsql-postgresql-socket) -@end lisp +@enumerate +@item asdf -- @uref{http://www.cliki.net/asdf} +@item uffi -- version 1.5.17 or later, @uref{http://uffi.b9.com/} or @uref{http://www.cliki.net/UFFI} +@item cl-base64 -- @uref{http://www.cliki.net/cl-base64} +@item gcc -- Your system needs GCC (or Cygwin) to build the Elephant C-based serializer library. (Precompiled DLL's are available for Windows platforms on the @uref{http://www.common-lisp.net/project/elephant/downloads.html, download page}. +@item rt -- The RT regression test sytem is required to run the test suite: @uref{http://www.cliki.net/RT} +@end enumerate + +Follow the instructions at these URLs to download and setup the +libraries. (Note: uffi and cl-base64 are +@uref{http://www.cliki.net/ASDF-Install, asdf-installable} for those +of you with asdf-install on your system) + +In addition to these libraries, each data store has their own +dependencies as discussed in @ref{Berkeley DB Data Store} and +@ref{CL-SQL Data Store}. + +@node Configuring Elephant +@comment node-name, next, previous, up +@section Configuring Elephant -in order for clsql to find the PostGres library libpq.so, for example. +Before you can load the elephant packages into your running lisp, you +need to setup the configuration file. First, copy the reference file +config.sexp from the root directory to my-config.sexp. my-config.sexp +contains a lisp reader-formatted list of key-value pairs that tells +elephant where to find various libraries, how to build libraries, etc. + +For example: -Without modifcation, Elephant uses this as it's lib path: @lisp -/usr/local/share/common-lisp/elephant-0.3/ -@end lisp +#+(and (or sbcl allegro) macosx) +((:berkeley-db-include-dir . "/opt/local/include/db45/") + (:berkeley-db-lib-dir . "/opt/local/lib/db45/") + (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") + (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") + (:pthread-lib . nil) + (:clsql-lib . nil) + (:compiler . :gcc)) +@end lisp + +The following is a guide to the various parameters. For simplicity, +we include all the parameters here, although we will go into more +detail in each of the data store sections. + +@itemize +@item @strong{:compiler} -- This tells Elephant which compiler to use to build any C libraries. The only options currently are :gcc on Unix platforms and :cygwin for the Windows platform. +@item @strong{:berkeley-db-include-dir} -- The pathname for the Berkeley DB include files (db.h) +@item @strong{:berkeley-db-lib-dir} -- The pathname for all the Berkeley DB library files +@item @strong{:berkeley-db-lib} -- The full pathname for the specific Berkeley DB library (libdb45.so) +@item @strong{:berkeley-db-deadlock} -- The full pathname to the BDB utility function db_deadlock +@item @strong{:pthread-lib} -- Not needed for SBCL 9.17+ +@item @strong{:clsql-lib} -- Currently unused, adds paths to the CL-SQL library search function +@end itemize + +The config.sexp file contains a set of example configurations to start +from, but you will most likely need to modify it for your system. + +Elephant has one small C library that it uses for binary serialization +which means that you need to have gcc in your path (@pxref{Elephant on +Windows} for exceptions on the Windows platform). + +@node Loading Elephant +@comment node-name, next, previous, up +@section Loading Elephant -So you could put a symbolic link to libpq.so there, where libmemutil.so and -libsleepycat.so will also reside. +@subsection Loading Elephant via ASDF -Elephant is designed to allow multi-repository operation; -so you could concievably use two or more repositories at the -same time. More particularly, you can seamlessly migrate your -data from one repository to a different one at a later date. -In a long duration project, this might occur because of a licensing -or performance issue with a particular respository. Migrating to -a new repository of the same type is a cheap form of GC although -migration is limited to the total size of main memory to store -a hash table that tracks all copied object ID's. - -@node Test-Suites -@comment node-name, next, previous, up -@section Test-Suites - -Elephant is moderately mature. Hopefully, it will work out-of-the-box -for you. - -However, if you are using an LISP implementation different than the ones -on which it is developed and maintained (currently OpenMCL, SBCL, and ACL), -or as the repositories evolve, or just because of mistakes, you may need -to run the test suites. If you report a bug, we will ask you -to run these tests and report the output. Running them when you -first install things may give you a sense of confidence and understanding -that makes it worth the trouble. +Now that you have loaded all the dependencies and created your +configuration file you can load the Elephant packages and +definitions: -There are three files that execute the tests. You should choose -one as a starting point based on what backend(s) you are using. -If using BerekleyDB, use @lisp -BerkeleyDB-tests.lisp +(asdf:operate 'asdf:load-op :elephant) @end lisp -If using both, use both of the above and also use: +This will load the cl-base64 and uffi libraries. It will also +automatically compile and load the C library. The build process no +longer depends on a Makefile. This build process has been verified on +most platforms, but if you have a problem please report it, and any +output you can capture, to the developers at +@email{elephant-devel@@common-lisp.net}. We will update the FAQ at +@uref{http://trac.common-lisp.net/elephant} with common problems users +run into. + +@subsection Two-Phase Load Process + +Elephant uses a two-phase load process. The core code is loaded and +the code for a given data store is loaded on demand when you call +@code{open-store} with a specification referencing that data store. +The second phase of the load process requires ASDF to be installed on +your system. + +(NOTE: There are some good reasons and not so good reasons for this +process. One reason you cannot load ele-bdb.asd directly as it +depends on lisp code defined in elephant.asd. We decided not to fix +this in this release although later releases may avoid the oddity of +the two phase loading) + +@subsection Packages + +Now that Elephant has been loaded, you can call @code{use-package} in +the cl-user package or create a new package that imports the symbols +exported from package :elephant. + @lisp -MigrationTests.lisp +CL-USER> (use-package :elephant) +=> T + +OR + +(defpackage :elephant-user + (:use :common-lisp :elephant)) @end lisp -The text of this file is included here to give the -casual reader an idea of how elepant test can be run in general: -@lisp -;; This file is an example of how to perform the -;; migration tests. You will have to modify it -;; slightly depending on the systems that want to test... -;; You can test migration even between two BDB respositories if you wish -(asdf:operate 'asdf:load-op :elephant) -(asdf:operate 'asdf:load-op :ele-clsql) -(asdf:operate 'asdf:load-op :clsql-postgresql-socket) -(asdf:operate 'asdf:load-op :ele-bdb) -(asdf:operate 'asdf:load-op :elephant-tests) -;; For sqlite-3.. -;; (asdf:operate 'asdf:load-op :ele-sqlite3) +Beginners can skip to the end of this section. -(in-package "ELEPHANT-TESTS") +Elephant has a common package called elephant that exports a set of +generic functions. It also contains a dispatcher based on the first +element of a specification list that calls the relevant backend +version of @code{open-controller}, the internal method that creates a +@code{store-controller}. Each backend has it's own subclass +implementing the abstract interface of @code{store-controller}. -;; The primary and secondary test-paths are -;; use for the migration tests. +@subsection Opening a Store -;; This this configuration for testing between BDB and SQL.... -(setq *test-path-primary* *testpg-path*) -;; (setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path*) +As discussed in the tutoral, you can now open a store to begin using +Elephant: -;; This this configuration for testing from one BDB repository to another... -(setq *test-path-primary* *testdb-path*) -;; (setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path2*) +@lisp +(open-store '(:BDB "/Users/owner/db/my-bdb/")) +... +ASDF loading messages +... +=> #<BDB-STORE-CONTROLLER> -(do-migrate-test-spec *test-path-primary*) +(open-store '(:CLSQL (:POSTGRESQL "localhost.localdomain" "mydb" "myuser" "")))) +... +ASDF loading messages +... +=> #<SQL-STORE-CONTROLLER> @end lisp -The appropriate test should execute for you with no errors. -If you get errors, you may wish to report it the -@code{ elephant-devel at common-lisp.net} email list. +The first time you load a specific data store, Elephant will call ASDF +to load all the specified data store's dependencies, connect to a +database and return the @code{store-controller} subclass instance for +that backend. -@node Berkeley DB Repository +@node Berkeley DB Data Store @comment node-name, next, previous, up -@section Berkeley DB Repository +@section Berkeley DB Data Store + -@node SQL Repository +@node Berkeley DB Example +@comment node-name, next, previous, up +@section Setting up Berkeley DB + +@node CL-SQL Data Store @comment node-name, next, previous, up -@section SQL Repository +@section CL-SQL Data Store Although originally designed as an interface to the BerkeleyDB system, the original Elephant system has been experimenetally extended to @@ -227,44 +302,14 @@ the multi-repository version somewhat complicates the underlying persistent object management. -@node Multi-repository Operation +@node PostGres Examples @comment node-name, next, previous, up -@section Multi-repository Operation - -Elephant now keeps a small hashtables that maps ``database specifications'' into -actual database connections. - -If a database spec is a string, it is assumed to be a BerkeleyDB path. -If it is a list, it is a assumed to be a CL-SQL connection specification. -For example: -@lisp -ELE-TESTS> *testdb-path* -"/home/read/projects/elephant/elephant/tests/testdb/" -ELE-TESTS> *testpg-path* -(:postgresql "localhost.localdomain" "test" "postgres" "") -ELE-TESTS> -@end lisp - -The tests now have a function @code{do-all-tests-spec} that take a spec and -based on its type attempt to open the correct kind of store controller and -perform the tests. - -The routine @code{get-controller} takes this specifiation. - -The basic strategy is that the ``database specification'' object is stored in -every persistent object and collection so that the repository can be found. - -In this way, objects that reside in different repositories can coexist within -the LISP object space, allowing data migration. - - - +@section Setting up PostGres @node Setting up PostGres @comment node-name, next, previous, up @section Setting up PostGres - To set up a PostGres based back end, you should: @enumerate @@ -294,7 +339,6 @@ Before you attempt to connect with Elephant. [114 lines skipped] --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 13:55:15 1.4 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/30 14:34:34 1.5 @@ -16,11 +16,14 @@ (load docstrings-path) (defun make-docs () - (when t - (elephant:open-store elephant-tests::*testbdb-spec*) - (make-instance 'elephant::persistent-collection) - (make-instance 'elephant::secondary-cursor) - (make-instance 'elephant::indexed-btree) - (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" (find-package :elephant) (find-package :elephant-backend) (find-package 'elephant-memutil) (find-package 'elephant-system)))) + (elephant:open-store elephant-tests::*testbdb-spec*) + (make-instance 'elephant::persistent-collection) + (make-instance 'elephant::secondary-cursor) + (make-instance 'elephant::indexed-btree) + (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" + (find-package :elephant) + (find-package :elephant-backend) + (find-package :elephant-memutil) + (find-package :elephant-system))) (make-docs) --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/24 12:16:02 1.6 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 14:34:34 1.7 @@ -14,7 +14,6 @@ * Collections:: BTrees and indices. * Cursors:: Traversing BTrees. * Transactions:: Transactions. -* Multithreading:: Multithreading. * Migration and Upgrading:: Migration and upgrading. @end menu @@ -56,23 +55,48 @@ @section Persistent Object Indexing @cindex Persistent Object Indexing -@include includes/fun-get-instances-by-class.texinfo -@include includes/fun-get-instance-by-value.texinfo -@include includes/fun-get-instances-by-value.texinfo -@include includes/fun-get-instances-by-range.texinfo +@subsection Indexed Object Accessors + +@include includes/fun-elephant-map-class.texinfo +@include includes/fun-elephant-map-class-index.texinfo + +@include includes/fun-elephant-get-instances-by-class.texinfo +@include includes/fun-elephant-get-instance-by-value.texinfo +@include includes/fun-elephant-get-instances-by-value.texinfo +@include includes/fun-elephant-get-instances-by-range.texinfo + +@include includes/fun-elephant-drop-instances.texinfo + +@subsection Direct Class Index Manipulation + +@include includes/fun-elephant-find-class-index.texinfo +@include includes/fun-elephant-find-inverted-index.texinfo +@include includes/fun-elephant-make-class-cursor.texinfo +@include includes/macro-elephant-with-class-cursor.texinfo +@include includes/fun-elephant-make-inverted-cursor.texinfo +@include includes/macro-elephant-with-inverted-cursor.texinfo + +@subsection Dynamic Indexing API @include includes/fun-elephant-enable-class-indexing.texinfo @include includes/fun-elephant-disable-class-indexing.texinfo -@include includes/fun-add-class-slot-index.texinfo -@include includes/fun-remove-class-slot-index.texinfo -@include includes/fun-add-class-derived-index.texinfo -@include includes/fun-remove-class-derived-index.texinfo +@include includes/fun-elephant-add-class-slot-index.texinfo +@include includes/fun-elephant-remove-class-slot-index.texinfo +@include includes/fun-elephant-add-class-derived-index.texinfo +@include includes/fun-elephant-remove-class-derived-index.texinfo @node Query Interfaces @comment node-name, next, previous, up @section Query Interfaces @cindex Query Interfaces +Query interfaces are currently unimplemented. An example query +interface is provided for reference only, a new system is under +development for the 0.7 release. + +@include includes/fun-elephant-get-query-results.texinfo +@include includes/fun-elephant-map-class-query.texinfo + @node Collections @comment node-name, next, previous, up @section Collections @@ -145,12 +169,9 @@ @include includes/fun-elephant-commit-transaction.texinfo @include includes/fun-elephant-abort-transaction.texinfo -@node Multithreading -@comment node-name, next, previous, up -@section Multithreading -@cindex Multithreading - @node Migration and Upgrading @comment node-name, next, previous, up @section Migration and Upgrading @cindex Migration and Upgrading + +@include includes/fun-elephant-migrate.texinfo --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/26 03:37:27 1.9 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/30 14:34:34 1.10 @@ -914,8 +914,8 @@ complicated. The best strategy at the beginning is a conservative one, break things up into the smallest logical sets of primitive operations and only wrap higher level functions in transactions when -they absolutely have to commit together. @xref{Transaction details} -for all the gory detail of transactions and @pxref{Usage scenarios} -for more examples of how systems can be designed using transactions. +they absolutely have to commit together. See @ref{Transaction details} +for the full details and @pxref{Usage scenarios} for more examples of +how systems can be designed and tuned using transactions. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/26 03:37:27 1.3 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/30 14:34:34 1.4 @@ -16,6 +16,7 @@ * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. * Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls. +* Multi-repository Operation:: Specifying repositories. * Repository Migration and Upgrade:: How to move objects from one repository to another. * Garbage collection:: How to recover storage and OIDs in long-lived repositories. * Performance tuning:: How to get the most from Elephant. @@ -82,10 +83,9 @@ @code{with-open-controller} macro. Opening and closing a controller is very expensive. - -@node{Class indices} +@node Class Indices @comment node-name, next, previous, up -@section Class indicies +@section Class Indicies You can enable/disable class indexing for an entire class. When you disable indexing all references to instances of that class are lost. If you re-enable @@ -130,7 +130,7 @@ somewhat user customizable; documentation for this exists in the source file referenced above. -@node{Using BTrees} +@node Using BTrees @comment node-name, next, previous, up @section Using BTrees @@ -225,7 +225,7 @@ to the target repository which you can then overwrite. To avoid the default persistent slot copying, bind the dynamic variable @code{*inhibit-slot-writes*} in your user method using -@code(with-inhibited-slot-copy () ...)} a convenience macro. +@code{with-inhibited-slot-copy} a convenience macro. @node Threading @@ -318,3 +318,33 @@ ensure that multiple threads do not interleave access so single user mode is not suitable for use in web servers or other typically multi-threaded applications. + +@node Multi-repository Operation +@comment node-name, next, previous, up +@section Multi-repository Operation + +Elephant now keeps a small hashtables that maps ``database specifications'' into +actual database connections. + +If a database spec is a string, it is assumed to be a BerkeleyDB path. +If it is a list, it is a assumed to be a CL-SQL connection specification. +For example: +@lisp +ELE-TESTS> *testdb-path* +"/home/read/projects/elephant/elephant/tests/testdb/" +ELE-TESTS> *testpg-path* +(:postgresql "localhost.localdomain" "test" "postgres" "") +ELE-TESTS> +@end lisp + +The tests now have a function @code{do-all-tests-spec} that take a spec and +based on its type attempt to open the correct kind of store controller and +perform the tests. + +The routine @code{get-controller} takes this specifiation. + +The basic strategy is that the ``database specification'' object is stored in +every persistent object and collection so that the repository can be found. + +In this way, objects that reside in different repositories can coexist within +the LISP object space, allowing data migration.

1 0

[elephant-cvs] CVS elephant/src/db-bdb
by ieslick 30 Mar '07

30 Mar '07

Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv31032/src/db-bdb Modified Files: bdb-controller.lisp Log Message: Significant documentation string and documentation edits towards 0.6.1 manual. Clean up packages so elephant exports user visible symbols and backend exports backend-relevant symbols. Change required fix in serializer packages also. Added :elephant-user package. --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp 2007/03/16 14:44:44 1.31 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp 2007/03/30 14:34:35 1.32 @@ -135,7 +135,7 @@ (db-open db :file "%ELEPHANTOID" :database "%ELEPHANTOID" :auto-commit t :type DB-BTREE :create t :thread thread) (let ((oid-seq (db-sequence-create db))) - (db-sequence-set-cachesize oid-seq *cachesize*) + (db-sequence-set-cachesize oid-seq 100) (db-sequence-set-flags oid-seq :seq-inc t :seq-wrap t) (db-sequence-set-range oid-seq 0 most-positive-fixnum) (db-sequence-initial-value oid-seq 0)

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 30 Mar '07

30 Mar '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv31032/src/elephant Modified Files: backend.lisp classindex.lisp controller.lisp package.lisp query.lisp serializer.lisp serializer1.lisp serializer2.lisp variables.lisp Log Message: Significant documentation string and documentation edits towards 0.6.1 manual. Clean up packages so elephant exports user visible symbols and backend exports backend-relevant symbols. Change required fix in serializer packages also. Added :elephant-user package. --- /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/02/16 23:02:53 1.11 +++ /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 14:34:35 1.12 @@ -23,108 +23,33 @@ (:documentation "Backends should use this to get access to internal symbols of elephant that importers of elephant shouldn't see. Backends should also import elephant to get use-api generic function symbols, classes and globals") + (:use #:elephant) (:import-from #:elephant ;; Variables - #:*cachesize* - #:*dbconnection-spec* ;; shouldn't need this + #:*dbconnection-spec* #:connection-is-indeed-open - ;; Persistent objects - #:oid #:get-con - #:next-oid - #:persistent-slot-writer - #:persistent-slot-reader - #:persistent-slot-boundp - #:persistent-slot-makunbound - ;; Controllers - #:*elephant-code-version* - #:store-controller - #:open-controller - #:database-version - #:close-controller - #:controller-serialize - #:controller-deserialize - #:controller-spec - #:controller-root - #:controller-version - #:controller-class-root - #:root #:spec #:class-root - #:flush-instance-cache - #:controller-symbol-cache #:controller-symbol-id-cache - ;; Collection generic functions - #:btree #:btree-index #:indexed-btree - #:build-indexed-btree #:build-btree #:existsp - #:map-indices - ;; Serialization - #:deserialize #:serialize - #:deserialize-from-base64-string - #:serialize-to-base64-string - ;; Serialization callbacks - #:lookup-persistent-symbol - #:lookup-persistent-symbol-id - ;; Cursor accessors - #:cursor - #:cursor-btree - #:cursor-oid - #:cursor-initialized-p - ;; Transactions - #:*current-transaction* - #:make-transaction-record - #:transaction-store - #:transaction-object - #:execute-transaction - #:controller-start-transaction - #:controller-commit-transaction - #:controller-abort-transaction - ;; Misc - #:slot-definition-name - #:remove-indexed-element-and-adjust - #:register-backend-con-init - #:lookup-backend-con-init - ;; Lisp specific - #+(or sbcl cmu) #:%bignum-ref - ) - (:export - ;; Variables - #:*cachesize* - #:*dbconnection-spec* ;; shouldn't need this - #:connection-is-indeed-open ;; Persistent objects #:oid #:get-con - #:next-oid + #:next-oid #:persistent-slot-writer #:persistent-slot-reader #:persistent-slot-boundp #:persistent-slot-makunbound + ;; Controllers #:*elephant-code-version* - #:store-controller #:open-controller - #:database-version #:close-controller + #:database-version + #:controller-spec #:controller-serialize #:controller-deserialize - #:controller-spec - #:controller-root - #:controller-class-root - #:controller-version #:root #:spec #:class-root - #:flush-instance-cache - #:controller-symbol-cache #:controller-symbol-id-cache - ;; Collection generic functions - #:btree #:btree-index #:indexed-btree - #:build-indexed-btree #:build-btree #:existsp - #:map-indices ;; Serialization - #:deserialize #:serialize - #:serialize-symbol-complete #:deserialize-from-base64-string #:serialize-to-base64-string - ;; Serialization callbacks - #:lookup-persistent-symbol - #:lookup-persistent-symbol-id ;; Cursor accessors - #:cursor #:cursor-btree #:cursor-oid #:cursor-initialized-p @@ -133,14 +58,58 @@ #:make-transaction-record #:transaction-store #:transaction-object - #:execute-transaction - #:controller-start-transaction - #:controller-commit-transaction - #:controller-abort-transaction + ;; Registration + #:register-backend-con-init + #:lookup-backend-con-init ;; Misc #:slot-definition-name + #:slots-and-values + #:struct-slots-and-values #:remove-indexed-element-and-adjust - #:register-backend-con-init - #:lookup-backend-con-init - )) + ) + (:export + ;; Variables + #:*cachesize* + #:*dbconnection-spec* + #:connection-is-indeed-open + + ;; Persistent objects + #:oid #:get-con + #:next-oid + #:persistent-slot-writer + #:persistent-slot-reader + #:persistent-slot-boundp + #:persistent-slot-makunbound + + ;; Controllers + #:*elephant-code-version* + #:open-controller + #:close-controller + #:database-version + #:controller-spec + #:controller-version + #:controller-serialize + #:controller-deserialize + #:root #:spec #:class-root + ;; Serialization + #:deserialize-from-base64-string + #:serialize-to-base64-string + ;; Cursor accessors + #:cursor-btree + #:cursor-oid + #:cursor-initialized-p + ;; Transactions + #:*current-transaction* + #:make-transaction-record + #:transaction-store + #:transaction-object + ;; Registration + #:register-backend-con-init + #:lookup-backend-con-init + ;; Misc + #:slot-definition-name + #:slots-and-values + #:struct-slots-and-values + #:remove-indexed-element-and-adjust + )) --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/03/25 14:57:49 1.33 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/03/30 14:34:35 1.34 @@ -141,7 +141,7 @@ (map-btree (lambda (class-name index) (declare (ignore index)) (let ((class (find-class class-name nil))) - (when class + (when (and class (subtypep class 'persistent-metaclass)) (setf (%index-cache class) nil)))) (controller-class-root sc))) (t (e) (warn "Unable to clear class index caches ~A" e))))) @@ -333,16 +333,22 @@ ;; USER CURSOR API ;; =================== -(defgeneric make-inverted-cursor (persistent-metaclass name) +(defgeneric make-inverted-cursor (class name) (:documentation "Define a cursor on the inverted (slot or derived) index")) -(defgeneric make-class-cursor (persistent-metaclass) +(defgeneric make-class-cursor (class) (:documentation "Define a cursor over all class instances")) + (defmethod make-inverted-cursor ((class persistent-metaclass) name) (make-cursor (find-inverted-index class name))) +(defmethod make-inverted-cursor ((class symbol) name) + (make-cursor (find-inverted-index class name))) + (defmacro with-inverted-cursor ((var class name) &body body) + "Bind the var argument to an inverted cursor on the index + specified the provided class and index name" `(let ((,var (make-inverted-cursor ,class ,name))) (unwind-protect (progn ,@body) (cursor-close ,var)))) @@ -350,7 +356,12 @@ (defmethod make-class-cursor ((class persistent-metaclass)) (make-cursor (find-class-index class))) +(defmethod make-class-cursor ((class symbol)) + (make-cursor (find-class-index class))) + (defmacro with-class-cursor ((var class) &body body) + "Bind the var argument in the body to a class cursor on the + index specified the provided class or class name" `(let ((,var (make-class-cursor ,class))) (unwind-protect (progn ,@body) (cursor-close ,var)))) @@ -361,8 +372,8 @@ ;; ====================== (defun map-class (fn class) - "Perform a map operation across all instances of class. Takes a - function of one argument, the class instance" + "Perform a map operation over all instances of class. Takes a + function of one argument, a class instance" (let* ((class (if (symbolp class) (find-class class) class)) @@ -374,9 +385,22 @@ (map-btree #'map-fn class-idx)))) (defun map-class-index (fn class index &rest args &key start end value) - "To map over a subset of instances, pick an index by slot name - or derived index name and specify the bounds for the traversal. - Otherwise use map-class for all instances. " + "This function maps over a subset of class instances in the + order defined by the index. Specify the class and index by + quoted name. The index may be a slot index or a derived + index. + + To map only a subset of key-value pairs, specify the range + using the :start and :end keywords; all elements greater than + or equal to :start and less than or equal to :end will be + traversed regardless of whether the start or end value is in + the index. + + Use nil in the place of start or end to specify the first + element or last element, respectively. + + To map a single value, iff it exists, use the :value keyword. + This is the only way to travers all nil values." (declare (dynamic-extent args) (ignorable args)) (let* ((index (if (symbolp index) @@ -395,11 +419,14 @@ (defgeneric get-instances-by-class (persistent-metaclass) (:documentation "Retrieve all instances from the class index as a list of objects")) + (defgeneric get-instance-by-value (persistent-metaclass slot-name value) (:documentation "Retrieve instances from a slot index by value. Will return only the first instance if there are duplicates.")) + (defgeneric get-instances-by-value (persistent-metaclass slot-name value) (:documentation "Returns a list of all instances where the slot value is equal to value.")) + (defgeneric get-instances-by-range (persistent-metaclass slot-name start end) (:documentation "Returns a list of all instances that match values between start and end. An argument of @@ -458,6 +485,8 @@ (nreverse instances))) (defun drop-instances (instances &key (sc *store-controller*)) + "Removes a list of persistent objects from all class indices + and unbinds any slot values" (when instances (assert (consp instances)) (do-subsets (subset 500 instances) --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/24 03:03:00 1.41 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 14:34:35 1.42 @@ -23,16 +23,23 @@ ;; TRACKING OBJECT STORES ;; -(defparameter *elephant-backends* +(defvar *elephant-backends* '((:bdb (:ele-bdb)) (:clsql (:ele-clsql)) ) - "Entries have the form of (backend-type asdf-depends-list") + "Tells the main elephant code the tag used in a store spec to + refer to a given backend. The second argument is an asdf + dependency list. Entries have the form of (backend-type + asdf-depends-list") (defvar *elephant-controller-init* (make-hash-table)) (defun register-backend-con-init (name controller-init-fn) - "Backends call this during evalution to register their init function's name" + "Backends must call this function during the + loading/compilation process to register their initialization + function for the tag name in *elephant-backends*. The + initialization function returns a fresh instance of the + backends store-controller subclass" (setf (gethash name *elephant-controller-init*) controller-init-fn)) (defun lookup-backend-con-init (name) @@ -102,6 +109,9 @@ ;; (defun get-user-configuration-parameter (name) + "This function pulls a value from the key-value pairs stored in + my-config.sexp so backends can have their own pairs for appropriate + customization after loading." (elephant-system::get-config-option name (asdf:find-system :elephant))) @@ -114,26 +124,51 @@ ((spec :type list :accessor controller-spec :initarg :spec - :documentation "Backend create functions should pass in :spec during make-instance") + :documentation "Backend initialization functions are + expected to initialize :spec on the call to + make-instance") ;; Generic support for the object, indexing and root protocols (root :reader controller-root - :documentation "This should be a persistent btree instantiated by the backend") + :documentation "This is an instance of the backend + persistent btree. It should have an OID that is fixed in + the code and does not change between sessions. Usually + it this is something like 0, 1 or -1") (class-root :reader controller-class-root - :documentation "This should be a persistent indexed btree instantiated by the backend") + :documentation + "This is another root for class indexing that is + also a backend specific persistent btree instance + with a unique OID that persists between sessions.") (instance-cache :accessor instance-cache :initform (make-cache-table :test 'eql) - :documentation "This is an instance cache and part of the metaclass - protocol. Backends should not override") + :documentation + "This is an instance cache and part of the + metaclass protocol. Backends should not + override the default behavior.") (instance-cache-lock :accessor instance-cache-lock :initform (ele-make-lock) - :documentation "Protection for updates to the cache from multiple threads") + :documentation "Protection for updates to + the cache from multiple threads. Do not + override.") ;; Upgradable serializer strategy - (serializer-version :accessor controller-serializer-version :initform nil) - (serialize :accessor controller-serialize :initform nil) - (deserialize :accessor controller-deserialize :initform nil) - ) + (serializer-version :accessor controller-serializer-version :initform nil + :documentation "Governs the default + behavior regarding which serializer + version the current elephant core is + using. Backends can override by creating + a method on initialize-serializer.") + (serialize :accessor controller-serialize :initform nil + :documentation "Accessed by elephant::serialize to + get the entry point to the default serializer or to + a backend-specific serializer") + (deserialize :accessor controller-deserialize :initform nil + :documentation "Contains the entry point for the + specific serializer to be called by + elephant::deserialize")) (:documentation - "Class of objects responsible for the book-keeping of holding DB - handles, the cache, table creation, counters, locks, the root - (for garbage collection,) et cetera.")) + "Superclass for the data store controller, the main interface + to any book-keeping, references to DB handles, the instance + cache, btree table creation, counters, locks, the roots (for + garbage collection,) et cetera. Behavior is shared between + the superclass and subclasses. See slot documentation for + details.")) ;; ;; Per-controller instance caching @@ -324,12 +359,22 @@ (defgeneric open-controller (sc &key recover recover-fatal thread &allow-other-keys) (:documentation "Opens the underlying environment and all the necessary -database tables.")) +database tables. Different backends may use different keys so +all methods should &allow-other-keys. There are three standard +keywords: :recover, :recover-fatal and :thread. Recover means +that recovery should be checked for or performed on startup. +Recover fatal means a full rebuild from log files is requested. +Thread merely indicates to the backend that it is a threaded +application and any steps that need to be taken (for example +transaction implementation) are taken. :thread is usually +true.")) (defgeneric close-controller (sc) (:documentation - "Close the db handles and environment. Tries to wipe out -references to the db handles.")) + "Close the db handles and environment. Should be in a state + where lisp could be shut down without causing an inconsistent + state in the db. Also, the object could be used by + open-controller to reopen the database")) (defmethod close-controller :after ((sc store-controller)) "Delete connection spec so store-controller operations on cached @@ -422,42 +467,53 @@ (defun add-to-root (key value &key (sc *store-controller*)) "Add an arbitrary persistent thing to the root, so you can -retrieve it in a later session. N.B. this means it (and -everything it points to) won't get gc'd." + retrieve it in a later session. Anything referenced by an + object added to the root is considered reachable and thus live" (declare (type store-controller store-controller)) (assert (not (eq key *elephant-properties-label*))) (setf (get-value key (controller-root sc)) value)) (defun get-from-root (key &key (sc *store-controller*)) - "Get a something from the root." + "Get the value associated with key from the root. Returns two + values, the value, or nil, and a boolean indicating whether a + value was found or not (so you know if nil is a value or an + indication of non-presence)" (declare (type store-controller sc)) (get-value key (controller-root sc))) (defun root-existsp (key &key (sc *store-controller*)) - "Test whether a key exists in the root" + "Test whether a given key is instantiated in the root" (declare (type store-controller sc)) (if (existsp key (controller-root sc)) t nil)) (defun remove-from-root (key &key (sc *store-controller*)) - "Remove something from the root." + "Remove something from the root by the key value" (declare (type store-controller sc)) (remove-kv key (controller-root sc))) (defun map-root (fn &key (sc *store-controller*)) - "Map over all key-value pairs in the root" + "Takes a function of two arguments, key and value, to map over + all key-value pairs in the root" (map-btree fn (controller-root sc))) ;; ;; Explicit storage reclamation ;; +(defgeneric drop-pobject (persistent-object) + (:documentation "drop-pobject reclaims persistent object storage by unbinding + all persistent slot values. It can also helps catch errors + where an object should be unreachable, but a reference still + exists elsewhere in the DB. On access, the unbound slots + should flag an error in the application program. IMPORTANT: + this function does not clear the cached object instance or any + serialized references still in the db. Need a migration or GC + for that! drop-instances is preferred as it implements the proper + behavior for indexed classes")) + (defmethod drop-pobject ((inst persistent-object)) - "Reclaim persistent object storage by unbinding slot values. - This does not delete the cached object instance or any - serialized references still in the db. - Need a migration or GC for that!" (let ((pslots (persistent-slots (class-of inst)))) (dolist (slot pslots) (slot-makunbound inst slot)))) --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/03/23 16:08:10 1.25 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/03/30 14:34:35 1.26 @@ -25,107 +25,6 @@ (:documentation "Elephant: an object-oriented database for Common Lisp with multiple backends for Berkeley DB, SQL and others.") - (:export #:*store-controller* #:*current-transaction* - #:*elephant-lib-path* #:*elephant-code-version* - #:with-elephant-variables - - #:store-controller #:controller-root #:controller-class-root - #:open-store #:close-store #:with-open-store - #:add-to-root #:get-from-root #:remove-from-root #:root-existsp - #:map-root #:get-cached-instance #:flush-instance-cache - #:controller-symbol-cache #:controller-symbol-id-cache - #:controller-fast-symbols-p - #:optimize-layout #:drop-pobject - #:get-user-configuration-parameter - #:database-version - - #:upgrade - - #:controller-version #:controller-serializer-version - #:controller-serialize #:controller-deserialize - #:serialize-database-version-key - #:serialize-database-version-value - #:deserialize-database-version-value - #:serialize-database-serializer-version-value - #:deserialize-database-serializer-version-value - #:initialize-serializer - - #:with-transaction #:ensure-transaction - #:start-ele-transaction #:commit-transaction #:abort-transaction - - #:persistent #:persistent-object #:persistent-metaclass - #:persistent-collection #:defpclass - - #:btree #:make-btree #:get-value #:remove-kv #:existp - #:indexed-btree #:make-indexed-btree - #:add-index #:get-index #:remove-index #:map-indices - #:btree-index #:get-primary-key - #:primary #:key-form #:key-fn - - #:struct-constructor - - #:migrate #:set-oid-spec #:*inhibit-slot-copy* - #:add-symbol-conversion #:add-package-conversion - #:*always-convert* - - #:translate-and-intern-symbol - #:lookup-persistent-symbol - #:lookup-persistent-symbol-id - #:int-byte-spec - - #:cursor #:secondary-cursor #:make-cursor #:make-simple-cursor - #:cursor-close #:cursor-init - #:cursor-duplicate #:cursor-current #:cursor-first - #:cursor-last #:cursor-next #:cursor-next-dup - #:cursor-next-nodup #:cursor-prev #:cursor-prev-nodup - #:cursor-set #:cursor-set-range #:cursor-get-both - #:cursor-get-both-range #:cursor-delete #:cursor-put - #:cursor-pcurrent #:cursor-pfirst #:cursor-plast - #:cursor-pnext #:cursor-pnext-dup #:cursor-pnext-nodup - #:cursor-pprev #:cursor-pprev-nodup #:cursor-pset - #:cursor-pset-range #:cursor-pget-both - #:cursor-pget-both-range - - ;; Class indexing management API - #:*default-indexed-class-synch-policy* - #:find-class-index #:find-inverted-index - #:enable-class-indexing #:disable-class-indexing - #:add-class-slot-index #:remove-class-slot-index - #:add-class-derived-index #:remove-class-derived-index - #:describe-db-class-index - #:report-indexed-classes - #:class-indexedp-by-name - - ;; Low level cursor API - #:make-inverted-cursor #:make-class-cursor - #:with-inverted-cursor #:with-class-cursor - - ;; Primitive mapping API - #:with-btree-cursor - #:map-btree - #:map-index - - ;; BTREE Utilities - #:empty-btree-p - #:dump-btree - #:btree-keys - #:btree-differ-p - - ;; Class mapping API - #:map-class - #:map-class-index - - ;; Instance query API - #:get-instances-by-class - #:get-instance-by-value - #:get-instances-by-value - #:get-instances-by-range - #:drop-instances - - ;; Utilities - #:slots-and-values - #:struct-slots-and-values - ) #+cmu (:import-from :pcl compute-class-precedence-list @@ -296,7 +195,71 @@ slot-definition-allocation slot-definition-initargs compute-slots) + (:export + #:*store-controller* + #:store-controller #:controller-root #:controller-class-root + #:open-store #:close-store #:with-open-store + #:add-to-root #:get-from-root #:remove-from-root #:root-existsp #:map-root + #:flush-instance-cache + #:optimize-layout + + #:persistent #:persistent-object #:persistent-metaclass #:defpclass + #:persistent-collection #:drop-pobject + + #:btree #:make-btree + #:get-value #:remove-kv #:existp + #:indexed-btree #:make-indexed-btree + #:add-index #:get-index #:remove-index #:map-indices + #:get-primary-key #:primary #:key-form #:key-fn + #:with-btree-cursor #:map-btree #:map-index + #:empty-btree-p #:dump-btree #:btree-keys #:btree-differ-p + + #:cursor #:secondary-cursor #:make-cursor #:make-simple-cursor + #:cursor-close #:cursor-init + #:cursor-duplicate #:cursor-current #:cursor-first + #:cursor-last #:cursor-next #:cursor-next-dup + #:cursor-next-nodup #:cursor-prev #:cursor-prev-nodup + #:cursor-set #:cursor-set-range #:cursor-get-both + #:cursor-get-both-range #:cursor-delete #:cursor-put + #:cursor-pcurrent #:cursor-pfirst #:cursor-plast + #:cursor-pnext #:cursor-pnext-dup #:cursor-pnext-nodup + #:cursor-pprev #:cursor-pprev-nodup #:cursor-pset + #:cursor-pset-range #:cursor-pget-both + #:cursor-pget-both-range + + #:find-class-index #:find-inverted-index + #:enable-class-indexing #:disable-class-indexing + #:add-class-slot-index #:remove-class-slot-index + #:add-class-derived-index #:remove-class-derived-index + #:describe-db-class-index + #:report-indexed-classes + #:class-indexedp-by-name + + #:map-class #:map-class-index + #:get-instances-by-class + #:get-instance-by-value + #:get-instances-by-value + #:get-instances-by-range + #:drop-instances + #:make-inverted-cursor #:make-class-cursor + #:with-inverted-cursor #:with-class-cursor + #:*default-indexed-class-synch-policy* + + #:with-transaction #:ensure-transaction + #:controller-start-transaction + #:controller-abort-transaction + #:controller-commit-transaction + + #:upgrade #:migrate + #:set-oid-spec #:*inhibit-slot-copy* + #:add-symbol-conversion #:add-package-conversion + #:*always-convert* + #:translate-and-intern-symbol + #:lookup-persistent-symbol + #:lookup-persistent-symbol-id + #:struct-constructor + ) ) (in-package "ELE") @@ -304,3 +267,10 @@ #+cmu (eval-when (:compile-toplevel) (proclaim '(optimize (ext:inhibit-warnings 3)))) + +(defpackage :elephant-user + (:use :common-lisp :elephant) + (:nicknames :ele-user) + (:documentation + "A user package for experimenting with Elephant")) + \ No newline at end of file --- /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/03/01 02:45:45 1.1 +++ /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/03/30 14:34:35 1.2 @@ -40,6 +40,7 @@ (number (funcall (relation-number-function rel) ival (first tvals))))) (defun get-query-instances (constraints) + "Get a list of instances according to the query constraints" (let ((list nil)) (flet ((collect (inst) (push inst list))) @@ -48,7 +49,9 @@ (defun map-class-query (fn constraints) "Map instances using the query constaints to filter objects, exploiting - slot indices (for last query) and stack allocated test closures" + slot indices (for last query) and stack allocated test closures. This is + a minimally optimizing version that uses the first index it finds, and + then does a nested loop join on the rest of the parameters." (assert (not (null constraints))) (destructuring-bind (class slot relation &rest values) (first constraints) (flet ((filter-by-relation (inst) --- /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/03/03 17:24:59 1.25 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/03/30 14:34:35 1.26 @@ -92,18 +92,21 @@ ;; Database Version (a list of integers = [version major minor]) (defun serialize-database-version-key (bs) + "Given a buffer-stream, encode a key indicating the version using + the constant +elephant-version+" (serialize-reserved-tag bs) (serialize-system-tag +elephant-version+ bs)) (defun serialize-database-version-value (version bs) - "Simple serializes a list containing three integers" - (assert (consp version)) + "Serializes a list containing three integers to the buffer stream bs" + (assert (and (= (length version) 3))) (destructuring-bind (version major minor) version (serialize-system-integer version bs) (serialize-system-integer major bs) (serialize-system-integer minor bs))) (defun deserialize-database-version-value (bs) + "Deserializes the 3 integer list from buffer stream bs" (let ((version (deserialize-system-integer bs)) (major (deserialize-system-integer bs)) (minor (deserialize-system-integer bs))) --- /project/elephant/cvsroot/elephant/src/elephant/serializer1.lisp 2007/02/26 19:12:18 1.12 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer1.lisp 2007/03/30 14:34:35 1.13 @@ -24,11 +24,12 @@ #+sbcl (:import-from :sb-bignum %bignum-ref) - (:import-from :elephant + (:import-from #:elephant get-cached-instance slot-definition-allocation slot-definition-name compute-slots + slots-and-values oid int-byte-spec array-type-from-byte --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/03/21 14:29:31 1.33 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/03/30 14:34:35 1.34 @@ -29,6 +29,8 @@ slot-definition-allocation slot-definition-name compute-slots + slots-and-values + struct-slots-and-values oid int-byte-spec array-type-from-byte --- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/02/02 23:51:58 1.10 +++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/03/30 14:34:35 1.11 @@ -38,9 +38,6 @@ ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;; Optimization parameters -(defvar *cachesize* 100 - "Size of the OID sequence cache.") - (defvar *circularity-initial-hash-size* 50 "This is the default size of the circularity cache used in the serializer")

1 0

[elephant-cvs] CVS elephant
by ieslick 30 Mar '07

30 Mar '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv31032 Modified Files: TODO elephant.asd Log Message: Significant documentation string and documentation edits towards 0.6.1 manual. Clean up packages so elephant exports user visible symbols and backend exports backend-relevant symbols. Change required fix in serializer packages also. Added :elephant-user package. --- /project/elephant/cvsroot/elephant/TODO 2007/03/24 12:16:02 1.75 +++ /project/elephant/cvsroot/elephant/TODO 2007/03/30 14:34:34 1.76 @@ -18,7 +18,7 @@ - Verify db_deadlock for other lisps (launch and kill background program I/F) Bugs: -- Fix any bugs found during BETA +- Support for asdf-install? Test coverage: - Clean up interface to tests @@ -47,6 +47,20 @@ 0.6.1 - Features COMPLETED to date ---------------------------------- +POST BETA CHECKINS: + +Bugs: +x Fix duplicate opening of CLSQL db bug that caused errors in SQLite +x Fix for persistent-object inheritance calculation when inheriting from standard classes +x Remove problematic asserts in lisp-compare-eq make equal for strings instead of eq +x Fix map over nils bug in map-index and get-instances-by-value; clarify map-index interface + +Tweaks: +x Add bounds to map-btree +x Change wipe-class-indexing so it can be called without the class object being created +x add-to-root, etc now uses :sc instead of :store-controller for brevity +x Remove hard coded paths in ele-clsql + POST ALPHA CHECKINS: Major Bugs: --- /project/elephant/cvsroot/elephant/elephant.asd 2007/03/19 20:35:30 1.38 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/03/30 14:34:34 1.39 @@ -294,11 +294,11 @@ (:file "transactions") (:file "metaclasses") (:file "classes") + (:file "cache") (:file "serializer") (:file "serializer1") ;; 0.6.0 db's (:file "serializer2") ;; 0.6.1 db's (:file "unicode2") - (:file "cache") (:file "controller") (:file "collections") (:file "classindex-utils") @@ -309,4 +309,3 @@ :depends-on (memutil utils))))) :serial t :depends-on (:uffi :cl-base64)) -

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 26 Mar '07

26 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv18053 Modified Files: tutorial.texinfo user-guide.texinfo Log Message: First cut new tutorial --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/25 11:04:38 1.8 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/26 03:37:27 1.9 @@ -355,16 +355,9 @@ Using the @code{persistent-metaclass} metaclass declares all slots to be persistent by default. To make a non-persistent slot use the -@code{:transient t} flag. Class slots are never persisted, for either -persistent or ordinary classes. (Someday, if we choose to store class -objects, this policy decision may change). - -Readers, writers, accessors, and @code{slot-value-using-class} are -instrumented, so override these with care. Because @code{slot-value, -slot-unboundp, slot-makunbound} are not generic functions, they are -not guaranteed to work properly with persistent slots. Use the -@code{*-using-class} versions or the @code{closer-to-mop} MOP compliance -layer by Pascal Costanza (we may integrate this in later versions). +@code{:transient t} flag. Class slots @code{:allocation :class} are +never persisted, for either persistent or ordinary classes. (Someday, +if we choose to store class objects, this policy may change). Persistent classes may inherit from other classes. Slots inherited from persistent classes remain persistent. Transient slots and slots @@ -372,47 +365,105 @@ cannot inherit from persistent classes -- otherwise persistent slots could not be stored! -Note that the database is read every time you access a slot. This is -a feature, not a bug, especially in concurrent situations: you want -the most recent commits, right? Note that this can be used as a weak -form of IPC. But also note that in particular, if your slot value is -not an immediate value, reading will cons or allocate the value. Gets -are not an expensive operation; you can perform thousands to tens of -thousands of primitive reads per second. However, if you're -concerned, cache large values in memory. +@lisp +(defclass stdclass1 () + ((slot1 :initarg :slot1 :accessor slot1))) + +(defclass stdclass2 (stdclass1) + ((slot2 :initarg :slot2 :accessor slot2))) + +(defpclass pclass1 (stdclass2) + ((slot1 :initarg :slot1 :accessor slot1) + (slot3 :initarg :slot3 :accessor slot3))) + +(make-instance 'pclass1 :slot1 1 :slot2 2 :slot3 3) +=> #<PCLASS1 @{x10deb88a@}> + +(add-to-root 'pinst *) +=> #<PCLASS1 @{x10deb88a@}> + +(slot1 pinst) +=> 1 + +(slot2 pinst) +=> 2 + +(slot3 pinst) +=> 3 +@end lisp + +Now we can simulate a new lisp session by flushing the instance cache, +reloading our object then see what slots remain. Here persistent +slot1 should shadow the standard slot1 and thus be persistent. Slot3 +is persistent by default and slot2, since it is inherited from a +standard class should be transient. + +@lisp +(elephant::flush-instance-cache *store-controller*) +=> #<EQL hash-table with weak values, 0 entries @{x11198a02@}> + +(setf pinst (get-from-root 'pinst)) +=> #<PCLASS1 @{x1119b652@}> + +(slot1 pinst) +=> 1 + +(slot-boundp pinst slot2 pinst) +=> nil + +(slot3 pinst) +=> 3 +@end lisp + +Using persistent objects has implications for the performance of your +system. Note that the database is read every time you access a slot. +This is a feature, not a bug, especially in concurrent situations: you +want the most recent commits by other threads, right? This can be +used as a weak form of IPC. But also note that in particular, if your +slot value is not an immediate value or persistent object, reading +will cons or freshly allocate storage for the value. + +Gets are not an expensive operation; you can perform thousands to tens +of thousands of primitive reads per second. However, if you're +concerned, cache large values in memory and avoid writing them back to +disk as long as you can. @node Persistent collections @comment node-name, next, previous, up @section Persistent collections -The remaining problem outlined in @ref{Serialization} is that -operations which mutate aggregate objects are not persistent. While -we solved this problem for objects, there is no collection type such -as arrays, hashes or lists which provide this ability. Elephant -provides two primary types of collections, a @code{btree} and a -@code{indexed-btree}. - -We will focus on the core concepts of BTrees in this section, for a -detailed review including the behavior of indexed BTrees, @pxref{Using -BTrees}, @ref{Secondary Indices} and @ref{Using Cursors} in the -@ref{User Guide}. - -Elephant provides a rich data structure called a BTree for storing -large sets of key-value pairs. Every key-value pair is stored -independantly in Elephant just like persistent object slots. -Therefore they inherit all the nice properties of persistent objects: -identity, fast serialization / deserialization, no merge conflicts, -etc. +The remaining problem outlined in the section on @ref{Serialization} +is that operations which mutate collection types do not have +persistent side effects. We have solved this problem for objects, but +not for collections such as as arrays, hashes or lists. Elephant's +solution to this problem is the @code{btree} class which provides +persistent addition, deletion and mutation of elements. + +The BTree stores arbitrarily sized sets of key-value pairs ordered by +key. Every key-value pair is stored independantly in Elephant just +like persistent object slots. They inherit all the important +properties of persistent objects: btree identity and fast +serialization / deserialization. They also resolve the mutated +substructure and nested aggregates problem for collections. Every +mutating write to a btree is an independent and persistent operation +and you can serialize or deserialize a btree without serializing any +of it's key-value pairs. The primary interface to @code{btree} objects is through -@code{get-value}. You can also @code{setf} @code{get-value} to store -key-value pairs. +@code{get-value}. You use @code{setf} @code{get-value} to store +key-value pairs. This interface is very similar to @code{gethash}. + +The following example creates a btree called +@code{*friends-birthdays*} and adds it to the root so we can retrieve +it during a later sessions. We then will add two key-value pairs +consisting of the name of a friend and a universal time encoding their +birthday. @lisp (defvar *friends-birthdays* (make-btree)) => *FRIENDS-BIRTHDAYS* -(add-to-root "friends-birthdays" *friends-birthdays*) +(add-to-root 'friends-birthdays *friends-birthdays*) => #<BTREE @{4951CF6D@}> (setf (get-value "Ben" *friends-birthdays*) @@ -445,8 +496,8 @@ due to serialization semantics may be strange for other values like arrays, lists, standard-objects, etc. -Because elements are sorted by value, we should be able to iterate -over all the elements of the BTree in order. We entered the data in +Because elements are sorted by value, we can iterate over all the +elements of the BTree in order. Notice that we entered the data in reverse alphabetic order, but will read it out in alphabetical order. @lisp @@ -459,16 +510,21 @@ => NIL @end lisp -But what if we want to read out our friends from oldest to youngest, -or youngest to oldest? In the @ref{User Guide}, specifically the -section on @ref{Secondary indices} you will discover ways to sort -according to the order defined by a lisp function of the key-value pair. +But what if we want to read out our friends from oldest to youngest? +One way is to employ another btree that maps birthdays to names, but +this will require storing values multiple times for each update and +increases the burden on the programmer. Elephant provides a better +way. + +The next section @ref{Indexing Persistent Classes} shows you how to +order and retrieve persistent classes by one or more slot values. + @node Indexing Persistent Classes @comment node-name, next, previous, up @section Indexing Persistent Classes -Class indices simplify the recording and retrieving of persistent +Class indexing simplifies the storing and retrieval of persistent objects. An indexed class stores every instance of the class that is created, ensuring that every object is automatically persisted between sessions. @@ -571,16 +627,20 @@ increase the cost of writes and disk storage, each entry is only slightly larger than the size of the slot value. Numbers, small strings and symbols are good candidate types for indexed slots, but -any value may be used, even different types. +any value may be used, even different types. Once a slot is indexed, +we can use the index to retrieve objects by slot values. -Once we've indexed a slot, we can use another set of -@code{get-instances} and @code{map} functions to access objects -in-order and by their slot value. +@code{get-instances-by-value} will retrieve all instances that are +equal to the value argument. @lisp (get-instances-by-value 'friends 'name "Carlos") => (#<Carlos>) +@end lisp +But more interestingly, we can retrieve objects for a range of values. + +@lisp (get-instances-by-range 'friends 'name "Adam" "Devin") => (#<Adriana> #<Carlos>) @@ -591,78 +651,67 @@ name: Zaid birthdate: (14 8 1976) name: Adriana birthdate: (24 4 1980) => (#<Zaid> #<Adriana>) +@end lisp + +To retrieve all instances of a class in the order of the index instead +of the arbitrary order returned by @code{get-instances-by-class} you +can use nil in the place of the start and end values to indicate the +first or last element. (Note: to retrieve instances null values, use +@code{get-instances-by-value} with nil as the argument). -(map-class-index #'print-friend 'friend 'name "Carlos" "Carlos") +@lisp +(get-instances-by-range 'friend 'name nil "Sandra") +=> (#<Adriana> #<Carlos>) + +(get-instances-by-range 'friend 'name nil nil) +=> (#<Adriana> #<Carlos> #<Zaid>) +@end lisp + +There are also functions for mapping over instances of a slot index. +To map over values, use the :value keyword argument. To map by range, +use the :start and :end arguments. + +@lisp +(map-class-index #'print-friend 'friend 'name :value "Carlos") name: Carlos birthdate: (1 1 1972) => NIL -(map-class-index #'print-friend 'friend 'name "Adam" "Devin") +(map-class-index #'print-friend 'friend 'name :start "Adam" :end "Devin") name: Adriana birthdate: (24 4 1980) name: Carlos birthdate: (1 1 1972) => NIL (map-class-index #'print-friend 'friend 'birthday - (encode-birthday '(1 1 1974)) - (encode-birthday '(31 12 1984))) + :start (encode-birthday '(1 1 1974)) + :end (encode-birthday '(31 12 1984))) name: Zaid birthdate: (14 8 1976) name: Adriana birthdate: (24 4 1980) => NIL -(map-class-index #'print-friend 'friend 'birthday nil (encode-birthday '(10 10 1978))) +(map-class-index #'print-friend 'friend 'birthday + :start nil + :end (encode-birthday '(10 10 1978))) name: Carlos birthdate: (1 1 1972) name: Zaid birthdate: (14 8 1976) => NIL (map-class-index #'print-friend 'friend 'birthday - (encode-birthday '(10 10 1975)) - nil) + :start (encode-birthday '(10 10 1975)) + :end nil) name: Zaid birthdate: (14 8 1976) name: Adriana birthdate: (24 4 1980) => NIL @end lisp -You can enable/disable class indexing for an entire class. When you disable -indexing all references to instances of that class are lost. If you re-enable -class indexing only newly created classes will be stored in the class index. -You can manually restore them by using @code{find-class-index} to get the -clas index BTree if you have an alternate in-memory index. - -You can add/remove a secondary index for a slot. So long as the class index -remains, this can be done multiple times without losing any data. - -There is also a facility for defining 'derived slots'. These can be non-slot -parameters which are a function of the class's persistent slot values. For -example you can use an index to keep an alternate representation available -for fast indexing. If an object has an x,y coordinate, you could define a -derived index for r,theta which stored references in polar coordinates. -These would be ordered so you could iterate over a class-index to get objects -in order of increasing radius from the origin or over a range of theta. - -Beware, however, that derived indices have to compute their result every -time you update any persistent instance's slot. This is because there is -no way to know which persistent slots the derived index value(s) depends -on. Thus there is a fairly significant computational cost to objects -with frequent updates having derived indices. The storage cost, however, -may be less as all that is added is the index value and an OID reference -into the class index. To add a slot value you add a serialized -OID+class-ref+slotname to index value which can be much larger if you -use long slotnames and package names and unicode. - -Thus, the question of if and how a given class should be indexed is -very flexible and dynamic, and does not need to be determined at the -beginning of your development. This represents the ability to ``late bind'' -the decision of what to index. - -In general, there is always a tradeoff: an indexed slot increases storage -associated with that slot and slows down write operations. Reads however remain -as fast as for unindexed persistent slots. The Elephant system -makes it simple to choose where and when one wants to utilize this tradeoff. - -Finally, that file @file{src/elephant/classindex-utils.lisp} documents -tools for handling class redefinitions and the policy that should be -used for synchronizing the classes with the database. This process is -somewhat user customizable; documentation for this exists in the source -file referenced above. +The @ref{User Guide} contains a descriptions of the advanced features +of @ref{Class indices} such as ``derived indicies'' that allow you to +order classes according to an arbitrary function, a dynamic API for +adding and removing slots and how to set a policy for resolving +conflicts between the code image and a database where the indexing +specification differs. + +This same facility is also available for your own use. For more +information @pxref{Using Indexed BTrees}. @node Using Transactions @@ -670,24 +719,24 @@ @section Using Transactions One of the most important features of a database is that operations -satisfy the ACID properties: Atomic, Consistent, Isolated, and +enforce the ACID properties: Atomic, Consistent, Isolated, and Durable. In plainspeak, this means that a set of changes is made all at once, that the database is never partially updated, that each set of changes happens sequentially and that a change, once made, is not lost. Elephant provides this protection for all primitive operations. For -example, when you write a value to an indexed BTree, the update to the -BTree and all of its indices is protected by a transaction that -peforms atomic updates to all the BTrees, thus maintaining their -consistency. - -Most real applications will need to have explicit transactions because -you will want one or more read-modify-update operations to happen as -an atomic unit. A common motivating example for this is a banking -system. If a thread is going to modify a balance, we don't want -another thread modifying it in the middle of the operation or one of -the modifications may be lost. +example, when you write a value to an indexed slot, the update to the +persistent slot record as well as the slot index is protected by a +transaction that performs all the updates atomically and thus +enforcing consistency. + +Most real applications will need to use explicit transactions rather +than relying on the primitives alone because you will want multiple +read-modify-update operations act as an atomic unit. A good example +for this is a banking system. If a thread is going to modify a +balance, we don't want another thread modifying it in the middle of +the operation or one of the modifications may be lost. @lisp (defvar *accounts* (make-btree)) --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/25 11:04:38 1.2 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/26 03:37:27 1.3 @@ -34,6 +34,12 @@ @code{initforms} are always evaluated, so beware. (What is the current model here?) +Readers, writers, accessors, and @code{slot-value-using-class} are +employed in redirecting slot accesses to the database, so override +these with care. Because @code{slot-value, slot-boundp, +slot-makunbound} are not generic functions, they are not guaranteed by +the specification to work properly with persistent slots. However the +proper behavior has been verified on SBCL, Allegro and Lispworks. @node The Store Controller @comment node-name, next, previous, up

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 25 Mar '07

25 Mar '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv32631/elephant Modified Files: classindex.lisp collections.lisp Log Message: Another fix for map-index / map-class-index and adding ranges for map-btree (but not map-class --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/03/24 12:16:03 1.32 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/03/25 14:57:49 1.33 @@ -373,10 +373,12 @@ (declare (dynamic-extent map-fn)) (map-btree #'map-fn class-idx)))) -(defun map-class-index (fn class index start end) - "If you want to map over a subset of instances, pick an index - and specify bounds for the traversal. Otherwise use map-class - for all instances" +(defun map-class-index (fn class index &rest args &key start end value) + "To map over a subset of instances, pick an index by slot name + or derived index name and specify the bounds for the traversal. + Otherwise use map-class for all instances. " + (declare (dynamic-extent args) + (ignorable args)) (let* ((index (if (symbolp index) (find-inverted-index class index) index))) @@ -384,7 +386,7 @@ (declare (ignore key pkey)) (funcall fn value))) (declare (dynamic-extent wrapper)) - (map-index #'wrapper index :start start :end end)))) + (map-index #'wrapper index :start start :end end :value value)))) ;; ================= @@ -426,8 +428,7 @@ (declare (ignore k pk)) (push v instances))) (declare (dynamic-extent collector)) - (map-index #'collector (find-inverted-index class slot-name) - :start value :end value)) + (map-index #'collector (find-inverted-index class slot-name) :value value)) (nreverse instances))) (defmethod get-instance-by-value ((class symbol) slot-name value) --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/03/23 16:18:59 1.18 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/03/25 14:57:49 1.19 @@ -318,11 +318,6 @@ different key.) Returns has-tuple / secondary key / value / primary key.")) - -;; ======================================= -;; Generic Mapping Functions -;; ======================================= - (defmacro with-btree-cursor ((var bt) &body body) "Macro which opens a named cursor on a BTree (primary or not), evaluates the forms, then closes the cursor." @@ -331,16 +326,9 @@ (progn ,@body) (cursor-close ,var)))) -(defmethod map-btree (fn (btree btree)) - "Like maphash. Default implementation - overridable - Function of two arguments key and value" - (ensure-transaction (:store-controller (get-con btree)) - (with-btree-cursor (curs btree) - (loop - (multiple-value-bind (more k v) (cursor-next curs) - (declare (dynamic-extent more k v)) - (unless more (return nil)) - (funcall fn k v)))))) +;; ======================================= +;; Generic Mapping Functions +;; ======================================= (defun lisp-compare<= (a b) (etypecase a @@ -348,15 +336,52 @@ (string (string<= a b)) (persistent (<= (oid a) (oid b))))) -(defun lisp-compare-eq (a b) - (eq a b)) +(defun lisp-compare-equal (a b) + (equal a b)) -(defmethod map-index (fn (index btree-index) &rest args &key start end) - "Like map-btree, but takes a function of three arguments key, value and primary key - if you want to get at the primary key value, otherwise use map-btree" +;; NOTE: the use of nil for the last element in a btree only works because the C comparison +;; function orders by type tag and nil is the highest valued type tag so nils are the last +;; possible element in a btree ordered by value. +(defmethod map-btree (fn (btree btree) &rest args &key start end (value nil value-set-p)) + "Map btree maps over a btree from the value start to the value of end. + If values are not provided, then it maps over all values. BTrees + do not have duplicates, but map-btree can also be used with indices + in the case where you don't want access to the primary key so we + require a value argument as well for mapping duplicate value sets." + (let ((end (if value-set-p value end))) + (ensure-transaction (:store-controller (get-con btree)) + (with-btree-cursor (curs btree) + (multiple-value-bind (exists? key value) + (cond (value-set-p + (cursor-set curs value)) + ((null start) + (cursor-first curs)) + (t (cursor-set-range curs start))) + (if exists? + (funcall fn key value) + (return-from map-btree nil)) + (loop + (multiple-value-bind (exists? k v) + (cursor-next curs) + (declare (dynamic-extent exists? k v)) + (if (and exists? (or (null end) (lisp-compare<= k end))) + (funcall fn k v) + (return nil))))))))) + +(defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p)) + "Map-index is like map-btree but for secondary indices, it + takes a function of three arguments: key, value and primary + key. As with map-btree the keyword arguments start and end + determine the starting element and ending element, inclusive. + Also, start = nil implies the first element, end = nil implies + the last element in the index. If you want to traverse only a + set of identical key values, for example all nil values, then + use the value keyword which will override any values of start + and end." (declare (dynamic-extent args) (ignorable args)) - (let ((sc (get-con index))) + (let ((sc (get-con index)) + (end (or value end))) (ensure-transaction (:store-controller sc) (with-btree-cursor (cur index) (labels ((next-range () @@ -379,8 +404,8 @@ (next-range)))))) (declare (dynamic-extent next-range next-in-range)) (multiple-value-bind (exists? skey val pkey) - (cond ((lisp-compare-eq start end) - (cursor-pset cur start)) + (cond (value-set-p + (cursor-pset cur value)) ((null start) (cursor-pfirst cur)) (t (cursor-pset-range cur start))) @@ -393,7 +418,6 @@ nil))))))) - ;; =============================== ;; Some generic utility functions ;; ===============================

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 25 Mar '07

25 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv8936 Modified Files: tutorial.texinfo user-guide.texinfo Log Message: Latest documentation edits - still very broken so don't look! --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/24 12:16:02 1.7 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/25 11:04:38 1.8 @@ -6,15 +6,15 @@ @cindex Tutorial @menu -* Overview:: Overview of elphant's features +* Overview:: Overview of elephant's features. * Getting Started:: Opening and accessing a store. * The Store Root:: Accessing persistent data. * Serialization:: Storage semantics for lisp values. * Persistent Classes:: Persistent semantics for objects. * Rules about Persistent Classes:: What you need to know. -* Persistent collections:: Keep track of collections of objects. -* Class Indices:: Simple way to keep track of instances. -* Using Transactions:: Enabling ACID database properties. +* Persistent collections:: Keep track of collections of values. +* Indexing Persistent Classes:: Simple way to keep track of persistent instances. +* Using Transactions:: Providing ACID database properties. @end menu @node Overview @@ -377,82 +377,249 @@ the most recent commits, right? Note that this can be used as a weak form of IPC. But also note that in particular, if your slot value is not an immediate value, reading will cons or allocate the value. Gets -are not an expensive operation; you can perform tens of thousands of -primitive reads per second. However, if you're concerned, cache -large values. +are not an expensive operation; you can perform thousands to tens of +thousands of primitive reads per second. However, if you're +concerned, cache large values in memory. @node Persistent collections @comment node-name, next, previous, up @section Persistent collections +The remaining problem outlined in @ref{Serialization} is that +operations which mutate aggregate objects are not persistent. While +we solved this problem for objects, there is no collection type such +as arrays, hashes or lists which provide this ability. Elephant +provides two primary types of collections, a @code{btree} and a +@code{indexed-btree}. + +We will focus on the core concepts of BTrees in this section, for a +detailed review including the behavior of indexed BTrees, @pxref{Using +BTrees}, @ref{Secondary Indices} and @ref{Using Cursors} in the +@ref{User Guide}. +Elephant provides a rich data structure called a BTree for storing +large sets of key-value pairs. Every key-value pair is stored +independantly in Elephant just like persistent object slots. +Therefore they inherit all the nice properties of persistent objects: +identity, fast serialization / deserialization, no merge conflicts, +etc. + +The primary interface to @code{btree} objects is through +@code{get-value}. You can also @code{setf} @code{get-value} to store +key-value pairs. + +@lisp +(defvar *friends-birthdays* (make-btree)) +=> *FRIENDS-BIRTHDAYS* + +(add-to-root "friends-birthdays" *friends-birthdays*) +=> #<BTREE @{4951CF6D@}> + +(setf (get-value "Ben" *friends-birthdays*) + (encode-universal-time 0 0 0 14 4 1973)) +=> 2312600400 + +(setf (get-value "Andrew" *friends-birthdays*) + (encode-universal-time 0 0 0 22 12 1976)) +=> 2429071200 + +(get-value "Andrew" *friends-birthdays*) +=> 2429071200 +=> T + +(decode-universal-time *) +=> 0 + 0 + 0 + 22 + 12 + 1976 + 2 + NIL + 6 +@end lisp + +In addition to the hash-table like interface, @code{btree} stores +pairs sorted by the lisp value of the key, lowest to highest. This is +works well for numbers, strings, symbols and persistent objects, but +due to serialization semantics may be strange for other values like +arrays, lists, standard-objects, etc. + +Because elements are sorted by value, we should be able to iterate +over all the elements of the BTree in order. We entered the data in +reverse alphabetic order, but will read it out in alphabetical order. + +@lisp +(map-btree (lambda (k v) + (format t "name: ~A utime: ~A~%" k + (subseq (multiple-value-list (decode-universal-time v)) 3 6))) + *friends-birthdays*) +"Andrew" +"Ben" +=> NIL +@end lisp -@node Class Indices +But what if we want to read out our friends from oldest to youngest, +or youngest to oldest? In the @ref{User Guide}, specifically the +section on @ref{Secondary indices} you will discover ways to sort +according to the order defined by a lisp function of the key-value pair. + +@node Indexing Persistent Classes @comment node-name, next, previous, up -@section Class Indices +@section Indexing Persistent Classes + +Class indices simplify the recording and retrieving of persistent +objects. An indexed class stores every instance of the class that is +created, ensuring that every object is automatically persisted between +sessions. + +@lisp +(defpclass friend () + ((name :accessor name :initarg :name) + (birthday :initarg :birthday)) + (:index t)) +=> #<PERSISTENT-METACLASS FRIEND> + +(defmethod print-object ((f friend) stream) + (format stream "#<~A>" (name f))) + +(defun encode-birthday (dmy) + (apply #'encode-universal-time + (append '(0 0 0) dmy))) + +(defmethod (setf birthday) (dmy (f friend)) + (setf (slot-value f 'birthday) + (encode-birthday dmy)) + dmy) + +(defun decode-birthday (utime) + (subseq (multiple-value-list (decode-universal-time utime)) 3 6)) + +(defmethod birthday ((f friend)) + (decode-birthday (slot-value f 'birthday))) +@end lisp + +Notice the class argument ``:index t''. This tells Elephant to store +a reference to this class. Under the covers, there are a set of +btrees that keep track of classes, but we won't need to worry about +that as all the functionality has been nicely packaged for you. + +We also created our own birthday accessor for convenience so it +accepts and returns birthdays in a list consisting of month, day and +year such as @code{(27 3 1972)}. The index key will be the encoded +universal time, however. -Class indices are a very convenient way of gaining the efficiency that -BTrees provide. If a given object is most often sought by the value -of one of its slots, which is of course quite common, it is convenient -to define a class index on that slot, although the same functionality -can be gained in a more complicated way through the use of secondary -indices. - -The file @file{tests/testindexing.lisp} provides many useful examples -of both declaring class indexes and using the API to seek objects using them. - -The following code from that file in the test ``indexing-range'' demonstrates -the convenience of a class indexes and the function ``get-instances-by-range''. -Note in the definition of the ``slot1'' the keyword ``:index'' is used to -specify that this slot should be indexed. - -@lisp - (defclass idx-four () - ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) - (:metaclass persistent-metaclass)) - - - (defun make-idx-four (val) - (make-instance 'idx-four :slot1 val)) - - (with-transaction () - (mapc #'make-idx-four '(1 1 1 2 2 4 5 5 5 6 10))) - - (let ((x1 (get-instances-by-range 'idx-four 'slot1 2 6)) - (x2 (get-instances-by-range 'idx-four 'slot1 0 2)) - (x3 (get-instances-by-range 'idx-four 'slot1 6 15)) - ) - (format t " x1 = ~A~%" (mapcar #'slot1 x1)) - (format t " x2 = ~A~%" (mapcar #'slot1 x2)) - (format t " x3 = ~A~%" (mapcar #'slot1 x3)) -@end lisp - -Additionally, the test -@lisp -(do-test 'INDEXING-TIMING) -@end lisp -Can be used to judge the performance of indexing a medium sized dataset. - -The file @file{src/elephant/classindex.lisp} provides the source code and -some crisp documentation of the class indexing system. - -Note that for retrieving items, the API is provided by three functions: - -@lisp -(defgeneric get-instances-by-class (persistent-metaclass)) -(defgeneric get-instances-by-value (persistent-metaclass slot-name value)) -(defgeneric get-instances-by-range (persistent-metaclass slot-name start end)) -@end lisp - -By using these functions, any class that is a subclass of persistent-metaclass -can also be thought of as a container of all of its instances, which are -persistent in the database between lisp invocations. Moreover an individual -object can be looked up on O(log n) time via a value on which it is indexed. - -At the top of this same file, you will find the a description of the API -which can be used to dynamically add and remove indexes. (Adding and -removing indexes can also be performed by a re-execution of the ``defclass'' -macro with different values.) +Now we can easily manipulate all the instances of a class. + +@lisp +(defun print-friend (friend) + (format t " name: ~A birthdate: ~A~%" (name friend) (birthday friend))) + +(make-instance 'friend :name "Carlos" :birthday (encode-birthday '(1 1 1972))) +(make-instance 'friend :name "Adriana" :birthday (encode-birthday '(24 4 1980))) +(make-instance 'friend :name "Zaid" :birthday (encode-birthday '(14 8 1976))) + +(get-instances-by-class 'friends) +=> (#<Carlos> #<Adriana> #<Zaid>) + +(mapcar #'print-friend *) + name: Carlos birthdate: (1 1 1972) + name: Adriana birthdate: (24 4 1980) + name: Zaid birthdate: (14 8 1976) +=> (#<Carlos> #<Adriana> #<Zaid>) +@end lisp + +But what if we have thousands of friends? Aside from never getting +work done, our get-instances-by-class will be doing a great deal of +consing, eating up lots of memory and wasting our time. Fortunately +there is a more efficient way of dealing with all the instances of a +class. + +@lisp +(map-class #'print-friend 'friend) + name: Carlos birthdate: (1 1 1972) + name: Adriana birthdate: (24 4 1980) + name: Zaid birthdate: (14 8 1976) +=> NIL +@end lisp + +@code{map-class} has the advantage that it does not keep references to +objects after they are processed. The garbage collector can come +along, clear references from the weak instance cache so that your +working set is finite. The list version above conses all objects into +memory before you can do anything with them. The deserialization +costs are very low in both cases. + +Notice that the order in which the records are printed are not sorted +according to either name or birthdate. Elephant makes no guarantee +about the ordering of class elements, so you cannot depend on the +insertion ordering shown here. + +So what if we want ordered elements? How do we access our friends +according to name and birthdate? This is where slot indices come into +play. + +@lisp +(defpclass friend () + ((name :accessor name :initarg :name :index t) + (birthday :initarg :birthday :index t))) +@end lisp + +Notice the :index argument to the slots. Also notice that we dropped +the class :index argument. Specifying that a slot is indexed +automatically registers the class as indexed. While slot indices +increase the cost of writes and disk storage, each entry is only +slightly larger than the size of the slot value. Numbers, small +strings and symbols are good candidate types for indexed slots, but +any value may be used, even different types. + +Once we've indexed a slot, we can use another set of +@code{get-instances} and @code{map} functions to access objects +in-order and by their slot value. + +@lisp +(get-instances-by-value 'friends 'name "Carlos") +=> (#<Carlos>) + +(get-instances-by-range 'friends 'name "Adam" "Devin") +=> (#<Adriana> #<Carlos>) + +(get-instances-by-range 'friend 'birthday (encode-birthday '(1 1 1974)) (encode-birthday '(31 12 1984))) +=> (#<Zaid> #<Adriana>) + +(mapc #'print-friend *) + name: Zaid birthdate: (14 8 1976) + name: Adriana birthdate: (24 4 1980) +=> (#<Zaid> #<Adriana>) + +(map-class-index #'print-friend 'friend 'name "Carlos" "Carlos") + name: Carlos birthdate: (1 1 1972) +=> NIL + +(map-class-index #'print-friend 'friend 'name "Adam" "Devin") + name: Adriana birthdate: (24 4 1980) + name: Carlos birthdate: (1 1 1972) +=> NIL + +(map-class-index #'print-friend 'friend 'birthday + (encode-birthday '(1 1 1974)) + (encode-birthday '(31 12 1984))) + name: Zaid birthdate: (14 8 1976) + name: Adriana birthdate: (24 4 1980) +=> NIL + +(map-class-index #'print-friend 'friend 'birthday nil (encode-birthday '(10 10 1978))) + name: Carlos birthdate: (1 1 1972) + name: Zaid birthdate: (14 8 1976) +=> NIL + +(map-class-index #'print-friend 'friend 'birthday + (encode-birthday '(10 10 1975)) + nil) + name: Zaid birthdate: (14 8 1976) + name: Adriana birthdate: (24 4 1980) +=> NIL +@end lisp You can enable/disable class indexing for an entire class. When you disable indexing all references to instances of that class are lost. If you re-enable @@ -606,16 +773,14 @@ to retry the transaction a fixed number of times by re-executing the whole body. -You can see in the example that a single statement in lisp can include -several primitive Elephant operations as in the decf statement in -withdraw. It takes some careful thinking to properly implement -transactions, for a complete treatment @pxref{Transaction details}. - -The other thing transactions can give us is the ability to put off -synchronizing our data to disk. The expensive part of persistent -writes is flushing data to disk. Since a transaction caches values, -all the read and written values are kept in memory until the -transaction is complete, this can dramatically improve performance. +The other value transactions provide is the capability to delay +flushing dirty data to disk. The most time-intensive part of +persistent operations is flushing newly written data to disk. Using +the default auto-commit behavior requires a flush on every operation +which can become very expensive. Because a transaction caches values, +all the values read or written are cached in memory until the +transaction completes, dramatically decreasing the number of flushes +and the total time taken. @lisp (defpclass test () @@ -625,8 +790,9 @@ (make-instance 'test :slot1 i))) @end lisp -This can take a long time. Here each new objects that is created has -to independantly write its value to disk and accept a disk flush cost. +This can take a long time, well over a minute on the CLSQL data store. +Here each new objects that is created has to independantly write its +value to disk and accept a disk flush cost. @lisp (time (with-transaction () @@ -634,7 +800,8 @@ (make-instance 'test :slot1 i)))) @end lisp -Here, ....... +Wrapping this operation in a transaction dramatically increases the +time from 10's of seconds to a second or less. @lisp (time (with-transaction () @@ -642,14 +809,64 @@ (make-instance 'test :slot1 i)))) @end lisp -These are huge differences in performance! -Of course since we are caching all this data in memory, we cannot have -infinitely sized transactions. Large operations need to get split up -into subtransactions. When dealing with persistent objects a good -rule of thumb is to keep your transactions less than 1000 at a time. +When we increase the number of objects within the transaction, the +time cost does not go up linearly. This is because the total time to +write a hundred simple objects is still dominated by the final +synchronization step. + [56 lines skipped] --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 1.1 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/25 11:04:38 1.2 @@ -76,49 +76,71 @@ @code{with-open-controller} macro. Opening and closing a controller is very expensive. + +@node{Class indices} +@comment node-name, next, previous, up +@section Class indicies + +You can enable/disable class indexing for an entire class. When you disable +indexing all references to instances of that class are lost. If you re-enable +class indexing only newly created classes will be stored in the class index. +You can manually restore them by using @code{find-class-index} to get the +clas index BTree if you have an alternate in-memory index. + +You can add/remove a secondary index for a slot. So long as the class index +remains, this can be done multiple times without losing any data. + +There is also a facility for defining 'derived slots'. These can be non-slot +parameters which are a function of the class's persistent slot values. For +example you can use an index to keep an alternate representation available +for fast indexing. If an object has an x,y coordinate, you could define a +derived index for r,theta which stored references in polar coordinates. +These would be ordered so you could iterate over a class-index to get objects +in order of increasing radius from the origin or over a range of theta. + +Beware, however, that derived indices have to compute their result every +time you update any persistent instance's slot. This is because there is +no way to know which persistent slots the derived index value(s) depends +on. Thus there is a fairly significant computational cost to objects +with frequent updates having derived indices. The storage cost, however, +may be less as all that is added is the index value and an OID reference +into the class index. To add a slot value you add a serialized +OID+class-ref+slotname to index value which can be much larger if you +use long slotnames and package names and unicode. + +Thus, the question of if and how a given class should be indexed is +very flexible and dynamic, and does not need to be determined at the +beginning of your development. This represents the ability to ``late bind'' +the decision of what to index. + +In general, there is always a tradeoff: an indexed slot increases storage +associated with that slot and slows down write operations. Reads however remain +as fast as for unindexed persistent slots. The Elephant system +makes it simple to choose where and when one wants to utilize this tradeoff. + +Finally, that file @file{src/elephant/classindex-utils.lisp} documents +tools for handling class redefinitions and the policy that should be +used for synchronizing the classes with the database. This process is +somewhat user customizable; documentation for this exists in the source +file referenced above. + @node{Using BTrees} @comment node-name, next, previous, up @section Using BTrees -The btree class are to hash-tables as persistent-objects are to -ordinary objects. BTrees have a hash-table-like interface, but store -their keys and values directly as rows in a Sleepycat BTree. Btrees -may be persisted simply by their OID. Hence they have all the nice -properties of persistent objects: identity, fast serialization / -deserialization, no merge conflicts..... - - (defvar *friends-birthdays* (make-btree)) - => *FRIENDS-BIRTHDAYS* - - (add-to-root "friends-birthdays" *friends-birthdays*) - => #<BTREE {4951CF6D}> - - (setf (get-value "Andrew" *friends-birthdays*) - (encode-universal-time 0 0 0 22 12 1976)) - => 2429071200 - - (setf (get-value "Ben" *friends-birthdays*) - (encode-universal-time 0 0 0 14 4 1976)) - => 2407298400 - - (get-value "Andrew" *friends-birthdays*) - => 2429071200 - => T - - (decode-universal-time *) - => 0 - 0 - 0 - 22 - 12 - 1976 - 2 - NIL - 6 - -Because of serialization semantics, BTrees hash on a value, not -identity. This is probably ok for strings, numbers, and persistent -things, but may be strange for other values. +A BTree is a data structure designed for on-disk databases. +Traditional binary trees are a tree structure that stores a key and +value and two links in each node. To get to a value, you compare your +query key to the node key. If it is equal, return the value in this +node. If it is less, follow the first link and if it is greater, +follow the second link. The problem here is that every link requires +a disk seek. + +The BTree exploits the properties of memory/disk data heirarchies. +The key properties are that disk seeks are expensive, loading large +blocks of data is relatively inexpensive after a seek and comparisons +on objects that are stored in memory is cheap. + @node Repository Migration @comment node-name, next, previous, up

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv32267 Modified Files: Makefile make-ref.lisp Added Files: data-store-reference.texinfo elephant-design.texinfo lisp-data-store.texinfo scenarios.texinfo style.css user-guide.texinfo Removed Files: notes.texinfo Log Message: First batch of edits for new user manual --- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 12:16:02 1.3 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 13:55:15 1.4 @@ -1,8 +1,10 @@ +all: docs +includes-stuff: + cd includes; sbcl < ../make-ref.lisp docs: includes-stuff - makeinfo -v --html --force elephant.texinfo + makeinfo -v --html --css-include=style.css --force elephant.texinfo + makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo -includes-stuff: - cd includes; sbcl < ../make-ref.lisp --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 12:16:02 1.3 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 13:55:15 1.4 @@ -1,10 +1,21 @@ (require 'asdf) (asdf:operate 'asdf:load-op 'elephant-tests) -(sb-posix:chdir "/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/") -(load "/Users/eslick/Work/fsrc/elephant-cvs/doc/docstrings.lisp") +(defparameter include-dir-path + (namestring + (merge-pathnames + #p"doc/includes/" + (asdf:component-pathname (asdf:find-system 'elephant-tests))))) + +(defparameter docstrings-path + (namestring + (merge-pathnames + #p"doc/docstrings.lisp" + (asdf:component-pathname (asdf:find-system 'elephant-tests))))) + +(sb-posix:chdir include-dir-path) +(load docstrings-path) (defun make-docs () -;; (when (check-complete) (when t (elephant:open-store elephant-tests::*testbdb-spec*) (make-instance 'elephant::persistent-collection) --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*- @node Data Store API Reference @comment node-name, next, previous, up @chapter Data Store API Reference @cindex Data Store API Reference @cindex Data Store @cindex API Reference These are the functions that need to be overridden to implement support for a data store backend. Included are the exported elephant functions that need methods defined on them. Some functions here are utilities from the main elephant package that support store implementations. Migration, class indices and query interfaces are implemented on top of the store API and require no special support by implementors. @menu * Registration:: Register the backend to parse controller specifications * Store Controllers:: Subclassing the store controller. * Slot access:: Support for metaprotocol slot access. * Collections:: BTrees and indices. * Cursors:: Traversing BTrees. * Transactions:: Transaction implementation. * Multithreading:: Multithreading considerations. * Serialization:: Facilities for serializing objects. * C Utilities:: Writing primitive C types. * Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries @end menu @node Registration @comment node-name, next, previous, up @section Registration @cindex Registration @include includes/fun-elephant-register-backend-con-init.texinfo @include includes/fun-elephant-lookup-backend-con-init.texinfo @node Store Controllers @comment node-name, next, previous, up @section Store Controllers @cindex Store Controllers Subclass store-controller and implement store and close controller which are called by open-store and close-store respectively. @include includes/fun-elephant-store-controller.texinfo @include includes/fun-elephant-backend-open-controller.texinfo @include includes/fun-elephant-backend-close-controller.texinfo The slots for these accessors must be initialized. @include includes/fun-elephant-backend-database-version.texinfo @include includes/fun-elephant-backend-controller-serialize.texinfo @include includes/fun-elephant-backend-controller-deserialize.texinfo @include includes/fun-elephant-backend-root.texinfo @include includes/fun-elephant-backend-class-root.texinfo These functions are important utilities for implementing store-controllers. @include includes/fun-elephant-backend-oid.texinfo @include includes/fun-elephant-backend-get-con.texinfo @include includes/fun-elephant-backend-next-oid.texinfo @include includes/fun-elephant-backend-connection-is-indeed-open.texinfo @node Slot Access @comment node-name, next, previous, up @section Slot Access @cindex Slot Access These functions are called by the metaclass protocol to support operations on persistent class slots. @include includes/fun-elephant-backend-persistent-slot-writer.texinfo @include includes/fun-elephant-backend-persistent-slot-reader.texinfo @include includes/fun-elephant-backend-persistent-slot-boundp.texinfo @include includes/fun-elephant-backend-persistent-slot-makunbound.texinfo @node Collections @comment node-name, next, previous, up @section Collections @cindex Collections @c #:btree #:btree-index #:indexed-btree @c #:build-indexed-btree #:build-btree #:existsp @c #:map-indices @node Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors @c #:cursor @c #:cursor-btree @c #:cursor-oid @c #:cursor-initialized-p @node Transactions @comment node-name, next, previous, up @section Transactions @cindex Transactions @c #:*current-transaction* @c #:make-transaction-record @c #:transaction-store @c #:transaction-object @c #:execute-transaction @c #:controller-start-transaction @c #:controller-commit-transaction @c #:controller-abort-transaction @node Multithreading @comment node-name, next, previous, up @section Multithreading @cindex Multithreading @node Serialization @comment node-name, next, previous, up @section Serialization @cindex Serialization @c #:deserialize #:serialize @c #:serialize-symbol-complete @c #:deserialize-from-base64-string @c #:serialize-to-base64-string @node Memory utilities @comment node-name, next, previous, up @section Memory utilities @cindex Memory utilities @node Foreign libraries @comment node-name, next, previous, up @section Foreign libraries @cindex Foreign libraries --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 1.1 Debugger entered--Lisp error: (void-variable Design) eval(Design) eval-last-sexp-1(nil) eval-last-sexp(nil) call-interactively(eval-last-sexp) --- /project/elephant/cvsroot/elephant/doc/lisp-data-store.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/lisp-data-store.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*- @node Lisp Data Store @comment node-name, next, previous, up @chapter Lisp Data Store @cindex Lisp Data Store @cindex Data Store @cindex API Reference --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*- @node Usage Scenarios @comment node-name, next, previous, up @chapter Usage Scenarios @cindex Usage Scenarios Sorry, haven't written this section yet. Simple file replacement and indexing - Keep track of ordinary objects, ignore metaprotocol Persist system objects - Intermingle persistent objects and regular objects - Look up objects using class indices Full database system - storage, rich data models, references, queries, etc Multithreaded web applications - DB + multithreading Object-oriented data storage, large graph traversals --- /project/elephant/cvsroot/elephant/doc/style.css 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/style.css 2007/03/24 13:55:15 1.1 --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*- @node User Guide @comment node-name, next, previous, up @chapter User Guide @cindex User Guide @menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. * Reachability:: Determining liveness in a store. * Persistent objects:: All the dirt on persistent objects. * Class indices:: In-depth discussion about indexing persistent indices. * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. * Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Repository Migration and Upgrade:: How to move objects from one repository to another. * Garbage collection:: How to recover storage and OIDs in long-lived repositories. * Performance tuning:: How to get the most from Elephant. @end menu @node Persistent objects @comment node-name, next, previous, up @section Persistent Objects Finally, if you for some reason make an instance with a specified OID which already exists in the database, @code{initargs} take precedence over values in the database, which take precedences over @code{initforms}. Also currently there is a bug where @code{initforms} are always evaluated, so beware. (What is the current model here?) @node The Store Controller @comment node-name, next, previous, up @section The Store Controller What is @code{open-store} doing? It creates a @code{store-controller} object, and sets the special @code{*store-controller*} to point to it. The store controller holds the handles to the database environment and tables, and some other bookkeeping. If for some reason you need to run recovery on the database (see sleepycat docs) you can specify that with the @code{:recover} and @code{:recover-fatal} keys. To create one by hand one can do, @lisp * (setq *store-controller* (make-instance 'store-controller :path "testdb")) => #<STORE-CONTROLLER @{49252F75@}> * (open-controller *store-controller*) => #<STORE-CONTROLLER @{49252F75@}> @end lisp but @lisp * (open-store "testdb")) @end lisp is the preferred mechanism. This opens the environment and database. The @code{persistent-*} objects reference the @code{*store-controller*} special. (This is in part because slot accessors can't take additional arguments.) If for some reason you want to operate on 2 store controllers, you'll have to do that by flipping the @code{*store-controller*} special. @code{close-store} closes the store controller. Alternatively @code{close-controller} can be called on a handle. Don't forget to do this or else you may need to run recovery later. There is a @code{with-open-controller} macro. Opening and closing a controller is very expensive. @node{Using BTrees} @comment node-name, next, previous, up @section Using BTrees The btree class are to hash-tables as persistent-objects are to ordinary objects. BTrees have a hash-table-like interface, but store their keys and values directly as rows in a Sleepycat BTree. Btrees may be persisted simply by their OID. Hence they have all the nice properties of persistent objects: identity, fast serialization / deserialization, no merge conflicts..... (defvar *friends-birthdays* (make-btree)) => *FRIENDS-BIRTHDAYS* (add-to-root "friends-birthdays" *friends-birthdays*) => #<BTREE {4951CF6D}> (setf (get-value "Andrew" *friends-birthdays*) (encode-universal-time 0 0 0 22 12 1976)) => 2429071200 (setf (get-value "Ben" *friends-birthdays*) (encode-universal-time 0 0 0 14 4 1976)) => 2407298400 (get-value "Andrew" *friends-birthdays*) => 2429071200 => T (decode-universal-time *) => 0 0 0 22 12 1976 2 NIL 6 Because of serialization semantics, BTrees hash on a value, not identity. This is probably ok for strings, numbers, and persistent things, but may be strange for other values. @node Repository Migration @comment node-name, next, previous, up @section Repository Migration This version of Elephant supports migration between store controllers of any backend type. The tests @code{migrate1} - @code{migrate5} are demonstrations of this capability. There is a single generic function @code{migrate} that is used to copy different object types to a target repository. It is assumed that typically migrate will be called on two repositories and all live objects (those reachable in the root or class-root) will be copied to the target repository via recursive calls to migrate for specific objects. When persistent instances are copied, their internal pointer will be updated to point to the new repository so after migration the lisp image should be merely updated to refer to the target repository in the *store-controller* variable or whatever variable the application is using to store the primary controller instance. There are some limitations to the current migration implementation: @enumerate @item Migrate currently will not handle circular list objects @item Migrate does not support arrays with nested persistent objects @item Indexed classes only have their class index copied if you use the top level migration. Objects will be copied without slot data if you try to migrate an object outside of a store-to-store migration due to the class object belonging to one store or another @item Migrate assumes that after migration, indexed classes belong to the target store. @item In general, migration is a one-time activity and afterwards (or after a validation test) the source store should be closed. Any failures in migration should then be easy to catch. @item Each call to migration will be good about keeping track of already copied objects to avoid duplication. Duplication shouldn't screw up the semantics, just add storage overhead but is to be avoided. However this information is not saved between calls and there's no other way to do comparisons between objects across stores (different oid namespaces) so user beware of the pitfalls of partial migrations... @item Migrate keeps a memory-resident hash of all objects; this means you cannot currently migrate a store that has more data than your main memory. (This could be fixed by keeping the oid table in the target store and deleting it on completion) @item Migration does not maintain OID equivalence so any datastructures which index into those will have to have a way to reconstruct themselves (better to keep the object references themselves rather than oids in general) but they can overload the migrate method to accomplish this cleanly @end enumerate Users can customize migration if they create unusual datastructures that are not automatically supported by the existing @code{migrate} methods. For example, a datastructure that stores only object OIDs instead of serialized object references will need to overload migrate to ensure that all referenced objects are in fact copied (otherwise the OIDs will just be treated as fixnums potentially leaving dangling references. To customize migration overload a version of migrate to specialize on your specific persistent class type. @lisp (defmethod migrate ((dst store-controller) (src my-class))) @end lisp In the body of this method you can call @code{(call-next-method)} to get a destination repository object with all the slots copied over to the target repository which you can then overwrite. To avoid the default persistent slot copying, bind the dynamic variable @code{*inhibit-slot-writes*} in your user method using @code(with-inhibited-slot-copy () ...)} a convenience macro. @node Threading @comment node-name, next, previous, up @section Threading Sleepycat plays well with threads and processes. The store controller is thread-safe by default, that is, can be shared amongst threads. This is set by the @code{:thread} key. Transactions may not be shared amongst threads except serially. One thing which is NOT thread and process safe is recovery, which should be run when no one is else is talking to the database environment. Consult the Sleepycat docs for more information. Elephant uses some specials to hold parameters and buffers. If you're using a natively threaded lisp, you can initialize these specials to thread-local storage by using the @code{run-elephant-thread} function, assuming your lisp creates thread-local storage for let-bound specials. (This functionality is currently broken) [72 lines skipped]

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv6422/doc Modified Files: Makefile copying.texinfo elephant.texinfo installation.texinfo intro.texinfo make-ref.lisp package-elephant.texinfo reference.texinfo tutorial.texinfo Log Message: Cleanup indexing tests so we always have a clean slate --- /project/elephant/cvsroot/elephant/doc/Makefile 2005/11/23 17:51:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 12:16:02 1.3 @@ -5,4 +5,4 @@ makeinfo -v --html --force elephant.texinfo includes-stuff: - cd includes; lisp < ../make-ref.lisp + cd includes; sbcl < ../make-ref.lisp --- /project/elephant/cvsroot/elephant/doc/copying.texinfo 2006/03/02 14:44:49 1.2 +++ /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/03/24 12:16:02 1.3 @@ -15,15 +15,13 @@ as governed by the terms of the Lisp Lesser GNU Public License @uref{http://opensource.franz.com/preamble.html}, also known as the LLGPL. - - Copyrights include: Copyright (c) 2004 by Andrew Blumberg and Ben Lee -Copyright (c) 2006 by Ian Eslick +Copyright (c) 2006-2007 by Ian Eslick -Copyright (c) 2005,2006 by Robert L. Read +Copyright (c) 2005-2007 by Robert L. Read Portions of this program (namely the C unicode string --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2006/03/02 14:44:49 1.4 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/24 12:16:02 1.5 @@ -6,7 +6,7 @@ @copying Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. -Copyright @copyright{} 2006 Robert L. Read. +Copyright @copyright{} 2006-2007 Robert L. Read and Ian Eslick @quotation Permission is granted to copy, distribute and/or modify this document @@ -19,8 +19,8 @@ @titlepage @title Elephant User Manual -@subtitle Elephant version 0.6 -@author Ben Lee +@subtitle Elephant version 0.6.1 +@author Ben Lee and Ian Eslick @c The following two commands @c start the copyright page. @@ -29,24 +29,35 @@ @insertcopying @end titlepage -@c So the toc is printed at the start. -@contents - @ifnottex @node Top @comment node-name, next, previous, up -@top Elephant +@top Copyright @insertcopying @end ifnottex @menu -* Introduction:: Introducing Elephant! -* Tutorial:: A leisurely walk-through. -* Reference:: API documentation. -* Installation:: Installation and test-suite procedures and issues -* Design Notes:: Internals. -* Copying:: Your rights and freedoms. +* Table of Contents:: +@end menu + +@chapheading Chapters + +@menu +* Introduction:: Introduction to the Elephant Persistent Object System. +* Tutorial:: A basic ``getting started'' tutorial. +* Installation:: Installation and test-suite procedures. +* User Guide:: In depth discussion of all Elephant facilities and features. +* Usage scenarios:: Design scenarios for Elephant applications. +* User API Reference:: Function and class documentation of the user API. +* Elephant Design:: An overview of elephant's internal architecture. +* Data Store API Reference:: Function level documentation for data store implementors. +* Copying:: Your rights and freedoms. +@end menu + +@chapheading Appendices + +@menu * Concept Index:: * Object Index:: * Function / Macro Index:: @@ -54,11 +65,18 @@ * Colophon:: @end menu +@node Table of Contents +@comment node-name, next, previous, up +@contents + @include intro.texinfo @include tutorial.texinfo -@include reference.texinfo -@include notes.texinfo @include installation.texinfo +@include user-guide.texinfo +@include scenarios.texinfo +@include reference.texinfo +@include elephant-design.texinfo +@include data-store-reference.texinfo @include copying.texinfo @node Concept Index --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2006/05/15 13:02:26 1.3 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/24 12:16:02 1.4 @@ -8,10 +8,11 @@ @menu * Installation Basics:: Basic installation * Test-Suites:: Running the test suites -* SQL-Introduction:: The design and status of the SQL back-end extention. -* Extension Status:: The current status of the SQL back-end extention. -* Multi-repository Operation:: Specifying repositories -* Setting up PostGres:: An example +* Berkeley DB Introduction:: The Berkeley DB backend +* SQL Data Store:: The design and status of the SQL back-end extension. +* Lisp Data Store:: A native lisp-based repository. +* Multi-repository Operation:: Specifying repositories. +* Setting up PostGres:: An example. @end menu @node Installation Basics @@ -148,11 +149,14 @@ If you get errors, you may wish to report it the @code{ elephant-devel at common-lisp.net} email list. +@node Berkeley DB Repository +@comment node-name, next, previous, up +@section Berkeley DB Repository -@node SQL-Introduction +@node SQL Repository @comment node-name, next, previous, up -@section SQL-Introduction +@section SQL Repository Although originally designed as an interface to the BerkeleyDB system, the original Elephant system has been experimenetally extended to --- /project/elephant/cvsroot/elephant/doc/intro.texinfo 2006/04/26 17:53:43 1.4 +++ /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/03/24 12:16:02 1.5 @@ -5,66 +5,96 @@ @chapter Introduction @cindex Introduction -Elephant is a persistent object database for Common Lisp that -supports storing CLOS objects and most lisp primitives. -It supports persistent collections via a BTree interface. - -Elephant was originally developed as an interface layer on top -of the Sleepycat / Berkeley DB library, a widely-distributed -embedded database. Many unix systems have it installed by default. -Berkeley DB is ACID compliant, transactional, process and -thread safe, and fast relative to relational databases. Recently, -Elephant was extended to provide support for relational database backends. -It has been tested with Postgres and SQLite 3. It supports, with some -care, simultaneous multi-repository operation and enables convenient -migration of data between repositories. +Elephant is a persistent object protocol and database for Common +Lisp. The persistent protocol component of elephant overrides class +creation and standard slot accesses using the Meta-Object Protocol +(MOP) to render slot values persistent. Database functionality +includes the ability to persistently index and retrieve ordered sets +of class instances and ordinary lisp values. Elephant has an +extensive test suite and the core functionality is becoming quite +mature. + +The Elephant code base is available under the LLGPL license. Data +stores each come with their own, separate license and you will have to +evaluate the implications of using them yourself. + +@section History + +Elephant was originally envisioned as a lightweight interface layer on +top of the Berkeley DB library, a widely-distributed embedded database +that many unix systems have installed by default. Berkeley DB is ACID +compliant, transactional, process and thread safe, and fast relative +to relational databases. + +Elephant has been extended to provide support for multiple backends, +specifically a relational database backend based on CL-SQL which has +been tested with Postgres and SQLite 3. It supports, with some care, +multi-repository operation and enables convenient migration of data +between repositories. The support for relational backends and migration to the LLGPL was to allow for broader use of Elephant in both not-for-profit and commercial -settings. +settings. Several additional backends are planned for future releases +including a native Lisp implementation released under the LLGPL. -Elephant goals: +Elephant's current development focus is to enhance the feature set +including a native lisp backend, a simple query language, and flexible +persistence models that selectively break one or more of the ACID +constraints to improve performance. + +@section Elephant Goals @itemize -@item Transparency: most Lisp values are easy to persist without -signifcant effort or special syntax. Talk to the DB entirely from Lisp; -not requirement for domain-specific languages (such as SQL) to access persistent -resources. Enable interactive control of the database with no external -server dependencies. - -@item Safety: ACID, transactions. Concurrent with good multi-user and -multi-threaded semantics, isolation, locking and deadlock detection. -(Deadlock detection does require an external process to be launched) +@item @strong{Transparency:} most Lisp values are easy to persist without +significant effort or special syntax. You can interact with the DB +entirely from Lisp. There is no requirement to use domain-specific +languages, such as SQL, to access persistent resources. Elephant +loads via ASDF and requires no external server (except for some SQL +backends like Postgres). -@item Simplicity: a small library with few surprises for the +@item @strong{Simplicity:} a small library with few surprises for the programmer. Lisp and Berkeley DB together are an excellent substrate; -Elephant tries to leverage their features as much as possible. -Support for multiple backends should be load-time options and mostly +Elephant tries to leverage their features as much as possible. +Support for additional backends are load-time options and more or less transparent to the user. -@item Performance: leverage Sleepycat performance and +@item @strong{Safety:} ACID, transactions. Concurrent with good multi-user (BDB) and +multi-threaded semantics (BDB/SQL), isolation, locking and deadlock +detection. (Deadlock detection does require an external process to be +launched for Berkeley DB) + +@item @strong{Performance:} leverage Berkeley DB performance and/or Relational database reliability. In addition to fast concurrent / transactional modes, -elephant will (eventually) offer an accellerated single-user as -well as in-memory modes that should be comparable to prevalence -style solutions, but leverage a common interface. +elephant will (eventually) offer an accelerated single-user as well as +pure in-memory mode that should be comparable to prevalence style +solutions, but employ a common programmer interface. -@item Historical continuity: Elephant does not try to innovate +@item @strong{Historical continuity:} Elephant does not try to innovate significantly over prior Lisp persistent object stores such as AllegroStore (also based on Berkeley DB), the new AllegroCache, the Symbolics system Statice and PLOB. Anyone familiar with those systems will recognize the Elephant interface. -@item License Flexibility: Elephant is released under the LLGPL. +@item @strong{License Flexibility:} Elephant is released under the LLGPL. Because it supports multiple implementation of the backend, one can choose a backend with licensing and other features appropriate to your needs. @end itemize -Join the Elephant mailing lists to ask your questions and -receive updates. Pointers can be found on the Elephant website at +@section More Information + +Join the Elephant mailing lists to ask your questions and receive +updates. You can also review archives for past discussions and +questions. Pointers can be found on the Elephant website at @uref{http://www.common-lisp.net/project/elephant}. -Installation documents can be found in the file @file{INSTALL}. -Opportunities to contribute can be found in the file @file{TODO}. +Installation instructions can be found in the @ref{Installation} +section. Bugs can be reported via the Elephant Trac system at + +@uref{http://trac.common-lisp.net/elephant/}. + +This also serves as a good starting point for finding out what new +features or capabilities you can contribute to Elephant. The Trac +system also contains a wiki with design discussions and a FAQ. --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2005/11/23 17:51:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 12:16:02 1.3 @@ -1,10 +1,15 @@ (require 'asdf) -(require 'elephant) -(load "../docstrings.lisp") +(asdf:operate 'asdf:load-op 'elephant-tests) +(sb-posix:chdir "/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/") +(load "/Users/eslick/Work/fsrc/elephant-cvs/doc/docstrings.lisp") (defun make-docs () ;; (when (check-complete) (when t - (sb-texinfo:generate-includes #p"includes" (find-package :ele)))) + (elephant:open-store elephant-tests::*testbdb-spec*) + (make-instance 'elephant::persistent-collection) + (make-instance 'elephant::secondary-cursor) + (make-instance 'elephant::indexed-btree) + (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" (find-package :elephant) (find-package :elephant-backend) (find-package 'elephant-memutil) (find-package 'elephant-system)))) (make-docs) --- /project/elephant/cvsroot/elephant/doc/package-elephant.texinfo 2004/09/19 17:44:42 1.1 +++ /project/elephant/cvsroot/elephant/doc/package-elephant.texinfo 2007/03/24 12:16:02 1.2 @@ -1,6 +1,5 @@ @anchor{Package elephant} @defvr {Package} elephant -Elephant: an object-oriented database for Common Lisp. -Uses the @code{sleepycat} package to talk to Berkeley @code{db} / -Sleepycat. +Elephant: an object-oriented database for Common Lisp with + multiple backends for Berkeley @code{db}, @code{sql} and others. @end defvr --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2006/05/15 13:02:26 1.5 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/24 12:16:02 1.6 @@ -1,52 +1,42 @@ @c -*-texinfo-*- -@node Reference +@node User API Reference @comment node-name, next, previous, up -@chapter Reference -@cindex Reference +@chapter User API Reference +@cindex User API Reference @cindex API Reference @menu -* Controller:: The connection to Sleepycat. -* Transactions:: Transactions. -* Persistent Objects:: CLOS persistence. -* Persistent Slot Indexing:: Convenient indexing. +* Store Controllers:: Connecting to a data store. +* Persistent Objects:: Creating and using persistent objects. +* Persistent Object Indexing:: Convenient indexing. +* Query Interfaces:: Finding instances. * Collections:: BTrees and indices. * Cursors:: Traversing BTrees. -* Sleepycat:: Some functions from the low-level Sleepycat interface. +* Transactions:: Transactions. +* Multithreading:: Multithreading. +* Migration and Upgrading:: Migration and upgrading. @end menu -@node Controller +@node Store Controllers @comment node-name, next, previous, up -@section Controller -@cindex Controller +@section Store Controllers +@cindex Store Controllers + +Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller. + +@include includes/class-elephant-store-controller.texinfo +@include includes/var-elephant-star-store-controller-star.texinfo @include includes/fun-elephant-open-store.texinfo @include includes/fun-elephant-close-store.texinfo @include includes/macro-elephant-with-open-store.texinfo -@include includes/fun-elephant-add-to-root.texinfo @include includes/fun-elephant-get-from-root.texinfo -@include includes/fun-elephant-run-elephant-thread.texinfo - -@include includes/var-elephant-star-store-controller-star.texinfo -@include includes/class-elephant-store-controller.texinfo -@include includes/fun-elephant-open-controller.texinfo -@include includes/fun-elephant-close-controller.texinfo -@include includes/macro-elephant-with-open-controller.texinfo - -@node Transactions -@comment node-name, next, previous, up -@section Transactions -@cindex Transactions - -@include includes/macro-elephant-with-transaction.texinfo - -@include includes/var-elephant-star-auto-commit-star.texinfo -@include includes/var-elephant-star-current-transaction-star.texinfo -@include includes/fun-elephant-start-ele-transaction.texinfo -@include includes/fun-elephant-commit-transaction.texinfo -@include includes/fun-elephant-abort-transaction.texinfo +@include includes/fun-elephant-add-to-root.texinfo +@include includes/fun-elephant-remove-from-root.texinfo +@include includes/fun-elephant-root-existsp.texinfo +@include includes/fun-elephant-map-root.texinfo @node Persistent Objects @comment node-name, next, previous, up @@ -57,10 +47,14 @@ @include includes/class-elephant-persistent.texinfo @include includes/class-elephant-persistent-object.texinfo -@node Persistent Slot Indexing +@include includes/macro-elephant-defpclass.texinfo + +@include includes/fun-elephant-drop-pobject.texinfo + +@node Persistent Object Indexing @comment node-name, next, previous, up -@section Persistent Slot Indexing -@cindex Persistent Slot Indexing +@section Persistent Object Indexing +@cindex Persistent Object Indexing @include includes/fun-get-instances-by-class.texinfo @include includes/fun-get-instance-by-value.texinfo @@ -74,6 +68,11 @@ @include includes/fun-add-class-derived-index.texinfo @include includes/fun-remove-class-derived-index.texinfo +@node Query Interfaces +@comment node-name, next, previous, up +@section Query Interfaces +@cindex Query Interfaces + @node Collections @comment node-name, next, previous, up @section Collections @@ -133,24 +132,25 @@ @include includes/fun-elephant-cursor-set-range.texinfo @include includes/fun-elephant-cursor-set.texinfo -@node Sleepycat +@node Transactions @comment node-name, next, previous, up -@section Sleepycat -@cindex Sleepycat +@section Transactions +@cindex Transactions -@include includes/macro-elephant-with-lock.texinfo -@include includes/fun-elephant-db-env-get-flags.texinfo -@include includes/fun-elephant-db-env-get-lock-detect.texinfo -@include includes/fun-elephant-db-env-get-timeout.texinfo -@include includes/fun-elephant-db-env-lock-get.texinfo -@include includes/fun-elephant-db-env-lock-id-free.texinfo -@include includes/fun-elephant-db-env-lock-id.texinfo -@include includes/fun-elephant-db-env-lock-put.texinfo -@include includes/fun-elephant-db-env-set-flags.texinfo -@include includes/fun-elephant-db-env-set-lock-detect.texinfo -@include includes/fun-elephant-db-env-set-timeout.texinfo -@include includes/fun-elephant-db-transaction-abort.texinfo -@include includes/fun-elephant-db-transaction-begin.texinfo -@include includes/fun-elephant-db-transaction-commit.texinfo -@include includes/fun-elephant-db-transaction-id.texinfo +@include includes/macro-elephant-with-transaction.texinfo + +@include includes/var-elephant-star-auto-commit-star.texinfo +@include includes/var-elephant-star-current-transaction-star.texinfo +@include includes/fun-elephant-start-ele-transaction.texinfo +@include includes/fun-elephant-commit-transaction.texinfo +@include includes/fun-elephant-abort-transaction.texinfo +@node Multithreading +@comment node-name, next, previous, up +@section Multithreading +@cindex Multithreading + +@node Migration and Upgrading +@comment node-name, next, previous, up +@section Migration and Upgrading +@cindex Migration and Upgrading --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2006/04/26 17:53:43 1.6 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/24 12:16:02 1.7 @@ -6,306 +6,348 @@ @cindex Tutorial @menu -* Preliminaries:: Some general remarks. -* Getting Started:: Accessing a store. -* Running the Tests:: Gaining confidence. -* The Root:: Staying alive. -* Serialization:: Lisp -> (char *). -* Persistent Classes:: CLOS the Elephant way. +* Overview:: Overview of elphant's features +* Getting Started:: Opening and accessing a store. +* The Store Root:: Accessing persistent data. +* Serialization:: Storage semantics for lisp values. +* Persistent Classes:: Persistent semantics for objects. * Rules about Persistent Classes:: What you need to know. -* Using Transactions:: Using ACID. -* Using BTrees:: Storing lots of things. -* Using Cursors:: Tranversing BTrees. -* Secondary Indices:: By any other name... -* Class Indices:: Speed and Convenience. -* The Store Controller:: Behind the curtain. -* Repository Migration:: How to move objects from one repository to another -* Threading:: Playing nice with others. -* Performance Tips:: Bogoflops for your buck. +* Persistent collections:: Keep track of collections of objects. +* Class Indices:: Simple way to keep track of instances. +* Using Transactions:: Enabling ACID database properties. @end menu -@node Preliminaries +@node Overview @comment node-name, next, previous, up -@section Preliminaries +@section Overview -Elephant is a Common Lisp OODB. It provides a partial solution to the -problem of making Lisp data persistent. It does this through two mechanisms: -a simple API for storing and retrieving lisp values from a persistent store, -and the ability to make CLOS class slot values be persistent. - -When someone says "database," most people think of SQL Relation Data Base -Management Systems (e.g. Oracle, Postgresql, MySql). Elephant can use either -RDBMSs or Berkeley DB (Sleepycat) as a backend repository, but relies on -LISP as its data manipulation system. Unlike systems such as Hibernate -for Java, the user does not need to construct or worry about a mapping -from the object space into the database. Elephant is designed to be a -simple and convenient tool for the programmer. - -Elephant supports easy migration of data between different repositories and -different backends, allowing the user to choose which repository backend they -will use at a particular point in time. - -Berkeley DB/Sleepycat is a database library that was the initial inspiration for Elephant's -design and is well-matched to Elephant's data model. BDB is implemented as a C library, -not a client/server model, so access can be very fast. Berkeley DB is also quite mature, -robust and has many features, such as transactions and replication. While we hope -that you won't need to understand a specific backend to use Elephant, reading the -docs will certainly help you when things go wrong. For the Sleepycat backend, -they can be found at @uref{http://www.sleepycat.com}. +Elephant is a Persistence Metaprotocol and Database for Common Lisp. +It provides the ability for users to define and interact with +persistent objects and to transparently store ordinary lisp values. +Persistent objects are CLOS instances that overload the ordinary slot +access semantics so that every write to a slot is passed through and +written to disk. Non-persistent lisp objects and values can be +written to slots and will be automatically persisted. In addition, +Elephant provides a persistent index which maintains an ordered +collection of lisp values or persistent object references. + +When someone says "database," most people think of SQL Relational Data +Base Management Systems (e.g. Oracle, Postgresql, MySql). Those +systems store data in statically typed tables with unique shared +values to connect rows in separate tables. Objects can be mapped into +these tables in an object-relational mapping that assigns objects to +rows and slot values to columns in a row's table. If a slot +references another type of object, a unique ID can be used to +reference that object's table. CL-SQL, for example, provides +facilities for this kind of object-relational mapping and there are +many systems for other languages that do the same (i.e. Hibernate for +Java). + +While Elephant can use either RDBMSs or Berkeley DB as a data store, +the model it supports is that of objects stored in persistent indices. +Unlike systems such as Hibernate for Java, the user does not need to +construct or worry about a mapping from the object space into the +database. Elephant relies on LISP rather than SQL for its data +manipulation language. Elephant is designed to be a simple and +convenient tool for the programmer. + +Elephant consists of a small universe of basic concepts: + +@itemize +@item @strong{Store controller:} the interface between lisp and a data store. +Most operations require or accept a store controller, or a default +store controller stored in @code{*store-controller*} to function. +@item @strong{BTrees:} Elephant provides a persistent key-value +abstraction based on the BTree data structure. Values can be written +to or read from a BTree and are stored in a sorted order. +@item @strong{Values:} most lisp values, including standard objects, arrays, etc +can be used as either key or value in a persistent BTree. +@item @strong{Persistent objects:} An object where most slot values are stored in +the data store and are written to or retrieved from disk on slot +accesses. Any value that can be written to an index can be written to +a persistent slot. +@item @strong{Transactions:} a dynamic context for executing operations on persistent +objects or BTrees that ensures that a set of changes is made atomically. +@item @strong{BTree indices:} A BTree index is a BTree that stores +an alternative ordering of the elements in a reference BTree. +@end itemize + +There are a set of more advanced concepts you will learn about later, +but these basic concepts will serve to acquaint you with Elephant. + +If you do not already have Elephant installed and building correctly, +read the @ref{Installation} section of this manual and then move on to +@ref{Getting Started}. -Elephant can also use RDBMS backends via the excellent CL-SQL package. -It has been tested with Postgres and SQLite3, and can probably easily work with others. - -@node Running the Tests +@node Getting Started @comment node-name, next, previous, up -@section Running the Tests - -There are three files in the directory @code{tests} that make running -the automated tests particularly easy. @code{BerkeleyDB-tests.lisp} is -for running against the BerkeleyDB backend, and @code{SQLDB-tests.lisp} is -for running agains the CL-SQL backend. @code{MigrationTests.lisp} is -for testing data migration functions, and can be used with either or both backends. - -The normal way to execute the tests, following the instruction in the file -@code{INSTALL}, is to open a listener and execute the lines found in -one of these files, such as: -@lisp -(asdf:operate 'asdf:load-op :elephant-tests) +@section Getting Started -(in-package "ELEPHANT-TESTS") +The first step in using elephant is to open a store controller. A +store controller is an object that coordinates lisp program access +to the chosen data store. -(setf *default-spec* *testbdb-spec*) +To obtain a store controller, you call @code{open-store} with a store +specification. A store specification is a list containing a backend +specifier (@code{:BDB} or @code{:CLSQL}) and a backend-specific +reference. -(do-backend-tests) -@end lisp +For :BDB, the second element is a string or pathname that references a +local directory for the database files. This directory must be +created prior to calling open-store. -The SQL test file differs only in using a different ``controller spec'': -@lisp(setf *default-spec* *testpg-spec*) -@end lisp -These default parameters are set in @file{tests/elephant-tests.lisp}, -they will looks something like this in a default distribution: -@lisp -(:BDB "/home/read/projects/sql-back-end/elephant/tests/testdb/") -@end lisp -and for postgres: @lisp -(:CLSQL (:POSTGRESQL "localhost.localdomain" "test" "postgres" "")) +(open-store '(:BDB ``/users/me/db/my-db/'')) @end lisp +For :CLSQL the second argument is another list consisting of a +specific SQL database and the name of a database file or connection +record to the SQL server. Examples are: -@node Getting Started -@comment node-name, next, previous, up -@section Getting Started - -In order to use Elephant, you have to have an open store controller. -To obtain an open store controller you call @code{open-store} - -The chapter ``SQL back-end'' has information about setting up a -SQL based backend; this tutorial will assume that you are using -Berkeley-DB as a backend. - -Make a directory to put your database store in. (This is called the -environment in Sleepycat terminology.) That's all you need to set up -your store! We'll assume in this tutorial you created a folder -@code{testdb} in the current directory. - -It is strongly recommended that you run the automated tests @xref{Running the Tests} that -come with Elephant before you begin this tutorial; this takes less -than five minutes and if will give you both confidence and clarity -and your continued work. Since the default distribution comes -with a directory structure set up, this is actually the easiest -way to get started with Elephant before beginning this tutorial. -If the tests fail for you, the Elephant developers will help you -solve the problem, but will want to know the outcome of the tests -as a starting point. - -If you have run the tests successfully, you can just do: @lisp -(open-store *default-spec*) +(open-store '(:CLSQL (:SQLITE "/users/me/db/sqlite.db"))) +(open-store '(:CLSQL (:POSTGRESQL "localhost.localdomain" "mydb" "myuser" "")))) @end lisp -But if not you might have to set up your own controller specifier like this: + +We use Berkeley DB as our example backend. To open a BDB +store-controller we can do the following: + @lisp (asdf:operate 'asdf:load-op :elephant) -(use-package "ELE") -(setf *testbdb-spec* -'(:BDB "/home/read/projects/sql-back-end/elephant/tests/testdb/")) -(open-store *testbdb-spec*) -@end lisp +(use-package :elephant) +(setf *test-db-spec* + '(:BDB "/home/me/db/testdb/")) +(open-store *test-db-spec*) +@end lisp + +We do not need to store the reference to the store just now as it is +automatically assigned to the variable, @code{*store-controller*}. +For a deeper discussion of store controller management see the +@ref{User Guide}. + +When you're done with your session, release the store-controller's +resources by calling @code{close-store}. + +Also there is a convenience macro @code{with-open-store} that will +open and close the store, but opening the store is an expensive +operation so it is generally better to leave the store open until your +application no longer needs it. + +@node The Store Root +@comment node-name, next, previous, up +@section The Store Root + +What values live between lisp sessions is called @emph{liveness}. +Liveness in a store is determined by whether the value can be reached +from the root of the store. The root is a special BTree in which +other BTrees and lisp values can be stored. This BTree has a special +interface through the store controller. (There is a second root BTree +called the class root which will be discussed later.) -When you're done with your session, don't forget to +You can put something into the root object by @lisp -* (close-store) -=> NIL +(add-to-root "my key" "my value") +=> "my value" @end lisp -Also there is a convenience macro @code{with-open-store}. +and get things out via -@node The Root -@comment node-name, next, previous, up -@section The Root +@lisp +(get-from-root "my key") +=> "my value" +=> T +@end lisp -Liveness in a store is determined by reachability from the root -object. Technically, liveness also applies to indexed -classes, as described in @xref{Class Indices}, which live in a -separate class-root namespace. When garbage collection is -implemented, dead objects will be collected on gc's.) The root and -class-root objects are BTrees, effectively a table with sorted keys -and log(N) access time. @xref{Using BTrees}. +The second value indicates whether the key was found. This is +important if your key-value pair can have nil as a value. -You can put something into the root object by +You can perform other basic operations as well. @lisp -* (add-to-root "my key" "my value") +(root-existsp "my key") +=> T +(remove-from-root "my key") +=> T +(get-from-root "my key") +=> NIL => NIL @end lisp -and get things out via +To access all the objects in the root, the simplest way is to +simply call @code{map-root} with a function to apply to each +key-value pair. @lisp -* (get-from-root "my key") -=> "my value" -=> T -@end lisp +(map-root + (lambda (k v) + (format t "key: ~A value:~A~%" k v))) +@end lisp -The root object is available as +You can also access the root object directly. @lisp -* (controller-root *store-controller*) -=> #<BTREE @{492AE37D@}> +(controller-root *store-controller*) +=> #<DB-BDB::BDB-BTREE @ #x10e86042> @end lisp -It is an instance of a class "btree"; @xref{Using BTrees}. +It is an instance of a class "btree"; @pxref{Using BTrees}. @node Serialization @comment node-name, next, previous, up @section Serialization -What can you put into the store? An ever-growing list of things: -numbers (except for complexes, which will be easy to support), +What can you put into the store besides strings? Almost all lisp +values and objects can be stored: numbers (except for complexes), symbols, strings, nil, characters, pathnames, conses, hash-tables, -arrays, CLOS objects. Nested and circular things are allowed. You -can store basically anything except lambdas, closures, structures, -packages and streams. (These may eventually get supported too.) - -Unfortunately Berekely DB doesn't understand Lisp, so Lisp data needs -to be serialized to enter the database (converted to byte arrays), and -deserialized to be read. This introduces some caveats (not unique to -Elephant!): +arrays, CLOS objects and structs. Nested and circular things are +allowed. You can store basically anything except lambdas, closures, +class objects, packages and streams. (These may eventually get +supported too.) + +Elephant needs to use a representation of data that is independant of +a specific lisp or data store. Therefore all lisp values that are +stored must be @emph{serialized} into a canonical format. Because +Berkeley DB supports variable length binary buffers, Elephant uses a +binary serialization system. This process has some important +consequences that it is very important to understand: @enumerate -@item Lisp identity can't be preserved. Since this is a store which +@item @strong{Lisp identity can't be preserved}. Since this is a store which persists across invocations of Lisp, this probably doesn't even make -sense. +sense. However if you get an object from the index, store it to a +lisp variable, then get it again - they will not be eq: @lisp -* (setq foo (cons nil nil)) +(setq foo (cons nil nil)) => (NIL) -* (add-to-root "my key" foo) -=> NIL -* (add-to-root "my other key" foo) -=> NIL -* (eq (get-from-root "my key") +(add-to-root "my key" foo) +=> (NIL) +(add-to-root "my other key" foo) +=> (NIL) +(eq (get-from-root "my key") (get-from-root "my other key")) => NIL @end lisp -@item Mutated substructure does not persist +@item @strong{Nested aggregates are stored in one buffer}. +If you store an set of objects in a hash table you try to store a hash +table, all of those objects will get stored in one large binary buffer +with the hash keys. This is true for all other aggregates that can +store type T (cons, array, standard object, etc). + +@item @strong{Mutated substructure does not persist}. @lisp -* (setf (car foo) T) +(setf (car foo) T) => T -* (get-from-root "my key") +(get-from-root "my key") => (NIL) @end lisp This will affect all aggregate types: objects, conses, hash-tables, et cetera. (You can of course manually re-store the cons.) In this sense elephant does not automatically provide persistent collections. If you [871 lines skipped]

1 0