Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv6422/doc

Modified Files: Makefile copying.texinfo elephant.texinfo installation.texinfo intro.texinfo make-ref.lisp package-elephant.texinfo reference.texinfo tutorial.texinfo Log Message: Cleanup indexing tests so we always have a clean slate

--- /project/elephant/cvsroot/elephant/doc/Makefile 2005/11/23 17:51:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 12:16:02 1.3 @@ -5,4 +5,4 @@ makeinfo -v --html --force elephant.texinfo

includes-stuff: - cd includes; lisp < ../make-ref.lisp + cd includes; sbcl < ../make-ref.lisp --- /project/elephant/cvsroot/elephant/doc/copying.texinfo 2006/03/02 14:44:49 1.2 +++ /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/03/24 12:16:02 1.3 @@ -15,15 +15,13 @@ as governed by the terms of the Lisp Lesser GNU Public License @uref{http://opensource.franz.com/preamble.html%7D, also known as the LLGPL.

- - Copyrights include:

Copyright (c) 2004 by Andrew Blumberg and Ben Lee

-Copyright (c) 2006 by Ian Eslick +Copyright (c) 2006-2007 by Ian Eslick

-Copyright (c) 2005,2006 by Robert L. Read +Copyright (c) 2005-2007 by Robert L. Read

Portions of this program (namely the C unicode string --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2006/03/02 14:44:49 1.4 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/24 12:16:02 1.5 @@ -6,7 +6,7 @@

@copying Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. -Copyright @copyright{} 2006 Robert L. Read. +Copyright @copyright{} 2006-2007 Robert L. Read and Ian Eslick

@quotation Permission is granted to copy, distribute and/or modify this document @@ -19,8 +19,8 @@

@titlepage @title Elephant User Manual -@subtitle Elephant version 0.6 -@author Ben Lee +@subtitle Elephant version 0.6.1 +@author Ben Lee and Ian Eslick

@c The following two commands @c start the copyright page. @@ -29,24 +29,35 @@ @insertcopying @end titlepage

-@c So the toc is printed at the start. -@contents - @ifnottex @node Top @comment node-name, next, previous, up -@top Elephant +@top Copyright

@insertcopying @end ifnottex

@menu -* Introduction:: Introducing Elephant! -* Tutorial:: A leisurely walk-through. -* Reference:: API documentation. -* Installation:: Installation and test-suite procedures and issues -* Design Notes:: Internals. -* Copying:: Your rights and freedoms. +* Table of Contents:: +@end menu + +@chapheading Chapters + +@menu +* Introduction:: Introduction to the Elephant Persistent Object System. +* Tutorial:: A basic ``getting started'' tutorial. +* Installation:: Installation and test-suite procedures. +* User Guide:: In depth discussion of all Elephant facilities and features. +* Usage scenarios:: Design scenarios for Elephant applications. +* User API Reference:: Function and class documentation of the user API. +* Elephant Design:: An overview of elephant's internal architecture. +* Data Store API Reference:: Function level documentation for data store implementors. +* Copying:: Your rights and freedoms. +@end menu + +@chapheading Appendices + +@menu * Concept Index:: * Object Index:: * Function / Macro Index:: @@ -54,11 +65,18 @@ * Colophon:: @end menu

+@node Table of Contents +@comment node-name, next, previous, up +@contents + @include intro.texinfo @include tutorial.texinfo -@include reference.texinfo -@include notes.texinfo @include installation.texinfo +@include user-guide.texinfo +@include scenarios.texinfo +@include reference.texinfo +@include elephant-design.texinfo +@include data-store-reference.texinfo @include copying.texinfo

@node Concept Index --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2006/05/15 13:02:26 1.3 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/24 12:16:02 1.4 @@ -8,10 +8,11 @@ @menu * Installation Basics:: Basic installation * Test-Suites:: Running the test suites -* SQL-Introduction:: The design and status of the SQL back-end extention. -* Extension Status:: The current status of the SQL back-end extention. -* Multi-repository Operation:: Specifying repositories -* Setting up PostGres:: An example +* Berkeley DB Introduction:: The Berkeley DB backend +* SQL Data Store:: The design and status of the SQL back-end extension. +* Lisp Data Store:: A native lisp-based repository. +* Multi-repository Operation:: Specifying repositories. +* Setting up PostGres:: An example. @end menu

@node Installation Basics @@ -148,11 +149,14 @@ If you get errors, you may wish to report it the @code{ elephant-devel at common-lisp.net} email list.

+@node Berkeley DB Repository +@comment node-name, next, previous, up +@section Berkeley DB Repository

-@node SQL-Introduction +@node SQL Repository @comment node-name, next, previous, up -@section SQL-Introduction +@section SQL Repository

Although originally designed as an interface to the BerkeleyDB system, the original Elephant system has been experimenetally extended to --- /project/elephant/cvsroot/elephant/doc/intro.texinfo 2006/04/26 17:53:43 1.4 +++ /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/03/24 12:16:02 1.5 @@ -5,66 +5,96 @@ @chapter Introduction @cindex Introduction

-Elephant is a persistent object database for Common Lisp that -supports storing CLOS objects and most lisp primitives. -It supports persistent collections via a BTree interface. - -Elephant was originally developed as an interface layer on top -of the Sleepycat / Berkeley DB library, a widely-distributed -embedded database. Many unix systems have it installed by default. -Berkeley DB is ACID compliant, transactional, process and -thread safe, and fast relative to relational databases. Recently, -Elephant was extended to provide support for relational database backends. -It has been tested with Postgres and SQLite 3. It supports, with some -care, simultaneous multi-repository operation and enables convenient -migration of data between repositories. +Elephant is a persistent object protocol and database for Common +Lisp. The persistent protocol component of elephant overrides class +creation and standard slot accesses using the Meta-Object Protocol +(MOP) to render slot values persistent. Database functionality +includes the ability to persistently index and retrieve ordered sets +of class instances and ordinary lisp values. Elephant has an +extensive test suite and the core functionality is becoming quite +mature. + +The Elephant code base is available under the LLGPL license. Data +stores each come with their own, separate license and you will have to +evaluate the implications of using them yourself. + +@section History + +Elephant was originally envisioned as a lightweight interface layer on +top of the Berkeley DB library, a widely-distributed embedded database +that many unix systems have installed by default. Berkeley DB is ACID +compliant, transactional, process and thread safe, and fast relative +to relational databases. + +Elephant has been extended to provide support for multiple backends, +specifically a relational database backend based on CL-SQL which has +been tested with Postgres and SQLite 3. It supports, with some care, +multi-repository operation and enables convenient migration of data +between repositories.

The support for relational backends and migration to the LLGPL was to allow for broader use of Elephant in both not-for-profit and commercial -settings. +settings. Several additional backends are planned for future releases +including a native Lisp implementation released under the LLGPL.

-Elephant goals: +Elephant's current development focus is to enhance the feature set +including a native lisp backend, a simple query language, and flexible +persistence models that selectively break one or more of the ACID +constraints to improve performance. + +@section Elephant Goals

@itemize -@item Transparency: most Lisp values are easy to persist without -signifcant effort or special syntax. Talk to the DB entirely from Lisp; -not requirement for domain-specific languages (such as SQL) to access persistent -resources. Enable interactive control of the database with no external -server dependencies. - -@item Safety: ACID, transactions. Concurrent with good multi-user and -multi-threaded semantics, isolation, locking and deadlock detection. -(Deadlock detection does require an external process to be launched) +@item @strong{Transparency:} most Lisp values are easy to persist without +significant effort or special syntax. You can interact with the DB +entirely from Lisp. There is no requirement to use domain-specific +languages, such as SQL, to access persistent resources. Elephant +loads via ASDF and requires no external server (except for some SQL +backends like Postgres).

-@item Simplicity: a small library with few surprises for the +@item @strong{Simplicity:} a small library with few surprises for the programmer. Lisp and Berkeley DB together are an excellent substrate; -Elephant tries to leverage their features as much as possible. -Support for multiple backends should be load-time options and mostly +Elephant tries to leverage their features as much as possible. +Support for additional backends are load-time options and more or less transparent to the user.

-@item Performance: leverage Sleepycat performance and +@item @strong{Safety:} ACID, transactions. Concurrent with good multi-user (BDB) and +multi-threaded semantics (BDB/SQL), isolation, locking and deadlock +detection. (Deadlock detection does require an external process to be +launched for Berkeley DB) + +@item @strong{Performance:} leverage Berkeley DB performance and/or Relational database reliability. In addition to fast concurrent / transactional modes, -elephant will (eventually) offer an accellerated single-user as -well as in-memory modes that should be comparable to prevalence -style solutions, but leverage a common interface. +elephant will (eventually) offer an accelerated single-user as well as +pure in-memory mode that should be comparable to prevalence style +solutions, but employ a common programmer interface.

-@item Historical continuity: Elephant does not try to innovate +@item @strong{Historical continuity:} Elephant does not try to innovate significantly over prior Lisp persistent object stores such as AllegroStore (also based on Berkeley DB), the new AllegroCache, the Symbolics system Statice and PLOB. Anyone familiar with those systems will recognize the Elephant interface.

-@item License Flexibility: Elephant is released under the LLGPL. +@item @strong{License Flexibility:} Elephant is released under the LLGPL. Because it supports multiple implementation of the backend, one can choose a backend with licensing and other features appropriate to your needs.

@end itemize

-Join the Elephant mailing lists to ask your questions and -receive updates. Pointers can be found on the Elephant website at +@section More Information + +Join the Elephant mailing lists to ask your questions and receive +updates. You can also review archives for past discussions and +questions. Pointers can be found on the Elephant website at

@uref{http://www.common-lisp.net/project/elephant%7D.

-Installation documents can be found in the file @file{INSTALL}. -Opportunities to contribute can be found in the file @file{TODO}. +Installation instructions can be found in the @ref{Installation} +section. Bugs can be reported via the Elephant Trac system at + +@uref{http://trac.common-lisp.net/elephant/%7D. + +This also serves as a good starting point for finding out what new +features or capabilities you can contribute to Elephant. The Trac +system also contains a wiki with design discussions and a FAQ. --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2005/11/23 17:51:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 12:16:02 1.3 @@ -1,10 +1,15 @@ (require 'asdf) -(require 'elephant) -(load "../docstrings.lisp") +(asdf:operate 'asdf:load-op 'elephant-tests) +(sb-posix:chdir "/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/") +(load "/Users/eslick/Work/fsrc/elephant-cvs/doc/docstrings.lisp")

(defun make-docs () ;; (when (check-complete) (when t - (sb-texinfo:generate-includes #p"includes" (find-package :ele)))) + (elephant:open-store elephant-tests::*testbdb-spec*) + (make-instance 'elephant::persistent-collection) + (make-instance 'elephant::secondary-cursor) + (make-instance 'elephant::indexed-btree) + (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" (find-package :elephant) (find-package :elephant-backend) (find-package 'elephant-memutil) (find-package 'elephant-system))))

(make-docs) --- /project/elephant/cvsroot/elephant/doc/package-elephant.texinfo 2004/09/19 17:44:42 1.1 +++ /project/elephant/cvsroot/elephant/doc/package-elephant.texinfo 2007/03/24 12:16:02 1.2 @@ -1,6 +1,5 @@ @anchor{Package elephant} @defvr {Package} elephant -Elephant: an object-oriented database for Common Lisp. -Uses the @code{sleepycat} package to talk to Berkeley @code{db} / -Sleepycat. +Elephant: an object-oriented database for Common Lisp with + multiple backends for Berkeley @code{db}, @code{sql} and others. @end defvr --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2006/05/15 13:02:26 1.5 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/24 12:16:02 1.6 @@ -1,52 +1,42 @@ @c -*-texinfo-*-

-@node Reference +@node User API Reference @comment node-name, next, previous, up -@chapter Reference -@cindex Reference +@chapter User API Reference +@cindex User API Reference @cindex API Reference

@menu -* Controller:: The connection to Sleepycat. -* Transactions:: Transactions. -* Persistent Objects:: CLOS persistence. -* Persistent Slot Indexing:: Convenient indexing. +* Store Controllers:: Connecting to a data store. +* Persistent Objects:: Creating and using persistent objects. +* Persistent Object Indexing:: Convenient indexing. +* Query Interfaces:: Finding instances. * Collections:: BTrees and indices. * Cursors:: Traversing BTrees. -* Sleepycat:: Some functions from the low-level Sleepycat interface. +* Transactions:: Transactions. +* Multithreading:: Multithreading. +* Migration and Upgrading:: Migration and upgrading. @end menu

-@node Controller +@node Store Controllers @comment node-name, next, previous, up -@section Controller -@cindex Controller +@section Store Controllers +@cindex Store Controllers + +Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller. + +@include includes/class-elephant-store-controller.texinfo +@include includes/var-elephant-star-store-controller-star.texinfo

@include includes/fun-elephant-open-store.texinfo @include includes/fun-elephant-close-store.texinfo @include includes/macro-elephant-with-open-store.texinfo

-@include includes/fun-elephant-add-to-root.texinfo @include includes/fun-elephant-get-from-root.texinfo -@include includes/fun-elephant-run-elephant-thread.texinfo - -@include includes/var-elephant-star-store-controller-star.texinfo -@include includes/class-elephant-store-controller.texinfo -@include includes/fun-elephant-open-controller.texinfo -@include includes/fun-elephant-close-controller.texinfo -@include includes/macro-elephant-with-open-controller.texinfo - -@node Transactions -@comment node-name, next, previous, up -@section Transactions -@cindex Transactions - -@include includes/macro-elephant-with-transaction.texinfo - -@include includes/var-elephant-star-auto-commit-star.texinfo -@include includes/var-elephant-star-current-transaction-star.texinfo -@include includes/fun-elephant-start-ele-transaction.texinfo -@include includes/fun-elephant-commit-transaction.texinfo -@include includes/fun-elephant-abort-transaction.texinfo +@include includes/fun-elephant-add-to-root.texinfo +@include includes/fun-elephant-remove-from-root.texinfo +@include includes/fun-elephant-root-existsp.texinfo +@include includes/fun-elephant-map-root.texinfo

@node Persistent Objects @comment node-name, next, previous, up @@ -57,10 +47,14 @@ @include includes/class-elephant-persistent.texinfo @include includes/class-elephant-persistent-object.texinfo

-@node Persistent Slot Indexing +@include includes/macro-elephant-defpclass.texinfo + +@include includes/fun-elephant-drop-pobject.texinfo + +@node Persistent Object Indexing @comment node-name, next, previous, up -@section Persistent Slot Indexing -@cindex Persistent Slot Indexing +@section Persistent Object Indexing +@cindex Persistent Object Indexing

@include includes/fun-get-instances-by-class.texinfo @include includes/fun-get-instance-by-value.texinfo @@ -74,6 +68,11 @@ @include includes/fun-add-class-derived-index.texinfo @include includes/fun-remove-class-derived-index.texinfo

+@node Query Interfaces +@comment node-name, next, previous, up +@section Query Interfaces +@cindex Query Interfaces + @node Collections @comment node-name, next, previous, up @section Collections @@ -133,24 +132,25 @@ @include includes/fun-elephant-cursor-set-range.texinfo @include includes/fun-elephant-cursor-set.texinfo

-@node Sleepycat +@node Transactions @comment node-name, next, previous, up -@section Sleepycat -@cindex Sleepycat +@section Transactions +@cindex Transactions

-@include includes/macro-elephant-with-lock.texinfo -@include includes/fun-elephant-db-env-get-flags.texinfo -@include includes/fun-elephant-db-env-get-lock-detect.texinfo -@include includes/fun-elephant-db-env-get-timeout.texinfo -@include includes/fun-elephant-db-env-lock-get.texinfo -@include includes/fun-elephant-db-env-lock-id-free.texinfo -@include includes/fun-elephant-db-env-lock-id.texinfo -@include includes/fun-elephant-db-env-lock-put.texinfo -@include includes/fun-elephant-db-env-set-flags.texinfo -@include includes/fun-elephant-db-env-set-lock-detect.texinfo -@include includes/fun-elephant-db-env-set-timeout.texinfo -@include includes/fun-elephant-db-transaction-abort.texinfo -@include includes/fun-elephant-db-transaction-begin.texinfo -@include includes/fun-elephant-db-transaction-commit.texinfo -@include includes/fun-elephant-db-transaction-id.texinfo +@include includes/macro-elephant-with-transaction.texinfo + +@include includes/var-elephant-star-auto-commit-star.texinfo +@include includes/var-elephant-star-current-transaction-star.texinfo +@include includes/fun-elephant-start-ele-transaction.texinfo +@include includes/fun-elephant-commit-transaction.texinfo +@include includes/fun-elephant-abort-transaction.texinfo

+@node Multithreading +@comment node-name, next, previous, up +@section Multithreading +@cindex Multithreading + +@node Migration and Upgrading +@comment node-name, next, previous, up +@section Migration and Upgrading +@cindex Migration and Upgrading --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2006/04/26 17:53:43 1.6 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/24 12:16:02 1.7 @@ -6,306 +6,348 @@ @cindex Tutorial

@menu -* Preliminaries:: Some general remarks. -* Getting Started:: Accessing a store. -* Running the Tests:: Gaining confidence. -* The Root:: Staying alive. -* Serialization:: Lisp -> (char *). -* Persistent Classes:: CLOS the Elephant way. +* Overview:: Overview of elphant's features +* Getting Started:: Opening and accessing a store. +* The Store Root:: Accessing persistent data. +* Serialization:: Storage semantics for lisp values. +* Persistent Classes:: Persistent semantics for objects. * Rules about Persistent Classes:: What you need to know. -* Using Transactions:: Using ACID. -* Using BTrees:: Storing lots of things. -* Using Cursors:: Tranversing BTrees. -* Secondary Indices:: By any other name... -* Class Indices:: Speed and Convenience. -* The Store Controller:: Behind the curtain. -* Repository Migration:: How to move objects from one repository to another -* Threading:: Playing nice with others. -* Performance Tips:: Bogoflops for your buck. +* Persistent collections:: Keep track of collections of objects. +* Class Indices:: Simple way to keep track of instances. +* Using Transactions:: Enabling ACID database properties. @end menu

-@node Preliminaries +@node Overview @comment node-name, next, previous, up -@section Preliminaries +@section Overview

-Elephant is a Common Lisp OODB. It provides a partial solution to the -problem of making Lisp data persistent. It does this through two mechanisms: -a simple API for storing and retrieving lisp values from a persistent store, -and the ability to make CLOS class slot values be persistent. - -When someone says "database," most people think of SQL Relation Data Base -Management Systems (e.g. Oracle, Postgresql, MySql). Elephant can use either -RDBMSs or Berkeley DB (Sleepycat) as a backend repository, but relies on -LISP as its data manipulation system. Unlike systems such as Hibernate -for Java, the user does not need to construct or worry about a mapping -from the object space into the database. Elephant is designed to be a -simple and convenient tool for the programmer. - -Elephant supports easy migration of data between different repositories and -different backends, allowing the user to choose which repository backend they -will use at a particular point in time. - -Berkeley DB/Sleepycat is a database library that was the initial inspiration for Elephant's -design and is well-matched to Elephant's data model. BDB is implemented as a C library, -not a client/server model, so access can be very fast. Berkeley DB is also quite mature, -robust and has many features, such as transactions and replication. While we hope -that you won't need to understand a specific backend to use Elephant, reading the -docs will certainly help you when things go wrong. For the Sleepycat backend, -they can be found at @uref{http://www.sleepycat.com%7D. +Elephant is a Persistence Metaprotocol and Database for Common Lisp. +It provides the ability for users to define and interact with +persistent objects and to transparently store ordinary lisp values. +Persistent objects are CLOS instances that overload the ordinary slot +access semantics so that every write to a slot is passed through and +written to disk. Non-persistent lisp objects and values can be +written to slots and will be automatically persisted. In addition, +Elephant provides a persistent index which maintains an ordered +collection of lisp values or persistent object references. + +When someone says "database," most people think of SQL Relational Data +Base Management Systems (e.g. Oracle, Postgresql, MySql). Those +systems store data in statically typed tables with unique shared +values to connect rows in separate tables. Objects can be mapped into +these tables in an object-relational mapping that assigns objects to +rows and slot values to columns in a row's table. If a slot +references another type of object, a unique ID can be used to +reference that object's table. CL-SQL, for example, provides +facilities for this kind of object-relational mapping and there are +many systems for other languages that do the same (i.e. Hibernate for +Java). + +While Elephant can use either RDBMSs or Berkeley DB as a data store, +the model it supports is that of objects stored in persistent indices. +Unlike systems such as Hibernate for Java, the user does not need to +construct or worry about a mapping from the object space into the +database. Elephant relies on LISP rather than SQL for its data +manipulation language. Elephant is designed to be a simple and +convenient tool for the programmer. + +Elephant consists of a small universe of basic concepts: + +@itemize +@item @strong{Store controller:} the interface between lisp and a data store. +Most operations require or accept a store controller, or a default +store controller stored in @code{*store-controller*} to function. +@item @strong{BTrees:} Elephant provides a persistent key-value +abstraction based on the BTree data structure. Values can be written +to or read from a BTree and are stored in a sorted order. +@item @strong{Values:} most lisp values, including standard objects, arrays, etc +can be used as either key or value in a persistent BTree. +@item @strong{Persistent objects:} An object where most slot values are stored in +the data store and are written to or retrieved from disk on slot +accesses. Any value that can be written to an index can be written to +a persistent slot. +@item @strong{Transactions:} a dynamic context for executing operations on persistent +objects or BTrees that ensures that a set of changes is made atomically. +@item @strong{BTree indices:} A BTree index is a BTree that stores +an alternative ordering of the elements in a reference BTree. +@end itemize + +There are a set of more advanced concepts you will learn about later, +but these basic concepts will serve to acquaint you with Elephant. + +If you do not already have Elephant installed and building correctly, +read the @ref{Installation} section of this manual and then move on to +@ref{Getting Started}.

-Elephant can also use RDBMS backends via the excellent CL-SQL package. -It has been tested with Postgres and SQLite3, and can probably easily work with others. - -@node Running the Tests +@node Getting Started @comment node-name, next, previous, up -@section Running the Tests - -There are three files in the directory @code{tests} that make running -the automated tests particularly easy. @code{BerkeleyDB-tests.lisp} is -for running against the BerkeleyDB backend, and @code{SQLDB-tests.lisp} is -for running agains the CL-SQL backend. @code{MigrationTests.lisp} is -for testing data migration functions, and can be used with either or both backends. - -The normal way to execute the tests, following the instruction in the file -@code{INSTALL}, is to open a listener and execute the lines found in -one of these files, such as: -@lisp -(asdf:operate 'asdf:load-op :elephant-tests) +@section Getting Started

-(in-package "ELEPHANT-TESTS") +The first step in using elephant is to open a store controller. A +store controller is an object that coordinates lisp program access +to the chosen data store.

-(setf *default-spec* *testbdb-spec*) +To obtain a store controller, you call @code{open-store} with a store +specification. A store specification is a list containing a backend +specifier (@code{:BDB} or @code{:CLSQL}) and a backend-specific +reference.

-(do-backend-tests) -@end lisp +For :BDB, the second element is a string or pathname that references a +local directory for the database files. This directory must be +created prior to calling open-store.

-The SQL test file differs only in using a different ``controller spec'': -@lisp(setf *default-spec* *testpg-spec*) -@end lisp -These default parameters are set in @file{tests/elephant-tests.lisp}, -they will looks something like this in a default distribution: -@lisp -(:BDB "/home/read/projects/sql-back-end/elephant/tests/testdb/") -@end lisp -and for postgres: @lisp -(:CLSQL (:POSTGRESQL "localhost.localdomain" "test" "postgres" "")) +(open-store '(:BDB ``/users/me/db/my-db/'')) @end lisp

+For :CLSQL the second argument is another list consisting of a +specific SQL database and the name of a database file or connection +record to the SQL server. Examples are:

-@node Getting Started -@comment node-name, next, previous, up -@section Getting Started - -In order to use Elephant, you have to have an open store controller. -To obtain an open store controller you call @code{open-store} - -The chapter ``SQL back-end'' has information about setting up a -SQL based backend; this tutorial will assume that you are using -Berkeley-DB as a backend. - -Make a directory to put your database store in. (This is called the -environment in Sleepycat terminology.) That's all you need to set up -your store! We'll assume in this tutorial you created a folder -@code{testdb} in the current directory. - -It is strongly recommended that you run the automated tests @xref{Running the Tests} that -come with Elephant before you begin this tutorial; this takes less -than five minutes and if will give you both confidence and clarity -and your continued work. Since the default distribution comes -with a directory structure set up, this is actually the easiest -way to get started with Elephant before beginning this tutorial. -If the tests fail for you, the Elephant developers will help you -solve the problem, but will want to know the outcome of the tests -as a starting point. - -If you have run the tests successfully, you can just do: @lisp -(open-store *default-spec*) +(open-store '(:CLSQL (:SQLITE "/users/me/db/sqlite.db"))) +(open-store '(:CLSQL (:POSTGRESQL "localhost.localdomain" "mydb" "myuser" "")))) @end lisp -But if not you might have to set up your own controller specifier like this: + +We use Berkeley DB as our example backend. To open a BDB +store-controller we can do the following: + @lisp (asdf:operate 'asdf:load-op :elephant) -(use-package "ELE") -(setf *testbdb-spec* -'(:BDB "/home/read/projects/sql-back-end/elephant/tests/testdb/")) -(open-store *testbdb-spec*) -@end lisp +(use-package :elephant) +(setf *test-db-spec* + '(:BDB "/home/me/db/testdb/")) +(open-store *test-db-spec*) +@end lisp + +We do not need to store the reference to the store just now as it is +automatically assigned to the variable, @code{*store-controller*}. +For a deeper discussion of store controller management see the +@ref{User Guide}. + +When you're done with your session, release the store-controller's +resources by calling @code{close-store}. + +Also there is a convenience macro @code{with-open-store} that will +open and close the store, but opening the store is an expensive +operation so it is generally better to leave the store open until your +application no longer needs it. + +@node The Store Root +@comment node-name, next, previous, up +@section The Store Root + +What values live between lisp sessions is called @emph{liveness}. +Liveness in a store is determined by whether the value can be reached +from the root of the store. The root is a special BTree in which +other BTrees and lisp values can be stored. This BTree has a special +interface through the store controller. (There is a second root BTree +called the class root which will be discussed later.)

-When you're done with your session, don't forget to +You can put something into the root object by

@lisp -* (close-store) -=> NIL +(add-to-root "my key" "my value") +=> "my value" @end lisp

-Also there is a convenience macro @code{with-open-store}. +and get things out via

-@node The Root -@comment node-name, next, previous, up -@section The Root +@lisp +(get-from-root "my key") +=> "my value" +=> T +@end lisp

-Liveness in a store is determined by reachability from the root -object. Technically, liveness also applies to indexed -classes, as described in @xref{Class Indices}, which live in a -separate class-root namespace. When garbage collection is -implemented, dead objects will be collected on gc's.) The root and -class-root objects are BTrees, effectively a table with sorted keys -and log(N) access time. @xref{Using BTrees}. +The second value indicates whether the key was found. This is +important if your key-value pair can have nil as a value.

-You can put something into the root object by +You can perform other basic operations as well.

@lisp -* (add-to-root "my key" "my value") +(root-existsp "my key") +=> T +(remove-from-root "my key") +=> T +(get-from-root "my key") +=> NIL => NIL @end lisp

-and get things out via +To access all the objects in the root, the simplest way is to +simply call @code{map-root} with a function to apply to each +key-value pair.

@lisp -* (get-from-root "my key") -=> "my value" -=> T -@end lisp +(map-root + (lambda (k v) + (format t "key: ~A value:~A~%" k v))) +@end lisp

-The root object is available as +You can also access the root object directly.

@lisp -* (controller-root *store-controller*) -=> #<BTREE @{492AE37D@}> +(controller-root *store-controller*) +=> #<DB-BDB::BDB-BTREE @ #x10e86042> @end lisp

-It is an instance of a class "btree"; @xref{Using BTrees}. +It is an instance of a class "btree"; @pxref{Using BTrees}.

@node Serialization @comment node-name, next, previous, up @section Serialization

-What can you put into the store? An ever-growing list of things: -numbers (except for complexes, which will be easy to support), +What can you put into the store besides strings? Almost all lisp +values and objects can be stored: numbers (except for complexes), symbols, strings, nil, characters, pathnames, conses, hash-tables, -arrays, CLOS objects. Nested and circular things are allowed. You -can store basically anything except lambdas, closures, structures, -packages and streams. (These may eventually get supported too.) - -Unfortunately Berekely DB doesn't understand Lisp, so Lisp data needs -to be serialized to enter the database (converted to byte arrays), and -deserialized to be read. This introduces some caveats (not unique to -Elephant!): +arrays, CLOS objects and structs. Nested and circular things are +allowed. You can store basically anything except lambdas, closures, +class objects, packages and streams. (These may eventually get +supported too.) + +Elephant needs to use a representation of data that is independant of +a specific lisp or data store. Therefore all lisp values that are +stored must be @emph{serialized} into a canonical format. Because +Berkeley DB supports variable length binary buffers, Elephant uses a +binary serialization system. This process has some important +consequences that it is very important to understand:

@enumerate -@item Lisp identity can't be preserved. Since this is a store which +@item @strong{Lisp identity can't be preserved}. Since this is a store which persists across invocations of Lisp, this probably doesn't even make -sense. +sense. However if you get an object from the index, store it to a +lisp variable, then get it again - they will not be eq:

@lisp -* (setq foo (cons nil nil)) +(setq foo (cons nil nil)) => (NIL) -* (add-to-root "my key" foo) -=> NIL -* (add-to-root "my other key" foo) -=> NIL -* (eq (get-from-root "my key") +(add-to-root "my key" foo) +=> (NIL) +(add-to-root "my other key" foo) +=> (NIL) +(eq (get-from-root "my key") (get-from-root "my other key")) => NIL @end lisp

-@item Mutated substructure does not persist +@item @strong{Nested aggregates are stored in one buffer}. +If you store an set of objects in a hash table you try to store a hash +table, all of those objects will get stored in one large binary buffer +with the hash keys. This is true for all other aggregates that can +store type T (cons, array, standard object, etc). + +@item @strong{Mutated substructure does not persist}.

@lisp -* (setf (car foo) T) +(setf (car foo) T) => T -* (get-from-root "my key") +(get-from-root "my key") => (NIL) @end lisp

This will affect all aggregate types: objects, conses, hash-tables, et cetera. (You can of course manually re-store the cons.) In this sense elephant does not automatically provide persistent collections. If you

[871 lines skipped]