Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv29083/doc

Modified Files: Makefile data-store-reference.texinfo elephant-design.texinfo elephant.texinfo installation.texinfo reference.texinfo tutorial.texinfo user-guide.texinfo Log Message: Latest documentation changes

--- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/30 23:36:52 1.5 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/01 14:33:24 1.6 @@ -9,4 +9,4 @@ makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo

pdf: includes-stuff - texi2dvi --pdf elphant.texinfo + texi2dvi --pdf elephant.texinfo --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 23:36:52 1.4 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5 @@ -3,9 +3,8 @@ @node Data Store API Reference @comment node-name, next, previous, up @chapter Data Store API Reference -@cindex Data Store API Reference @cindex Data Store -@cindex API Reference +@cindex API

This reference includes functions that need to be overridden, classes inherited from or other action taken to implement support for a new @@ -26,16 +25,16 @@ relevant to them.

@menu -* Registration:: Register the backend to parse controller specifications. -* Store Controllers:: Subclassing the store controller. -* Handling Serialization:: Available facilities for serializing objects. -* C Utilities:: Writing primitive C types. -* Slot access:: Support for metaprotocol slot access. -* Collections:: BTrees and indices. -* Cursors:: Traversing BTrees. -* Transactions:: Transaction implementation. -* Multithreading:: Multithreading considerations. -* Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries +* Registration: DSR Registration. Register the backend to parse controller specifications. +* Store Controllers: DSR Store Controllers. Subclassing the store controller. +* Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects. +* Memory Utilities: DSR Memory Utilities. Writing primitive C types. +* Persistent Objects and Slot access: DSR Persistent Objects and Slot Access. Support for metaprotocol slot access. +* Collections: DSR Collections. BTrees and indices. +* Cursors: DSR Cursors. Traversing BTrees. +* Transactions: DSR Transactions. Transaction implementation. +* Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations. +* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries @end menu

@node DSR Registration @@ -96,17 +95,28 @@ @section Slot Access @cindex Persistent Objects and Slot Access

+Persistence is implement with a metaclass and several required base +classes. + +@include includes/class-elephant-persistent-metaclass.texinfo @include includes/class-elephant-persistent.texinfo +@include includes/class-elephant-persistent-object.texinfo + +Persistent objects can be queries for their home store controller so +that functions such as map-btree do not need a store-controller +argument. (NOTE: Should this function be user visible?) + @include includes/fun-elephant-backend-get-con.texinfo -@c @include includes/fun-elephant-backend-oid.texinfo

-All objects require a unique id. During new object creation the -backend is asked to produce a unique id. +All objects require a unique object identifier. During new object +creation the backend is asked to produce a unique id.

@include includes/fun-elephant-backend-next-oid.texinfo

-These functions are called by the metaclass protocol to support -operations on persistent class slots. +These functions are called by the metaclass protocol to implement the +appropriate operations on persistent class slots. Unless protected by +a transaction, the side effects of these functions should be atomic, +persistent and visible to other threads on completion.

@include includes/fun-elephant-backend-persistent-slot-writer.texinfo @include includes/fun-elephant-backend-persistent-slot-reader.texinfo @@ -121,7 +131,9 @@ To support collections, the data store must subclass the following classes.

-@include includes/class-elephant-btree.texinfo.texinfo +@include includes/class-elephant-persistent-collection.texinfo + +@include includes/class-elephant-btree.texinfo @include includes/class-elephant-btree-index.texinfo @include includes/class-elephant-indexed-btree.texinfo

@@ -131,42 +143,48 @@ @include includes/fun-elephant-build-btree.texinfo @include includes/fun-elephant-build-indexed-btree.texinfo

-And every btree needs accessors, these must be implemented for btree, -indexed-btree and btree-index. +Most of the user-visible operations over BTrees must be implemented. +Class indexing functions such as @code{map-class} and +@code{get-instances-by-value} and related functions are all +implemented using map-btree and map-index. + +@itemize +@item @ref{Generic-Function elephant:get-value} +@item @ref{Generic-Function (setf elephant:get-value)} +@item @ref{Generic-Function elephant:existsp} +@item @ref{Generic-Function elephant:remove-kv} +@item @ref{Generic-Function elephant:get-index} +@item @ref{Generic-Function elephant:remove-index} +@item @ref{Generic-Function elephant:map-btree} +@item @ref{Generic-Function elephant:map-index} +@end itemize

-@include includes/fun-elephant-get-value.texinfo -@include includes/fun-elephant-setf-get-value.texinfo -@include includes/fun-elephant-existsp.texinfo -@include includes/fun-elephant-remove-kv.texinfo +Mapping over the indices of a btree is important to derived facilities +such as class indexing and the query subsystem.

@include includes/fun-elephant-map-indices.texinfo -@include includes/fun-elephant-get-index.texinfo -@include includes/fun-elephant-remove-index.texinfo - -Critical to indexing and queries are the map operators for collections - -@include includes/fun-elephant-map-btree.texinfo -@include includes/fun-elephant-map-index.texinfo

@node DSR Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors

-@include includes/class-cursor.texinfo -@c #:cursor-btree -@c #:cursor-oid -@c #:cursor-initialized-p +Data stores must subclass these cursor classes and implement all the +methods described in @ref{Cursors} except @ref{Macro +elephant:with-btree-cursor}. + +@include includes/class-elephant-cursor.texinfo +@include includes/class-elephant-secondary-cursor.texinfo

@node DSR Transactions @comment node-name, next, previous, up @section Transactions @cindex Transactions

-These functions must be implemented or stubbed in any -backend. +These functions must be implemented or stubbed by all data stores.

@include includes/fun-elephant-backend-execute-transaction.texinfo + @include includes/fun-elephant-backend-controller-start-transaction.texinfo @include includes/fun-elephant-backend-controller-commit-transaction.texinfo @include includes/fun-elephant-backend-controller-abort-transaction.texinfo @@ -179,33 +197,77 @@ @include includes/fun-elephant-backend-transaction-store.texinfo @include includes/fun-elephant-backend-transaction-object.texinfo

+;; Designer considerations: +;; - with-transaction passes *current-transaction* or the user parameter to execute-transaction +;; in the parent keyword argument. Backends allowing nested transactions can treat the transaction +;; as a parent, otherwise they can reuse the current transaction by ignoring it (inheriting the dynamic +;; value of *current-transaction*) or rebinding the dynamic context (whatever makes coding easier). +;; - ensure-transaction uses *current-transaction* to determine if there is a current transaction +;; in progress (not null). If so, it jumps to the body directly. Otherwise it executes the body in a +;; new transaction by calling ... +;; - execute-transaction contract: +;; - Backends must dynamically bind *current-transaction* to a meaningful identifier for the +;; transaction in progress and execute the provided closure in that context +;; - All non-local exists result in an abort; only regular return values result in a commit +;; - If a transaction is aborted due to a deadlock or read conflict, execute-transaction should +;; automatically retry with an appropriate default amount +;; - execute-transaction can take any number of backend-defined keywords, although designers should +;; make sure there are no semantic conflicts if there is a name overlap with existing backends +;; - A typical design approach is to make sure that the most primitive interfaces to the backend +;; database look at *current-transaction* to determine whether a transaction is active. Users code can also +;; access this parameter to check whether a transaction is active. + @node DSR Multithreading Considerations @comment node-name, next, previous, up @section Multithreading Considerations @cindex Multithreading

-@c utils locks -@c utils thread-vars +Generic locking utility functions + +Variable behavior in multithreading situations

@node DSR Handling Serialization @comment node-name, next, previous, up @section Handling Serialization @cindex Serialization

-@c #:deserialize #:serialize -@c #:serialize-symbol-complete -@c #:deserialize-from-base64-string -@c #:serialize-to-base64-string +Backends must initialize @ref{Class elephant:store-controller} with +internal serializer functions. Packages @code{elephant-serializer1} +and @code{elephant-serializer2} contains serialize and deserialize +methods on buffer-streams as defined in @code{elephant-memutil}. The +elephant functions @code{serialize} and @code{deserialize} dispatch on +the appropriate slot values of the store-controller. + +@verbatim +NOTE: This should perhaps become entirely the job of the data store to +decide how to serialize values and for a specific version, what +serializer to use. The elphant main package can define serializers +for use by different backends. +@end verbatim + +@include includes/fun-elephant-backend-serialize.texinfo +@include includes/fun-elephant-backend-deserialize.texinfo + +These utility functions are useful if a data store does not have the +ability to store variable length binary data. They are based on the +@code{cl-base64} library. + +@include includes/fun-elephant-backend-serialize-to-base64-string.texinfo +@include includes/fun-elephant-backend-deserialize-from-base64-string.texinfo

-@node DSR Memory utilities +@node DSR Memory Utilities @comment node-name, next, previous, up @section Memory utilities @cindex Memory utilities

-@node DSR Foreign libraries +Details about memory utilities here. + +@node DSR Foreign Libraries @comment node-name, next, previous, up @section Foreign libraries @cindex Foreign libraries

+How foreign libraries are built and used via UFFI. What functions are +in the .asd files or main lisp code to build & load libraries?

--- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/30 14:34:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/01 14:33:29 1.3 @@ -1,9 +1,37 @@ +@c -*-texinfo-*-

@node Elephant Design @comment node-name, next, previous, up -@section Elephant Design +@chapter Elephant Design @cindex design

+Elephant's early architecture was tightly coupled to the Berkeley DB +API. Over time we've moved towards a more modular architecture to +support easy upgrading, repository migration, shared functionality +between data stores and general hygene. + +The current architecture is modularized into the following pieces: + +@verbatim +[Picture describing] Metaclass, Store controller, persistent objects, +data stores, serializer & memutils. Derived features such as class +indexing, migration, query system and root operations can also be +illustrated? +@end verbatim + +To get a feeling for what is happening inside elephant, it is probably +most useful to walk through the various major protocols: + +@itemize +@item Initialization of a store controller +@item Creating a persistent object +@item Operations on persistent slots +@item Operations on persistent collections +@item Implementing @code{with-transaction} +@end itemize + +@section Initializing a store controller + When the main elephant @code{open-store} function is called, it calls @code{get-controller} which grabs an existing store controller if the spec is identical, or builds a new controller. Building the @@ -13,4 +41,14 @@ that returns a @code{store-controller} subclass instance specific to that backend.

-Elephant than calls open-controller +Elephant than calls open-controller to actually establish a connection +to or create the files of the data store. + +@section Persistent Object Creation +@section Persistent Slot Protocol +@section Persistent Slot Protocol +@section Persistent Collection Protocol +@section Implementing Transactions + + + --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/30 14:34:34 1.6 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/01 14:33:29 1.7 @@ -38,6 +38,7 @@ @insertcopying @end ifnottex

+@ifhtml @menu * Table of Contents:: @end menu @@ -49,7 +50,7 @@ * Tutorial:: A basic ``getting started'' tutorial. * Installation:: Installation and test-suite procedures. * User Guide:: In depth discussion of all Elephant facilities and features. -* Usage scenarios:: Design scenarios for Elephant applications. +* Usage Scenarios:: Design scenarios for Elephant applications. * User API Reference:: Function and class documentation of the user API. * Elephant Design:: An overview of elephant's internal architecture. * Data Store API Reference:: Function level documentation for data store implementors. @@ -66,6 +67,8 @@ * Colophon:: @end menu

+@end ifhtml + @node Table of Contents @unnumbered @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/30 14:34:34 1.5 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6 @@ -10,9 +10,9 @@ * Configuring Elephant:: Setting up Elephant and the configuration file. * Loading Elephant:: Loading Elephant and the data store loading protocol. * Berkeley DB Data Store:: Installing support for the Berkeley DB data store -* Berkeley DB Examples:: An example of installing and running the Berkeley DB data store. +* Berkeley DB Example:: An example of installing and running the Berkeley DB data store. * CL-SQL Data Store:: Install and connecting to the CL-SQL data store -* CL-SQL Examples:: An example of using the CL-SQL data store. +* CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows * Test Suites:: How to run and interpret the output of the regression test suite * Documentation:: Building documentation from texinfo sources. @@ -277,10 +277,6 @@ As of Elephant 0.3, Elephant has been tested to work with both Postgres, and SQLite 3, thanks do Dan Knapp.

-@node Extension Status -@comment node-name, next, previous, up -@section Extension Status - As far as is known at this writing, all functionality except nested transaction support and cursor-puts supported by the BerkeleyDB backend is supported by the CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested @@ -302,13 +298,9 @@ the multi-repository version somewhat complicates the underlying persistent object management.

-@node PostGres Examples -@comment node-name, next, previous, up -@section Setting up PostGres - -@node Setting up PostGres +@node CL-SQL Example @comment node-name, next, previous, up -@section Setting up PostGres +@section CL-SQL Example

To set up a PostGres based back end, you should:

@@ -351,6 +343,10 @@ sequences, and so on needed by the Elephant system will be created in the schema named ``test'' automatically.

+@node Elephant on Windows +@comment node-name, next, previous, up +@section Elephant on Windows + @node Test Suites @comment node-name, next, previous, up @section Test Suites --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 23:36:52 1.8 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/01 14:33:29 1.9 @@ -22,14 +22,20 @@ @section Store Controllers @cindex Store Controllers

-Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller. +Store controllers provide the persistent storage for CLOS objects and +BTree collections. Any persistent operations must be done in the +context of a store controller. The default store-controller is stored +in a global variable.

-@include includes/class-elephant-store-controller.texinfo @include includes/var-elephant-star-store-controller-star.texinfo

+@c @include includes/class-elephant-store-controller.texinfo +@ref{Class elephant:store-controller} is associated with the following +user methods and macros: + +@include includes/macro-elephant-with-open-store.texinfo @include includes/fun-elephant-open-store.texinfo @include includes/fun-elephant-close-store.texinfo -@include includes/macro-elephant-with-open-store.texinfo

@include includes/fun-elephant-get-from-root.texinfo @include includes/fun-elephant-add-to-root.texinfo @@ -42,12 +48,12 @@ @section Persistent Objects @cindex Persistent Objects

-@include includes/class-elephant-persistent-metaclass.texinfo -@include includes/class-elephant-persistent.texinfo -@include includes/class-elephant-persistent-object.texinfo +@ref{Class elephant:persistent-metaclass} can be used as the +:metaclass argument in a defclass form to create a persistent object. +Slots of the metaclass take the :index and :transient keyword +arguments and the class accepts the :index keyword argument.

@include includes/macro-elephant-defpclass.texinfo - @include includes/fun-elephant-drop-pobject.texinfo

@node Persistent Object Indexing @@ -94,22 +100,26 @@ interface is provided for reference only, a new system is under development for the 0.7 release.

-@include includes/fun-elephant-get-query-results.texinfo -@include includes/fun-elephant-map-class-query.texinfo +@c @include includes/fun-elephant-get-query-results.texinfo +@c @include includes/fun-elephant-map-class-query.texinfo

@node Collections @comment node-name, next, previous, up @section Collections @cindex Collections

-@include includes/class-elephant-persistent-collection.texinfo -@include includes/class-elephant-btree.texinfo -@include includes/class-elephant-indexed-btree.texinfo -@include includes/class-elephant-btree-index.texinfo +Persistent collections inherit from @ref{Class +elephant:persistent-collection} and consist of the @ref{Class +elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class +elephant:btree-index} classes. The following operations are defined +on most of these classes. More information can be found in @ref{Using +BTrees} and @ref{Secondary Indices}.

@include includes/fun-elephant-get-value.texinfo @include includes/fun-elephant-setf-get-value.texinfo @include includes/fun-elephant-remove-kv.texinfo +@include includes/fun-elephant-existsp.texinfo + @include includes/fun-elephant-map-btree.texinfo @include includes/fun-elephant-map-index.texinfo

@@ -118,17 +128,17 @@ @include includes/fun-elephant-get-primary-key.texinfo @include includes/fun-elephant-remove-index.texinfo

- @node Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors

-@include includes/class-elephant-cursor.texinfo -@include includes/class-elephant-secondary-cursor.texinfo +Cursors are objects of type cursor (@pxref{Class elephant:cursor}) +which provide methods for complex traversals of BTrees. + +@include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo -@include includes/macro-elephant-with-btree-cursor.texinfo

@include includes/fun-elephant-cursor-current.texinfo @include includes/fun-elephant-cursor-delete.texinfo --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/30 14:34:34 1.10 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11 @@ -15,6 +15,7 @@ * Persistent collections:: Keep track of collections of values. * Indexing Persistent Classes:: Simple way to keep track of persistent instances. * Using Transactions:: Providing ACID database properties. +* Advanced Topics:: Additional Elephant features covered in the User Guide. @end menu

@node Overview @@ -704,14 +705,14 @@ @end lisp

The @ref{User Guide} contains a descriptions of the advanced features -of @ref{Class indices} such as ``derived indicies'' that allow you to +of @ref{Class Indices} such as ``derived indicies'' that allow you to order classes according to an arbitrary function, a dynamic API for adding and removing slots and how to set a policy for resolving conflicts between the code image and a database where the indexing specification differs.

This same facility is also available for your own use. For more -information @pxref{Using Indexed BTrees}. +information @pxref{Secondary Indices}.

@node Using Transactions @@ -914,8 +915,71 @@ complicated. The best strategy at the beginning is a conservative one, break things up into the smallest logical sets of primitive operations and only wrap higher level functions in transactions when -they absolutely have to commit together. See @ref{Transaction details} -for the full details and @pxref{Usage scenarios} for more examples of +they absolutely have to commit together. See @ref{Transaction Details} +for the full details and @pxref{Usage Scenarios} for more examples of how systems can be designed and tuned using transactions.

+@node Advanced Topics +@comment node-name, next, previous, up +@section Advanced Topics + +The tutorial covers the essential topics and concepts for using +Elephant. Many people will find that these features are the ones that +are most often needed and used in ordinary applications. + +More sophisticated uses of Elephant may require additional features +that are covered in the user guide. The following is a list of major +features in the user guide that were not covered in this tutorial. + +@itemize +@item @strong{Class Heirarchies and Queries} + There are some subtle issues to take into account when querying +persistent classes. For example, how do you query a base class of +type people to get subclass instances such as employee, manager, +consultant, etc? +@item @strong{Derived Class Indices} + You can create your own indices for classes that are arbitrary +lisp functions of the persistent object. +@item @strong{Class Definition/Database Conflict Resolution} + When you startup lisp, there are potential conflicts between the +class definition and the indexing records in the database. There are +some constraints to account for and some facilities to manage +how slots, class indices and +@item @strong{Dynamic Class Index Management} + It is possible to add and remove indexes from classes at runtime. +@item @strong{BTree Cursors} + If you need to do more than iterate over a collection, or you need +to delete elements of the collection as you iterate cursors are an +important data structure. They implement a variety of operators for +moving backward and forward over a btree, including ranged operations +and iterating of duplicate or unique values. +@item @strong{Indexed BTrees} + Indexed BTrees are just like BTrees, except it is possible to add +indexes which are BTrees who's values are primary keys in the parent +@code{indexed-btree}. This allows for multiple ordering and groupings +of the values of a BTree. +@item @strong{Using the Map Operators} + Mapping operators can be very efficient if properly utilized. +@item @strong{Handling Errors and Conditions} + There are a variety of errors that can occur in Elephant that need +to be dealt with by applications. +@item @strong{Deadlock Detection in Berkeley DB} + Berkeley DB requires an external process to detect deadlock +conditions among transactions. The :deadlock-detect keyword argument +to open-store for Berkeley DB specs will launch this process on most +lisps. +@item @strong{Using Multiple Stores} + Multiple store controllers can be open simultaneously. However it +does make the code more complex and you need to be careful about how +you use them to avoid crashes and other unpleasant side effects. +@item @strong{Custom Transaction Architecture} + You can implement your own version of with-transaction using the +underlying controller methods for starting, aborting and committing +transactions. You had better know what you are doing, however! +@item @strong{Using Multiple Threads and Processes} + What constraints must be accommodated to use Elephant data stores in +multiple threads? What capabilities are there to share data stores +among multiple processes or machines? +@end itemize

+Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/30 14:34:34 1.4 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5 @@ -8,18 +8,19 @@ @menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. -* Reachability:: Determining liveness in a store. * Persistent objects:: All the dirt on persistent objects. -* Class indices:: In-depth discussion about indexing persistent indices. +* Class Indices:: In-depth discussion about indexing persistent indices. * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. -* Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls. +* Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Multi-repository Operation:: Specifying repositories. +* Multi-threaded Applications:: What considerations are required for safe multi-threading +* Multiple Processes and Distributed Applications:: Can elephant be run on multiple CPUs and multiple machines? * Repository Migration and Upgrade:: How to move objects from one repository to another. -* Garbage collection:: How to recover storage and OIDs in long-lived repositories. -* Performance tuning:: How to get the most from Elephant. +* Garbage Collection:: How to recover storage and OIDs in long-lived repositories. +* Performance Tuning:: How to get the most from Elephant. @end menu

@node Persistent objects @@ -83,9 +84,15 @@ @code{with-open-controller} macro. Opening and closing a controller is very expensive.

+@node Serialization details +@comment node-name, next, previous, up +@section Serialization details + +Empty. + @node Class Indices @comment node-name, next, previous, up -@section Class Indicies +@section Class Indices

You can enable/disable class indexing for an entire class. When you disable indexing all references to instances of that class are lost. If you re-enable @@ -130,6 +137,10 @@ somewhat user customizable; documentation for this exists in the source file referenced above.

+@node Querying persistent instances +@comment node-name, next, previous, up +@section Querying persistent instances + @node Using BTrees @comment node-name, next, previous, up @section Using BTrees @@ -147,10 +158,127 @@ blocks of data is relatively inexpensive after a seek and comparisons on objects that are stored in memory is cheap.

+@node Secondary Indices +@comment node-name, next, previous, up +@section Secondary Indices + +Empty. + +@node Using Cursors +@comment node-name, next, previous, up +@section Using Cursors + +Empty.

-@node Repository Migration +@node Transaction Details @comment node-name, next, previous, up -@section Repository Migration +@section Transaction Details + +;; Transaction architecture: +;; +;; User and designer considerations: +;; - *current-transaction* is reserved for use by dynamic transaction context. The default global +;; value must always be null (no transaction). Each backend can set it to a different parameter +;; within the dynamic context of an execute-transaction. +;; - Any closures returned from within a transaction cannot bind *current-transaction* +;; - Only a normal return value will result in the transaction being committed, any non-local exit +;; results in a transaction abort. If you want to do something more sophisticated, roll your own +;; using controller-start-transaction, etc. +;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc) +;; knowing that the transaction will be aborted +;; + +@node Multi-threaded Applications +@comment node-name, next, previous, up +@section Multi-threaded Applications + +Sleepycat plays well with threads and processes. The store controller +is thread-safe by default, that is, can be shared amongst threads. +This is set by the @code{:thread} key. Transactions may not be shared +amongst threads except serially. One thing which is NOT thread and +process safe is recovery, which should be run when no one is else is +talking to the database environment. Consult the Sleepycat docs for +more information. + +Elephant uses some specials to hold parameters and buffers. If you're +using a natively threaded lisp, you can initialize these specials to +thread-local storage by using the @code{run-elephant-thread} function, +assuming your lisp creates thread-local storage for let-bound +specials. (This functionality is currently broken) + +Persisting ordinary aggregate types (e.g. NOT persistent classes or +btrees) suffers from something called "merge-conflicts." Since +updating one value of an aggregate object requires the entire object +to be written to the database, in heavily threaded situations you may +overwrite changes another thread or process has committed. This is +not protected by transactions! + +Consider two processes operating on the same cons: + +@code{ +-----start---read--update-car--write--commit---------------- +-start------read--update-cdr-----------------write--commit-- +} + +Although the first process successfully committed its transaction, its +work (writing to the car) will be erased by the second transaction +(which writes both the car and cdr.) + +Persistent classes and persistent collections do not suffer from +merge-conflicts, since each slot / entry is a separate database entry. + +@node Multi-repository Operation +@comment node-name, next, previous, up +@section Multi-repository Operation + +Elephant now keeps a small hashtables that maps ``database specifications'' into +actual database connections. + +If a database spec is a string, it is assumed to be a BerkeleyDB path. +If it is a list, it is a assumed to be a CL-SQL connection specification. +For example: +@lisp +ELE-TESTS> *testdb-path* +"/home/read/projects/elephant/elephant/tests/testdb/" +ELE-TESTS> *testpg-path* +(:postgresql "localhost.localdomain" "test" "postgres" "") +ELE-TESTS> +@end lisp + +The tests now have a function @code{do-all-tests-spec} that take a spec and +based on its type attempt to open the correct kind of store controller and +perform the tests. + +The routine @code{get-controller} takes this specifiation. + +The basic strategy is that the ``database specification'' object is stored in +every persistent object and collection so that the repository can be found. + +In this way, objects that reside in different repositories can coexist within +the LISP object space, allowing data migration. + +;; Multiple stores + +;; Multiple store considerations: +;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to +;; work around: how to avoid writing one store with a transaction created in the context of another. +;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct; +;; this requirement may relax in the future +;; - The following macros accomodate multiple stores by requiring that execute-transaction return a +;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller +;; is the store instance it is associated with. A nested or ensured transaction is only indicated +;; in the call to execute transaction if the store controllers match, otherwise a new transaction +;; for that store is created + +@node Multiple Processes and Distributed Applications +@comment node-name, next, previous, up +@section Multiple Processes and Distributed Applications + +Can we do this? What do we have to say about this? + +@node Repository Migration and Upgrade +@comment node-name, next, previous, up +@section Repository Migration and Upgrade

This version of Elephant supports migration between store controllers of any backend type. @@ -227,49 +355,17 @@ @code{*inhibit-slot-writes*} in your user method using @code{with-inhibited-slot-copy} a convenience macro.

- -@node Threading +@node Garbage Collection @comment node-name, next, previous, up -@section Threading - -Sleepycat plays well with threads and processes. The store controller -is thread-safe by default, that is, can be shared amongst threads. -This is set by the @code{:thread} key. Transactions may not be shared -amongst threads except serially. One thing which is NOT thread and -process safe is recovery, which should be run when no one is else is -talking to the database environment. Consult the Sleepycat docs for -more information. - -Elephant uses some specials to hold parameters and buffers. If you're -using a natively threaded lisp, you can initialize these specials to -thread-local storage by using the @code{run-elephant-thread} function, -assuming your lisp creates thread-local storage for let-bound -specials. (This functionality is currently broken) - -Persisting ordinary aggregate types (e.g. NOT persistent classes or -btrees) suffers from something called "merge-conflicts." Since -updating one value of an aggregate object requires the entire object -to be written to the database, in heavily threaded situations you may -overwrite changes another thread or process has committed. This is -not protected by transactions! - -Consider two processes operating on the same cons: - -@code{ ------start---read--update-car--write--commit---------------- --start------read--update-cdr-----------------write--commit-- -} - -Although the first process successfully committed its transaction, its -work (writing to the car) will be erased by the second transaction -(which writes both the car and cdr.) +@section Garbage Collection

-Persistent classes and persistent collections do not suffer from -merge-conflicts, since each slot / entry is a separate database entry. +GC is not implemented, but migration (@pxref{Repository Migration and +Upgrade}) will consolidate storage and recover OIDs which emulates GC. +No online solution is currently supported.

-@node Performance Tips +@node Performance Tuning @comment node-name, next, previous, up -@section Performance Tips +@section Performance

Performance is usually measured in transactions per second. Database reads are cheap. To get more transactions throughput, consider @@ -290,8 +386,8 @@ things. It is fast but consing with floats and doubles. YMMV with other values, though I've tried to make them fast.

-Using @code{*auto-commit*} and not @code{with-transactions} is a great -way to have a huge number of transactions. You'll find that +Use @code{with-transactions} to avoid many automatic transactions, for +example you'll find that this construct

@lisp (dotimes (i 1000) (add-to-root "key" "value")) @@ -300,9 +396,8 @@ is much slower than

@lisp -(let ((*auto-commit* nil)) - (with-transaction () - (dotimes (i 1000) (add-to-root "key" "value")))) +(with-transaction () + (dotimes (i 1000) (add-to-root "key" "value")))) @end lisp

since there's only 1 transaction in the latter. However storing @@ -319,32 +414,3 @@ mode is not suitable for use in web servers or other typically multi-threaded applications.

-@node Multi-repository Operation -@comment node-name, next, previous, up -@section Multi-repository Operation - -Elephant now keeps a small hashtables that maps ``database specifications'' into -actual database connections. - -If a database spec is a string, it is assumed to be a BerkeleyDB path. -If it is a list, it is a assumed to be a CL-SQL connection specification. -For example: -@lisp -ELE-TESTS> *testdb-path* -"/home/read/projects/elephant/elephant/tests/testdb/" -ELE-TESTS> *testpg-path* -(:postgresql "localhost.localdomain" "test" "postgres" "") -ELE-TESTS> -@end lisp - -The tests now have a function @code{do-all-tests-spec} that take a spec and -based on its type attempt to open the correct kind of store controller and -perform the tests. - -The routine @code{get-controller} takes this specifiation. - -The basic strategy is that the ``database specification'' object is stored in -every persistent object and collection so that the repository can be found. - -In this way, objects that reside in different repositories can coexist within -the LISP object space, allowing data migration.