Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv16753/doc

Modified Files: copying.texinfo data-store-reference.texinfo elephant.texinfo installation.texinfo reference.texinfo scenarios.texinfo tutorial.texinfo user-guide.texinfo Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft

--- /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/03/30 14:34:34 1.4 +++ /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/04/28 02:31:07 1.5 @@ -8,29 +8,30 @@

@section Elephant Licensing

-@b{Elephant}: a persistent metaprotocol and object-oriented database -for Common Lisp. +Elephant is a persistent metaprotocol and object-oriented database for +Common Lisp. Detailed information and distributions can be found at +@uref{http://www.common-lisp.net/project/elephant%7D.

-Homepage: @uref{http://www.common-lisp.net/project/elephant%7D +The program is released under the following license:

@quotation -Elephant users are granted the rights to distribute and use this software -as governed by the terms of the Lisp Lesser GNU Public License -@uref{http://opensource.franz.com/preamble.html%7D, also known as the LLGPL. +Elephant users are granted the rights to distribute and use this +software as governed by the terms of the Lisp Lesser GNU Public +License @uref{http://opensource.franz.com/preamble.html%7D, also known +as the LLGPL. @end quotation

Copyrights include:

@quotation -Copyright (c) 2004 by Andrew Blumberg and Ben Lee - -Copyright (c) 2005-2007 by Robert L. Read - -Copyright (c) 2006-2007 by Ian Eslick +Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. @* +Version 0.5, Copyright @copyright{} 2006 Robert L. Read. @* +Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @* +Portions copyright respective contributors (see @file{CREDITS}). @end quotation

-Portions of this program (namely the C unicode string sorter) are -derived from IBM's @b{ICU}: @uref{http://oss.software.ibm.com/icu/, +Portions of the program (namely the C unicode string sorter) are +derived from IBM's ICU: @uref{http://oss.software.ibm.com/icu/, ICU Website} whose copyright and license follows below.

@quotation @@ -75,6 +76,33 @@ are the property of their respective owners. @end quotation

+@section Elephant Manual Copyright and Licensing + +@quotation +Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License. +@end quotation + +Copyrights include: + +@quotation +Original Version, Copyright @copyright{} 2004 Ben Lee. @* +Versions 0.5-0.6, Copyright @copyright{} 2006 Robert L. Read. @* +Current Version, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @* +@end quotation + +@section 3rd Party Libraries + +Elephant depends on 3rd party lisp libraries. See their respective +distributions for detailed copyright and licensing information. The +following is a brief summary. + +@itemize +@item @b{uffi}: By Kevin Rosenberg, no significant restrictions +@item @b{cl-base64}: By Kevin Rosenberg, no significant restrictions +@item @b{rt}: By Richard Waters, MIT License +@end itemize + @section Data Store Licensing Considerations

The Berkeley DB data store is based on the Berkeley DB C library, now @@ -100,5 +128,3 @@ @item MySQL: Dual licensing (similar to BDB), see @uref{http://www.mysql.com/company/legal/licensing/, the MySQL license page} @end itemize

- - --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/27 03:14:55 1.7 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/28 02:31:07 1.8 @@ -34,7 +34,7 @@ * Cursors: DSR Cursors. Traversing BTrees. * Transactions: DSR Transactions. Transaction implementation. * Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations. -* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries +@c * Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries @end menu

@node DSR Registration @@ -147,8 +147,8 @@ implemented using map-btree and map-index.

@itemize -@item @ref{Generic-Function elephant:get-value} -@item @ref{Generic-Function (setf elephant:get-value)} +@item @ref{Generic-Function elephant:get-value} (and @code{(setf get-value)}) +@c @item @ref{Generic-Function (setf elephant:get-value)} @item @ref{Generic-Function elephant:existsp} @item @ref{Generic-Function elephant:remove-kv} @item @ref{Generic-Function elephant:get-index} @@ -168,7 +168,7 @@ @cindex Cursors

Data stores must subclass these cursor classes and implement all the -methods described in @ref{Cursors} except @ref{Macro +methods described in @ref{DSR Cursors} except @ref{Macro elephant:with-btree-cursor}.

@include includes/class-elephant-cursor.texinfo @@ -220,9 +220,23 @@ @section Multithreading Considerations @cindex Multithreading

-Generic locking utility functions +This expands slightly on the multithreading discussion in +@ref{Multi-threaded Applications}.

-Variable behavior in multithreading situations +Elephant provides a set of generic locking functions in +@code{src/utils/locks.lisp} to help protect any shared structures. +There are standard locking functions (@code{ele-with-lock}) and then +a special locking interface called @code{ele-with-fast-lock} which +on some lisps provides a faster locking option than the standard OS +locks of the basic interface. (i.e. under Allegro this uses +@code{without-interrupts} because Allegro still runs in a single +OS process on all platforms, this is not true of SBCL). + +See the sections on Transaction handling, particularly the dynamic +behavior of @code{*current-transaction*}. Also read up on the +store controller section in the User Guide to better understand the +role of @code{*store-controller*}. At this time there are no other +global variables to worry about.

@node DSR Handling Serialization @comment node-name, next, previous, up @@ -253,19 +267,19 @@ @include includes/fun-elephant-data-store-serialize-to-base64-string.texinfo @include includes/fun-elephant-data-store-deserialize-from-base64-string.texinfo

-@node DSR Memory Utilities -@comment node-name, next, previous, up -@section Memory utilities -@cindex Memory utilities - -Details about memory utilities here. - -@node DSR Foreign Libraries -@comment node-name, next, previous, up -@section Foreign libraries -@cindex Foreign libraries +@c @node DSR Memory Utilities +@c @comment node-name, next, previous, up +@c @section Memory utilities +@c @cindex Memory utilities + +@c Details about memory utilities here. + +@c @node DSR Foreign Libraries +@c @comment node-name, next, previous, up +@c @section Foreign libraries +@c @cindex Foreign libraries

-How foreign libraries are built and used via UFFI. What functions are -in the .asd files or main lisp code to build & load libraries? +@c How foreign libraries are built and used via UFFI. What functions are +@c in the .asd files or main lisp code to build & load libraries?

--- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/27 03:14:55 1.10 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/28 02:31:07 1.11 @@ -2,19 +2,27 @@ @c %**start of header @setfilename elephant.info @settitle Elephant User Manual +@setchapternewpage odd @c %**end of header - -@copying -Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. -Version 0.5, Copyright @copyright{} 2006 Robert L. Read. -Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read

+@copying @quotation +Elephant System @* +Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. @* +Version 0.5, Copyright @copyright{} 2006 Robert L. Read. @* +Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @* +Portions copyright respective contributors (see @file{CREDITS}). @* + +Elephant Manual @* +Original Version, Copyright @copyright{} 2004 Ben Lee. @* +Versions 0.5-0.6, Copyright @copyright{} 2006 Robert L. Read. @* +Current Version, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @* + Permission is granted to copy, distribute and/or modify this document -under the terms of the GNU Free Documentation License. Portions of -Elephant are derived from ICU, IBM's Unicode library. Details about -copyright, license and warranty information can be found in -@file{LICENSE} or in the ``Copying'' section of this document. +under the terms of the GNU Free Documentation License. See the +Copyright and License chapter for details about copyright, license and +warranty for this manual and the Elephant system. + @end quotation @end copying

--- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/27 03:14:55 1.11 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 02:31:07 1.12 @@ -9,9 +9,10 @@ * Requirements:: Supported lisps and required libraries. * Configuring Elephant:: Setting up Elephant and the configuration file. * Loading Elephant:: Loading Elephant and the data store loading protocol. -* Berkeley DB:: Installing support for the Berkeley DB data store +* Berkeley DB:: Installing support for the Berkeley DB data store. * Berkeley DB Example:: An example of installing and running the Berkeley DB data store. -* CL-SQL:: Install and connecting to the CL-SQL data store +* Upgrading Berkeley DB Database:: How to upgrade to a new version of Berkeley DB. +* CL-SQL:: Install and connecting to the CL-SQL data store. * CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows * Test Suites:: How to run and interpret the output of the regression test suite @@ -275,6 +276,57 @@ @end lisp in your application.

+@node Upgrading Berkeley DB Databases +@comment node-name, next, previous, up +@section Upgrading Berkeley DB Databases + +At regular intervals, Elephant will require a specific version of Berkeley DB +because it does not parse the header files which tend to change with each release. +The patches are usually minor, but sometimes Elephant also depends on new +features of Berkeley DB. In this case, you have to update both the Berkeley DB +database files as well as any data Elephant has built within it. That creates +some special constraints for upgrading databases. + +@subsection Upgrading to 0.9 + +This section outlines how to upgrade from Elephant code base version 0.6.0 +using Berkeley DB 4.3. + +@enumerate +@item Install BDB 4.5 (keep 4.3 around for now) +@item Update my-config.sexp to point to the appropriate BDB 4.5 directories +@item Upgrade your database directory to 4.5 + @itemize + @item Run db43_recover in your 0.6 database + @item Optional: run db43_archive -d to remove all logs not part of a checkpoint + This will make catastrophic recovery impossible, but reduces the amount of data you + have to backup. + @item Backup your db files and remaining logs + @item Run db45_checkpoint -1 in your 0.6 database directory + @end itemize +@item Migrate 0.6 data to a new 0.6.1 database + @itemize + @item Open your old database: @code{(setf sc (open-store '(:BDB "/Users/me/db/ele060/")))} + @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele061/"))} + @end itemize +@item Test your new application and report any bugs that arise to elephant-devel@common-lisp.net +@end itemize + +@emph{(NOTE: close-store may fail when closing the old 0.6 database, this is OK.)} + +@emph{(NOTE: 64-bit lisps will not successfully upgrade 32-bit 0.6 databases. Use a 32-bit + version of your lisp to update to 0.6.1 and then open that database in your 64-bit + lisp. There should be no compatibility problems. Best to test your application on + a 32-bit lisp if you can, just to be sure.)} + +@subsection Upgrade from Elephant 0.5 + +Follow the upgrade procedures outlined in Elephant release 0.6.0 to migrate your database +from 0.5 to 0.6. Then follow the above procedures for upgrading from an 0.6 database. + +@emph{(NOTE: It may not take much work to make 0.9 upgrade directly from 0.5 +However there are so few (none?) 0.5 users that it wasn't deemed worth +the work given that there's an upgrade path available.)}

@node CL-SQL @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/27 03:14:55 1.13 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/28 02:31:07 1.14 @@ -140,13 +140,15 @@ Persistent collections inherit from @ref{Class elephant:persistent-collection} and consist of the @ref{Class elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class elephant:btree-index} classes. The following operations are defined -on most of these classes. More information can be found in @ref{Using BTrees} -and @ref{Secondary Indices}. +on most of these classes. More information can be found in @ref{Persistent BTrees} +and @ref{BTree Indexing}.

@include includes/fun-elephant-make-btree.texinfo - @include includes/fun-elephant-get-value.texinfo -@include includes/fun-elephant-setf-get-value.texinfo + +Values are written to a btree using the @code{setf} method on @code{get-value}. +@c @include includes/fun-elephant-setf-get-value.texinfo + @include includes/fun-elephant-remove-kv.texinfo @include includes/fun-elephant-existsp.texinfo @include includes/fun-elephant-drop-btree.texinfo @@ -178,7 +180,8 @@ @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo @include includes/fun-elephant-cursor-duplicate.texinfo -@include includes/fun-elephant-cursor-initialized-p.texinfo +@c fixme, not generated by docstrings +@c @include includes/fun-elephant-cursor-initialized-p.texinfo

Each of the following methods return multiple values consisting of @code{(exists? key value)}. @@ -197,8 +200,8 @@

@node Index Cursor API @comment node-name, next, previous, up -@section Cursor API -@cindex Cursors +@section Index Cursor API +@cindex Cursor @cindex Index @cindex Indices @cindex API @@ -224,12 +227,15 @@ @include includes/fun-elephant-cursor-pget-both.texinfo @include includes/fun-elephant-cursor-pget-both-range.texinfo

-@include includes/fun-elephant-cursor-next-dup.texinfo @include includes/fun-elephant-cursor-next-nodup.texinfo -@include includes/fun-elephant-cursor-pnext-dup.texinfo +@include includes/fun-elephant-cursor-next-dup.texinfo @include includes/fun-elephant-cursor-pnext-nodup.texinfo +@include includes/fun-elephant-cursor-pnext-dup.texinfo + @include includes/fun-elephant-cursor-prev-nodup.texinfo +@include includes/fun-elephant-cursor-prev-dup.texinfo @include includes/fun-elephant-cursor-pprev-nodup.texinfo +@include includes/fun-elephant-cursor-pprev-dup.texinfo

@node Transaction API @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/27 03:14:55 1.9 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/28 02:31:07 1.10 @@ -711,14 +711,15 @@ hash table indirections with a little macro:

@lisp -(defmacro def-snapshot-wrapper - (accessor-name (source-classname target-classname hashname uid)) +(defmacro def-snapshot-wrapper (accessor-name + (source-classname target-classname hashname uid)) (with-gensysms (obj key ref) `(progn (defmethod ,accessorname :around ((,obj ,source-classname)) (let ((,key (call-next-method))) (when ,key (gethash ,key ,hashname)))) - (defmethod (setf ,accessorname) :around (,ref (,obj ,source-classname)) + (defmethod (setf ,accessorname) :around + (,ref (,obj ,source-classname)) (cond ((subtypep (type-of ,ref) ,target-classname) (let ((,key (find-object ,ref ,hashname))) (if ,key --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/27 03:14:55 1.19 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/28 02:31:11 1.20 @@ -272,7 +272,7 @@

@item @strong{Merge-conflicts in heavily multi-process/threaded situations}. This is the common read-modify-write problem in all databases. We will talk -more about this in the @ref{Transactions} section. +more about this in the @ref{Using Transactions} section.

@end enumerate

@@ -761,7 +761,9 @@ (get-instances-by-range 'friends 'name "Adam" "Devin") => (#<Adriana> #<Carlos>)

-(get-instances-by-range 'friend 'birthday (encode-date '(1 1 1974)) (encode-date '(31 12 1984))) +(get-instances-by-range 'friend 'birthday + (encode-date '(1 1 1974)) + (encode-date '(31 12 1984))) => (#<Zaid> #<Adriana>)

(mapc #'print-friend *) @@ -1207,12 +1209,12 @@ and iterating of duplicate or unique values. @item @strong{Using the Map Operators} Mapping operators can be very efficient if properly utilized. -@item @strong{Using Multiple Stores} Multiple store controllers can -be open simultaneously. However it does make the code more complex -and you need to be careful about how you use them to avoid crashes -and other unpleasant side effects. -@item @strong{Custom Transaction Architecture} You can implement your -own version of @code{with-transaction} using the +@item @strong{Using Multiple Stores} +Multiple store controllers can be open simultaneously. However it +does make the code more complex and you need to be careful about how +you use them to avoid crashes and other unpleasant side effects. +@item @strong{Custom Transaction Architecture} +You can implement your own version of @code{with-transaction} using the underlying controller methods for starting, aborting and committing transactions. You had better know what you are doing, however! @item @strong{Handling Errors and Conditions} --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/27 03:14:55 1.19 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 02:31:11 1.20 @@ -188,7 +188,7 @@

@item @strong{Merge-conflicts in heavily multi-process/threaded situations}. This is the common read-modify-write problem in all databases. We will talk -more about this in the @ref{Transactions} section. +more about this in the @ref{Transaction Details} section.

@item @strong{Byte Ordering}. The primitive elements such as integers are written to disk in @@ -908,7 +908,7 @@ Once a cursor is properly initialized, it can be incremented or decremented, a simple constant-time operation on BTrees.

-@code{cursor-next} and @{cursor-prev} move the cursor a single step +@code{cursor-next} and @code{cursor-prev} move the cursor a single step forward or back across the sorted key-value pairs. @code{cursor-next} moves in ascending order, @code{cursor-prev} in descending order.

@@ -960,10 +960,11 @@ class indexing capabilities previously described.

An index is created by using the @code{add-index} function. This -function takes the @code{indexed-btree} you wish to index, an -index-name for later retrieval, a key-form which dictates how -the index populates it's keys as a function of the main btree's -keys and values. +function takes the @code{indexed-btree} you wish to index, an symbolic +name for the index and a key-form which dictates how the index +populates it's keys as a function of the main btree's keys and values. +(It is a function of three arguments: the index itself, the key and +the value).

A simple, contrived example is shown in the figure below:

@@ -1018,23 +1019,37 @@ Operations that have the same behavior, but return primary btree values and keys are:

-@table @samp +@multitable @columnfractions .3 .1 .35 +@headitem BTree Cursor Function @tab @tab Index Cursor Function + @item @code{cursor-first} -@code{cursor-pfirst} -@samp{@code{cursor-first}} +@tab => +@tab @code{cursor-pfirst} + @item @code{cursor-last } -@code{cursor-plast} +@tab => +@tab @code{cursor-plast} + @item @code{cursor-current } -@code{cursor-pcurrent} +@tab => +@tab @code{cursor-pcurrent} + @item @code{cursor-next } -@code{cursor-pnext} +@tab => +@tab @code{cursor-pnext} + @item @code{cursor-prev} -@code{cursor-pprev} +@tab => +@tab @code{cursor-pprev} + @item @code{cursor-set} -@code{cursor-pset} +@tab => +@tab @code{cursor-pset} + @item @code{cursor-set-range} -@code{cursor-set-prange} -@end table +@tab => +@tab @code{cursor-set-prange} +@end multitable

The big difference between btree cursors and index cursors is that indices can have duplicate key values. This means we have to choose @@ -1042,100 +1057,217 @@ a duplicate segment. There are cursor operations for each:

@itemize -@item Simple move. Standard btree operations plus @code{cursor-pnext} and @code{cursor-pprev}. -@item Move to next key value. @code{cursor-pnext-nodup} and @code{cursor-pprev-nodup}. -@item Move to next duplicate. +@item Simple move. Standard btree operations work plus @code{cursor-pnext} and @code{cursor-pprev} +@item Move to a different key value. @code{cursor-pnext-nodup} and @code{cursor-pprev-nodup} +@item Move to next duplicate key value. @code{cursor-pnext-dup} and @code{cursor-pprev-dup} @end itemize

-@c FINISH +After incrementing through a set of duplicate items using a +@code{xxx-dup} function, the last next operation returns nil +indicating there are no more duplicates. The consequence of this is +that the cursor is now uninitialized (@code{cursor-initialized-p}) and +needs to be reset by a set or set both call. + + @node Multi-threaded Applications @comment node-name, next, previous, up @section Multi-threaded Applications

-Berkeley DB plays well with threads and processes. The store -controller is thread-safe by default, that is, can be shared amongst -threads. This is enabled by the @code{:thread} keyword argument which -defaults to true. Transactions may not be shared amongst threads -except serially. One thing which is NOT thread and process safe is -recovery, which should be run when no one is else is talking to the -database environment. - -The following shared regions of Elephant are protected by standard locks: -- buffer stream pool -- access to serializer circular buffers -- writes to connection db -- where else? +Elephant is thread-safe by design. Users should not have to think +about threading except to follow a couple of simple rules. + +@enumerate +@item Do not perform transactions across multiple threads +@item Do not perform add/remove index operations on indexed-btrees +in more than one thread. +@end enumerate + +This and common coding sense should be sufficient! Elephant's +internal design for thread safety employs a number of policies +to try to minimize using lisp locks and simplify analysis of +multi threaded interactions: + +@enumerate +@item @strong{Rely on the thread safety of the data store databases} +@item @strong{Ensure transaction isolation} +@item @strong{Minimize dependency on thread-local special variables} +@item @strong{Protect shared resources for a given store controller}. +@item @strong{A use policy for shared objects (above)} +@end enumerate + +@subsection Shared Resources + +Elephant has a few shared resources which are protected by standard locks. +These are:

+@itemize +@item The store controller connection table +@item The instance cache +@item The circularity buffer pool for the serializer +@item The buffer-stream pool in memutils +@end itemize + +In some cases, and on some lisp platforms, we try to use a fast lock +strategy for frequently accessed items (the resource pools and +instance cache especially). + +@subsection Data Store Thread Safety and Transactions + +Both CLSQL and Berkelely DB backends are thread safe. In CLSQL this +is by ensuring that every thread has it's own handle into the SQL +libraries or sockets. Berkeley DB is reentrant and handles locking +internally. + +Elephant depends on these guarantees especially for the isolation +properties of transactions. All operations in the context of a +given transaction should be isolated and atomic. It is important +that a transaction not be shared across threads, however. + +@subsection Minimize Dependency on Thread-Local Specials + +Elephant uses several global variables as default arguments. Most +of these were removed leaving only a couple to handle: + +@itemize +@item @strong{@code{*store-controller*}}. Store controller objects can +be shared between threads and if a user resets this variable in a local +thread to another controller, there is no problem with that either. +Users of multiple concurrent stores can specify the store controller to +all elephant API commands that don't get it from a persistent object +implicitely. +@code{*current-transaction*}. This is always set to the proper null +value globally and should not be reset in local threads. Instead, +transactions take place in a dynamic context that rebinds this variable +as a special with the current transaction. This allows for a dynamic +transaction stack for data stores that can nest transactions or when +two datastores are both doing transactions concurrently. +@end itemize

-@c *** FINISH *** @node Transaction Details @comment node-name, next, previous, up @section Transaction Details

+Transactions are dynamic contexts in which all side effects to +persistent slots and other persistent objects such as BTrees are +guaranteed to have the ACID properties: atomicity, consistency, +isolation and durability. On a normal exit from context, the +side effects are committed as a group. On a non-local exit, +the transaction is aborted. + +For most users, the tutorial section @ref{Using Transactions} is the +best introduction to transactions. This section adds to that by +exposing some of the details of how it is implemented. + +To reiterate, there are a few important restrictions to adhere to: + +@itemize +@item @code{*current-transaction*} is reserved for use by the transaction system. Users should not override, manipulate or close over this variable. +@item The body of a transaction cannot throw, signal or jump without aborting the transaction. Any non-local exit is considered an aborting event. Catch signals inside the transaction and return a value instead. +@item The dynamic extent of a transaction body must stay within the same thread +@end itemize + +@subsection @code{with-transaction} internals + +The @code{with-transaction} macro wraps the body expression with an +anonymous lambda expression. This closure is passed to a call to the +@code{execute-transaction} generic function which is specialized to +the current data store. + +The only bookkeeping done by the macro is ensuring that the +@code{:parent} argument is checked for the current dynamic transaction +context. If it is not owned by the default or provided store +controller, then it is not passed to @code{execute-transaction}. This +maintains a continuous dynamic stack transactions through the +with/ensure transaction macros, but allows for a single leaf +transaction to another store controller. + +Be very careful about mixing transactions between store controller. +This facility was only added to ensure that migrate worked correctly. + +The macro processes keywords arguments @code{:store-controller} +(defaults to @code{*store-controller*}), @code{:parent} (defaults to +@code{*current-transaction*}) and @code{:retries} and passes the +remaining keywords to the call to @code{execute-transaction} allowing +the user to pass data store specific transaction keywords to their +preferred data store. The consumed keywords are analyzed and then +passed on to @code{execute-transaction}. + +Any non-standard keywords for a given data store will be ignored by +other data store implementation of @code{execute-transaction} so +portable programs should not use keywords that change the semantics of +the transaction. + +@code{ensure-transaction} only calls @code{execute-transaction} if +it needs to create a fresh transaction. If the transaction in +@code{*current-transaction*} exists and belongs to the store controller +passed to @code{ensure-transaction} then it merely calls the transaction +closure, relying on the environment that created the transaction to +handle any exit procedures and determining whether to abort or commit. + +@code{*current-transaction*} contains transaction records during the +dynamic execution of a transaction. These records capture any data +store specific bookkeeping as well as the store-controller that the +transaction is associated with. + +@subsection @code{execute-transaction} internals + +See the @ref{Elephant Architecture} section for details on how +execute-transaction works. It will provide some deeper insight +into the transaction system. + +@subsection Building your own transactional framework + +Data stores are required to implement three primitive transaction +methods: @code{controller-start-transaction}, +@code{controller-abort-transaction} and +@code{controller-commit-transaction}. These are wrappers for the data +store's primitive transaction mechanism. If you use these, it is up +to you to make sure that you properly manage nested transactions, +maintain the state of @code{*current-transaction*} handle any +automated retries you might want, and handle detecting + +If you use these, you are on your own - it is easy to make mistakes with +transactions and create very complex bugs that are hard to track down. +Most users are much better off sticking with the two transaction macros +and the underlying @code{execute-transaction} method. + +@subsection Analyzing Dynamic Transaction Behavior + You can trace @code{elephant::execute-transaction} to see the sequence -of calls to @code{execute-transaction} that occur dynamically and -detect where transactions are and are not happening. We may add some -transaction diagnosis and tracing tools in the future, such as -throwing a condition when @code{with-transaction} forms are nested -dynamically. - -;; Transaction architecture: -;; -;; User and designer considerations: -;; - *current-transaction* is reserved for use by dynamic transaction context. The default global -;; value must always be null (no transaction). Each data storeend can set it to a different parameter -;; within the dynamic context of an execute-transaction. -;; - Any closures returned from within a transaction cannot bind *current-transaction* -;; - Only a normal return value will result in the transaction being committed, any non-local exit -;; results in a transaction abort. If you want to do something more sophisticated, roll your own -;; using controller-start-transaction, etc. -;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc) -;; knowing that the transaction will be aborted -;; +of calls that occur dynamically and detect where and how many +transactions are and are not happening. + +@c IT WOULD BE GOOD TO EXPAND ON THIS REGARDING HOW TO SOLVE COMMON PROBLEMS

-@c *** FINISH *** @node Multi-repository Operation @comment node-name, next, previous, up @section Multi-repository Operation

-Elephant now keeps a small hashtables that maps ``database specifications'' into -actual database connections. +Elephant maintains a small hashtable that maps ``database +specifications'' into actual @code{store-controller} objects. + +The basic strategy is that the ``database specification'' object is +stored in every persistent object and collection so that the +repository can be found. In this way, objects that reside in +different repositories can coexist within the LISP object space, +allowing data migration or multiple user stores. + +All persistent instances store their oid and a store-controller +reference in internal slots. Slot access and other protocols use +this to provide access. This executes an auto-transaction or joins +a surrounding transaction if the @code{transaction-record} in +@code{*current-transaction*} matches the store. + +When operating with multiple stores and nested transactions there are +some subtle issues to work around: how to avoid writing one store with +a transaction created in the context of another. A nested or ensured +transaction is only indicated in the call to +@code{execute-transaction} if the store controllers match, otherwise a +new transaction for that store is created. + +@c A PICTURE OF THE DYNAMIC CONTEXT WOULD BE USEFUL HERE

-If a database spec is a string, it is assumed to be a BerkeleyDB path. -If it is a list, it is a assumed to be a CLSQL connection specification. -For example: -@lisp -ELE-TESTS> *testdb-path* -"/home/read/projects/elephant/elephant/tests/testdb/" -ELE-TESTS> *testpg-path* -(:postgresql "localhost.localdomain" "test" "postgres" "") -ELE-TESTS> -@end lisp

-The tests now have a function @code{do-all-tests-spec} that take a spec and -based on its type attempt to open the correct kind of store controller and -perform the tests. - -The routine @code{get-controller} takes this specifiation. - -The basic strategy is that the ``database specification'' object is stored in -every persistent object and collection so that the repository can be found. - -In this way, objects that reside in different repositories can coexist within -the LISP object space, allowing data migration. - -;; Multiple stores - -;; Multiple store considerations: -;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to -;; work around: how to avoid writing one store with a transaction created in the context of another. -;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct; -;; this requirement may relax in the future -;; - The following macros accomodate multiple stores by requiring that execute-transaction return a -;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller -;; is the store instance it is associated with. A nested or ensured transaction is only indicated -;; in the call to execute transaction if the store controllers match, otherwise a new transaction -;; for that store is created

@node Multiple Processes and Distributed Applications @comment node-name, next, previous, up