elephant-cvs
Threads by month
- ----- 2026 -----
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- 858 discussions
Update of /project/elephant/cvsroot/elephant/src/db-clsql
In directory clnet:/tmp/cvs-serv8188
Modified Files:
package.lisp
Log Message:
Not sure what happened here but this file is definitely needed for the test on
the sql side.
--- /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/02/07 22:54:12 1.2
+++ /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/04/01 22:01:51 1.3
@@ -21,7 +21,7 @@
(defpackage db-clsql
(:use :common-lisp :uffi :cl-base64
:elephant :elephant-memutil :elephant-backend
-;; :elephant-utils
+ :elephant-utils
#+sbcl :sb-thread
))
1
0
Update of /project/elephant/cvsroot/elephant/doc
In directory clnet:/tmp/cvs-serv27196/doc
Modified Files:
user-guide.texinfo
Log Message:
Fixed package bug. Minor tweaks elsewhere
--- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:22:24 1.6
+++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:56:19 1.7
@@ -141,8 +141,6 @@
@comment node-name, next, previous, up
@section Querying persistent instances
-
-
A SQL select-like interface is in the works, but for now queries are
limited to manual mapping over class instances or doing small queries
with @code{get-instances-*} functions. One advantage of this is that
1
0
Update of /project/elephant/cvsroot/elephant/src/elephant
In directory clnet:/tmp/cvs-serv27196/src/elephant
Modified Files:
backend.lisp package.lisp query-example.lisp query.lisp
Log Message:
Fixed package bug. Minor tweaks elsewhere
--- /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 23:36:53 1.15
+++ /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/04/01 20:56:19 1.16
@@ -56,6 +56,10 @@
#:controller-deserialize
#:root #:spec #:class-root
+ ;; Collections
+ #:build-btree
+ #:build-indexed-btree
+
;; Serializer tools/api's
#:serialize #:deserialize
#:deserialize-from-base64-string
--- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/03/30 14:55:54 1.27
+++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/01 20:56:19 1.28
@@ -206,9 +206,9 @@
#:persistent #:persistent-object #:persistent-metaclass #:defpclass
#:persistent-collection #:drop-pobject
- #:btree #:build-btree
+ #:btree #:make-btree
#:get-value #:remove-kv #:existsp
- #:indexed-btree #:build-indexed-btree
+ #:indexed-btree #:make-indexed-btree
#:btree-index
#:add-index #:get-index #:remove-index #:map-indices
#:get-primary-key #:primary #:key-form #:key-fn
@@ -260,6 +260,9 @@
#:lookup-persistent-symbol-id
#:struct-constructor
+
+ #:map-class-query
+ #:get-query-instances
)
)
--- /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/03/01 02:45:45 1.1
+++ /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/04/01 20:56:19 1.2
@@ -61,7 +61,7 @@
(slot-value (slot-value person 'department) 'name)))
(defun example-query1 ()
- "Performs a query against a single class. Trivial string & integer matchingA"
+ "Performs a query against a single class. Trivial string & integer matching"
(map-class-query #'print-person
'((person name = "Greg")
(person salary >= 100000))))
--- /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/03/30 14:34:35 1.2
+++ /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/04/01 20:56:19 1.3
@@ -41,6 +41,7 @@
(defun get-query-instances (constraints)
"Get a list of instances according to the query constraints"
+ (declare (dynamic-extent constraints))
(let ((list nil))
(flet ((collect (inst)
(push inst list)))
@@ -52,6 +53,7 @@
slot indices (for last query) and stack allocated test closures. This is
a minimally optimizing version that uses the first index it finds, and
then does a nested loop join on the rest of the parameters."
+ (declare (dynamic-extent constraints))
(assert (not (null constraints)))
(destructuring-bind (class slot relation &rest values) (first constraints)
(flet ((filter-by-relation (inst)
1
0
Update of /project/elephant/cvsroot/elephant/doc
In directory clnet:/tmp/cvs-serv22287
Modified Files:
tutorial.texinfo user-guide.texinfo
Log Message:
Documentation changes, mostly to transaction section of tutorial
--- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11
+++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 20:22:24 1.12
@@ -732,6 +732,8 @@
transaction that performs all the updates atomically and thus
enforcing consistency.
+@subsection Why do we need Transactions?
+
Most real applications will need to use explicit transactions rather
than relying on the primitives alone because you will want multiple
read-modify-update operations act as an atomic unit. A good example
@@ -815,6 +817,8 @@
And presto, we have an ACID compliant, thread-safe, persistent banking
system!
+@subsection Using @code{with-transaction}
+
What is @code{with-transaction} really doing for us? It first starts
a new transaction, attempts to execute the body, and if successful
commit the transaction. If anywhere along the way there is a deadlock
@@ -823,14 +827,145 @@
to retry the transaction a fixed number of times by re-executing the
whole body.
-The other value transactions provide is the capability to delay
-flushing dirty data to disk. The most time-intensive part of
-persistent operations is flushing newly written data to disk. Using
-the default auto-commit behavior requires a flush on every operation
-which can become very expensive. Because a transaction caches values,
-all the values read or written are cached in memory until the
-transaction completes, dramatically decreasing the number of flushes
-and the total time taken.
+And this brings us to two important caveats: nested transactions and
+idempotent side-effects.
+
+@subsection Nesting Transactions
+
+In general, you want to avoid nesting @code{with-transaction}
+statements. Nested transactions are valid for some data stores
+(namely Berkeley DB), but typically only a single transaction can be
+active at a time. The purpose of a nested transaction in data stores
+that provide it, is break a long transaction into chunks. This way if
+there is contention on a given subset of variables, only the inner
+transaction is restarted while the larger transaction can continue.
+When commit their results, those results become part of the outer
+transaction until it in turn commits.
+
+If you have transaction protected primitive operations (such as
+@code{deposit} and @code{withdraw}) and you want to perform a group of
+such transactions, for example a transfer between accounts, you can
+use the macro @code{ensure-transaction} instead of @code{with-transaction}.
+
+@lisp
+(defun deposit (account amount)
+ "Wrap the balance read and the setf with the new balance"
+ (ensure-transaction ()
+ (let ((balance (balance account)))
+ (setf (balance account)
+ (+ balance amount)))))
+
+(defun deposit (account amount)
+ "A more concise version with decf doing both read and write"
+ (ensure-transaction ()
+ (decf (balance account) amount)))
+
+(defun withdraw (account amount)
+ (ensure-transaction ()
+ (decf (balance account) amount)))
+
+(defun transfer (src dst amount)
+ "There are four primitive read/write operations
+ grouped together in this transaction"
+ (with-transaction ()
+ (withdraw src amount)
+ (deposit dst amount)))
+@end lisp
+
+@code{ensure-transaction} is exactly like @code{with-transaction}
+except it will reuse an existing transaction, if there is one, or
+create a new one. There is no harm, in fact, in using this macro all
+the time.
+
+Notice the use of @code{decf} and @code{incf} above. The primary
+reason to use Lisp is that it is good at hiding complexity using
+shorthand constructs just like this. This also means it is also going
+to be good at hiding data dependencies that should be captured in a
+transaction!
+
+@subsection Idempotent Side Effects
+
+Within the body of a with-transaction, any non database operations
+need to be @emph{idempotent}. That is the side effects of the body
+must be the same no matter how many times the body is executed. This
+is done automatically for side effects on the database, but not for
+side effects like pushing a value on a lisp list, or creating a new
+standard object.
+
+@lisp
+(defparameter *transient-objects* nil)
+
+(defun load-transients (n)
+ "This is the wrong way!"
+ (with-transaction ()
+ (loop for i from 0 upto n do
+ (push (get-from-root i) *transient-objects*))))
+@end lisp
+
+In this contrived example we are pulling a set of standard objects
+from the database using an integer key and pushing them onto a list
+for later use. However, if there is a conflict where some other
+process writes a key-value pair to a matching key, the whole
+transaction will abort and the loop will be run again. In a heavily
+contended system you might see results like the following.
+
+@lisp
+(defun test-list ()
+ (setf *transient-objects* nil)
+ (load-transients)
+ (length *transient-objects*))
+
+(test-list)
+=> 3
+
+(test-list)
+=> 5
+
+(test-list)
+=> 4
+@end lisp
+
+So the solution is to make sure that the operation on the lisp
+parameters is atomic if the transaction completes.
+
+@lisp
+(defun load-transients ()
+ "This is a better way"
+ (setq *transient-objects*
+ (with-transaction ()
+ (loop for i from 0 upto 3 collect
+ (get-from-root i)))))
+@end lisp
+
+Of course we would need to use @code{nreverse} if we cared about the
+order of instances in @code{*transient-objects*}. The best rule of
+thumb is that transaction bodies should be purely functional as above,
+except for side effects to the persistent store such as persistent
+slot writes, adding to btrees, etc).
+
+If you do need side effects to lisp memory, such as writes to
+transient slots, make sure they are idempotent and that other
+processes will not be reading the written values until the transaction
+completes.
+
+@subsection Transactions and Performance
+
+By now transactions almost look like more work than they are worth!
+Well there are still some significant benefits to be had. Part of how
+transactions are implemented is that they gather together all the
+writes that are supposed to made to the database and store them until
+the transaction commits, and then writes them atomically.
+
+The most time-intensive part of persistent operations is flushing
+newly written data to disk. Using the default auto-committing
+behavior requires a flush for every primitive write operation. This
+can become very expensive! Because all the values read or written are
+cached in memory until the transaction completes, the number of
+flushes can be dramatically reduced.
+
+But don't take my word for it, run the following statements and see
+for yourself the visceral impact transactions can have on system
+performance.
@lisp
(defpclass test ()
@@ -872,52 +1007,42 @@
thumb is to keep the number of objects touched in a transaction well
under 1000.
-And this brings us to the last caveat we'll introduce in this
-introductory tutorial: nested transactions.
-
-In general, avoid nesting transactions. Nested transactions are valid
-for some data stores (namely Berkeley DB), but typically only a single
-transaction is valid at a time. The purpose of a nested transaction
-is to allow a long transaction to be broken up into chunks. This way
-if there is contention on a given subset of variables, only the
-subtransaction is restarted while the larger transaction can continue.
-Subtransactions commit their results and they become part of the
-outer transaction until it in turn commits.
-
-If you have transaction protected primitive operations (such as
-@code{deposit} and @code{withdraw}) and you want to perform a group of
-such transactions, for example a transfer between accounts, you can
-use the macro @code{ensure-transaction} instead of @code{with-transaction}.
-
-@lisp
-(defun deposit (account amount)
- (ensure-transaction ()
- (let ((balance (balance account)))
- (setf (balance account)
- (+ balance amount)))))
-
-(defun withdraw (account amount)
- (ensure-transaction ()
- (decf (balance account) amount)))
-
-(defun transfer (src dst amount)
- (with-transaction ()
- (withdraw src amount)
- (deposit dst amount)))
-@end lisp
-
-@code{ensure-transaction} is exactly like @code{with-transaction}
-except it will reuse an existing transaction, if there is one, or
-create a new one. There is no harm, in fact, in using this macro all
-the time.
+@subsection Transactions and Applications
Designing and tuning a transactional architecture can become quite
-complicated. The best strategy at the beginning is a conservative
-one, break things up into the smallest logical sets of primitive
-operations and only wrap higher level functions in transactions when
-they absolutely have to commit together. See @ref{Transaction Details}
-for the full details and @pxref{Usage Scenarios} for more examples of
-how systems can be designed and tuned using transactions.
+complex. Moreover, bugs in your system can be very difficult to find
+as they only show up when transactions are interleaved within a
+larger, multi-threaded application.
+
+In many cases, however, you can ignore transactions. For example,
+when you don't have any other concurrent processes running. In this
+case all operations are sequential and there is no chance of
+conflicts. You would only want to use transactions for write
+performance.
+
+You can also ignore transactions if your application can guarantee
+that concurrency won't generate any conflicts. For example, a web app
+that guarantees only one thread will write to objects in a particular
+session can avoid transactions altogether. However, it is good to be
+careful about making these assumptions. In the above example, a
+reporting function that iterates over sessions, users or other objects
+may still see partial updates (i.e. a user's id was written prior to
+the query, but not the name). However, if you don't care about these
+infrequent glitches, this case would still hold.
+
+If these cases don't apply to your application, or you aren't sure,
+you will fare best by programming defensively. Break your system into
+the smallest logical sets of primitive operations
+(i.e. @code{withdraw} and @code{deposit}) using
+@code{ensure-transaction} and then wrap the highest level calls made
+to your system in with-transaction when the operations absolutely have
+to commit together or you need the extra performance. Try not to have
+more than two levels of transactional accesses with the top using
+with-transaction and the bottom using ensure-transaction.
+
+@xref{Transaction Details} for more details and @pxref{Usage
+Scenarios} for examples of how systems can be designed and tuned using
+transactions.
@node Advanced Topics
@comment node-name, next, previous, up
--- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5
+++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:22:24 1.6
@@ -23,26 +23,6 @@
* Performance Tuning:: How to get the most from Elephant.
@end menu
-@node Persistent objects
-@comment node-name, next, previous, up
-@section Persistent Objects
-
-Finally, if you for some reason make an instance with a specified OID
-which already exists in the database, @code{initargs} take precedence
-over values in the database, which take precedences over
-@code{initforms}.
-
-Also currently there is a bug where
-@code{initforms} are always evaluated, so beware.
-(What is the current model here?)
-
-Readers, writers, accessors, and @code{slot-value-using-class} are
-employed in redirecting slot accesses to the database, so override
-these with care. Because @code{slot-value, slot-boundp,
-slot-makunbound} are not generic functions, they are not guaranteed by
-the specification to work properly with persistent slots. However the
-proper behavior has been verified on SBCL, Allegro and Lispworks.
-
@node The Store Controller
@comment node-name, next, previous, up
@section The Store Controller
@@ -90,6 +70,26 @@
Empty.
+@node Persistent objects
+@comment node-name, next, previous, up
+@section Persistent Objects
+
+Finally, if you for some reason make an instance with a specified OID
+which already exists in the database, @code{initargs} take precedence
+over values in the database, which take precedences over
+@code{initforms}.
+
+Also currently there is a bug where
+@code{initforms} are always evaluated, so beware.
+(What is the current model here?)
+
+Readers, writers, accessors, and @code{slot-value-using-class} are
+employed in redirecting slot accesses to the database, so override
+these with care. Because @code{slot-value, slot-boundp,
+slot-makunbound} are not generic functions, they are not guaranteed by
+the specification to work properly with persistent slots. However the
+proper behavior has been verified on SBCL, Allegro and Lispworks.
+
@node Class Indices
@comment node-name, next, previous, up
@section Class Indices
@@ -141,6 +141,111 @@
@comment node-name, next, previous, up
@section Querying persistent instances
+
+
+A SQL select-like interface is in the works, but for now queries are
+limited to manual mapping over class instances or doing small queries
+with @code{get-instances-*} functions. One advantage of this is that
+it is easy to estimate the performance costs of your queries and to
+choose standard and derived indices that give you the ordering and
+performance you want.
+
+There is, however, a quick and dirty query API example that is not
+officially supported in the release but is intended to invite comment.
+This is an example of a full query system that would automatically
+perform joins, use the appropriate indices and perhaps even adaptively
+suggest or add indices to facilitate better performance on common
+queries.
+
+There are two functions @ref{Function elephant:get-query-instances}
+and @ref{Function elephant:map-class-query} which accept a set of
+constraints instead of the familiar value or range arguments.
+
+We'll use the classes @code{person} and @code{department} to
+illustrate how to perform queries over a set of objects that may be
+constrainted by their relationships to other objects.
+
+@lisp
+(defpclass person ()
+ ((name :initarg :name :index t)
+ (salary :initarg :salary :index t)
+ (department :initarg :dept)))
+
+(defmethod print-object ((p person) stream)
+ (format stream "#<PERS: ~A>" (slot-value p 'name)))
+
+(defun print-name (inst)
+ (format t "Name: ~A~%" (slot-value inst 'name)))
+
+(defpclass department ()
+ ((name :initarg :name)
+ (manager :initarg :manager)))
+
+(defmethod print-object ((d department) stream)
+ (format stream "#<DEPT ~A, mgr = ~A>"
+ (slot-value d 'name)
+ (when (slot-boundp d 'manager)
+ (slot-value (slot-value d 'manager) 'name))))
+@end lisp
+
+Here we have a simple employee database with managers (also of type
+person) and departments. This simple system will provide fodder for
+some reasonably complex constraints. Let's create a few departments.
+
+@lisp
+(setf marketing (make-instance 'department :name "Marketing"))
+(setf engineering (make-instance 'department :name "Engineering"))
+(setf sales (make-instance 'department :name "Sales"))
+@end lisp
+
+And manager @code{people} for the departments.
+
+@lisp
+(make-instance 'person :name "George" :salary 140000 :department marketing)
+(setf (slot-value marketing 'manager) *)
+
+(make-instance 'person :name "Sally" :salary 140000 :department engineering)
+(setf (slot-value engineering 'manager) *)
+
+(make-instance 'person :name "Freddy" :salary 180000 :department sales)
+(setf (slot-value sales 'manager) *)
+@end lisp
+
+And of course we need some folks to manage
+
+@lisp
+(defparameter *names*
+ '("Jacob" "Emily" "Michael" "Joshua" "Andrew" "Olivia" "Hannah" "Christopher"))
+
+(defun random-element (list)
+ "Choose a random element from the list and return it"
+ (nth (random (length list)) list))
+
+(with-transaction ()
+ (loop for i from 0 upto 40 do
+ (make-instance 'person
+ :name (format nil "~A~A" (random-elephant *names*) i)
+ :salary (floor (+ (* (random 1000) 100) 30000))
+ :department (case (random 3)
+ (0 marketing)
+ (1 engineering)
+ (2 sales)))))
+@end lisp
+
+Due to the random allocation of
+In the follwoing examples below, the results will be different due to the random
+allocation of employee names, etc. However, these examples are
+illustrative of what you should see if you run the same code.
+
+
+
+For those familiar with SQL, if an instance of @code{person} has a
+pointer to an instance of @code{department} then that relation can be
+used to perform a join. Of course joins in the object world won't
+return a table, instead they will return conjunctions of objects that
+satisfy a mutual set of constraints.
+
+
@node Using BTrees
@comment node-name, next, previous, up
@section Using BTrees
@@ -174,6 +279,14 @@
@comment node-name, next, previous, up
@section Transaction Details
+You can trace @code{elephant::execute-transaction} to see the sequence
+of calls to @code{execute-transaction} that occur dynamically and
+detect where transactions are and are not happening. We may add some
+transaction diagnosis and tracing tools in the future, such as
+throwing a condition when @code{with-transaction} forms are nested
+dynamically.
+
+
;; Transaction architecture:
;;
;; User and designer considerations:
1
0
Update of /project/elephant/cvsroot/elephant/src/elephant
In directory clnet:/tmp/cvs-serv29083/src/elephant
Modified Files:
classes.lisp controller.lisp metaclasses.lisp serializer.lisp
Log Message:
Latest documentation changes
--- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/30 23:36:53 1.25
+++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/01 14:33:34 1.26
@@ -39,9 +39,11 @@
(defclass persistent-object (persistent) ()
(:metaclass persistent-metaclass)
(:documentation
- "Superclass of all user-defined persistent classes. This is
+ "Superclass for all user-defined persistent classes. This is
automatically inherited if you use the persistent-metaclass
- metaclass."))
+ metaclass. This allows specialization of functions for user
+ objects that would not be appropriate for Elephant objects
+ such as persistent collections"))
;; ================================================
;; METACLASS INITIALIZATION AND CHANGES
--- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:42:35 1.44
+++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/01 14:33:34 1.45
@@ -48,11 +48,13 @@
(defvar *dbconnection-spec* (make-hash-table :test 'equal))
(defvar *dbconnection-lock* (ele-make-lock))
-(defmethod get-con ((instance persistent) &optional (sc *store-controller*))
- "This is used to find and validate the connection spec
+(defgeneric get-con (instance &optional sc)
+ (:documentation "This is used to find and validate the connection spec
maintained for in-memory persistent objects. Should
we re-open the controller from the spec if it's not
- cached? That might be dangerous so for now we error"
+ cached? That might be dangerous so for now we error"))
+
+(defmethod get-con ((instance persistent) &optional (sc *store-controller*))
(declare (ignore sc))
(let ((con (gethash (dbcn-spc-pst instance) *dbconnection-spec*)))
(cond ((not con)
--- /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/03/30 23:36:53 1.14
+++ /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/04/01 14:33:34 1.15
@@ -29,7 +29,7 @@
:documentation "Persistent objects use a spec pointer to identify which store
they are connected to"))
(:documentation "Abstract superclass for all persistent classes (common
- to user-defined classes and collections.)"))
+ to both user-defined classes and Elephant-defined objects such as collections.)"))
(defclass persistent-metaclass (standard-class)
((%persistent-slots :accessor %persistent-slots)
--- /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/03/30 14:34:35 1.26
+++ /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/04/01 14:33:34 1.27
@@ -47,6 +47,8 @@
;;
(defun serialize-to-base64-string (x sc)
+ "Encode object using the store controller's serializer format,
+ but encoded in a base64"
(with-buffer-streams (out-buf)
(cl-base64::usb8-array-to-base64-string
(elephant-memutil::buffer-read-byte-vector
@@ -62,6 +64,7 @@
)
(defun deserialize-from-base64-string (x sc)
+ "Decode a base64-string using the store controller's deserialize method"
(with-buffer-streams (other)
(deserialize
(elephant-memutil::buffer-write-byte-vector
1
0
Update of /project/elephant/cvsroot/elephant/doc
In directory clnet:/tmp/cvs-serv29083/doc
Modified Files:
Makefile data-store-reference.texinfo elephant-design.texinfo
elephant.texinfo installation.texinfo reference.texinfo
tutorial.texinfo user-guide.texinfo
Log Message:
Latest documentation changes
--- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/30 23:36:52 1.5
+++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/01 14:33:24 1.6
@@ -9,4 +9,4 @@
makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo
pdf: includes-stuff
- texi2dvi --pdf elphant.texinfo
+ texi2dvi --pdf elephant.texinfo
--- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 23:36:52 1.4
+++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5
@@ -3,9 +3,8 @@
@node Data Store API Reference
@comment node-name, next, previous, up
@chapter Data Store API Reference
-@cindex Data Store API Reference
@cindex Data Store
-@cindex API Reference
+@cindex API
This reference includes functions that need to be overridden, classes
inherited from or other action taken to implement support for a new
@@ -26,16 +25,16 @@
relevant to them.
@menu
-* Registration:: Register the backend to parse controller specifications.
-* Store Controllers:: Subclassing the store controller.
-* Handling Serialization:: Available facilities for serializing objects.
-* C Utilities:: Writing primitive C types.
-* Slot access:: Support for metaprotocol slot access.
-* Collections:: BTrees and indices.
-* Cursors:: Traversing BTrees.
-* Transactions:: Transaction implementation.
-* Multithreading:: Multithreading considerations.
-* Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries
+* Registration: DSR Registration. Register the backend to parse controller specifications.
+* Store Controllers: DSR Store Controllers. Subclassing the store controller.
+* Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects.
+* Memory Utilities: DSR Memory Utilities. Writing primitive C types.
+* Persistent Objects and Slot access: DSR Persistent Objects and Slot Access. Support for metaprotocol slot access.
+* Collections: DSR Collections. BTrees and indices.
+* Cursors: DSR Cursors. Traversing BTrees.
+* Transactions: DSR Transactions. Transaction implementation.
+* Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations.
+* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries
@end menu
@node DSR Registration
@@ -96,17 +95,28 @@
@section Slot Access
@cindex Persistent Objects and Slot Access
+Persistence is implement with a metaclass and several required base
+classes.
+
+@include includes/class-elephant-persistent-metaclass.texinfo
@include includes/class-elephant-persistent.texinfo
+@include includes/class-elephant-persistent-object.texinfo
+
+Persistent objects can be queries for their home store controller so
+that functions such as map-btree do not need a store-controller
+argument. (NOTE: Should this function be user visible?)
+
@include includes/fun-elephant-backend-get-con.texinfo
-@c @include includes/fun-elephant-backend-oid.texinfo
-All objects require a unique id. During new object creation the
-backend is asked to produce a unique id.
+All objects require a unique object identifier. During new object
+creation the backend is asked to produce a unique id.
@include includes/fun-elephant-backend-next-oid.texinfo
-These functions are called by the metaclass protocol to support
-operations on persistent class slots.
+These functions are called by the metaclass protocol to implement the
+appropriate operations on persistent class slots. Unless protected by
+a transaction, the side effects of these functions should be atomic,
+persistent and visible to other threads on completion.
@include includes/fun-elephant-backend-persistent-slot-writer.texinfo
@include includes/fun-elephant-backend-persistent-slot-reader.texinfo
@@ -121,7 +131,9 @@
To support collections, the data store must subclass the following
classes.
-@include includes/class-elephant-btree.texinfo.texinfo
+@include includes/class-elephant-persistent-collection.texinfo
+
+@include includes/class-elephant-btree.texinfo
@include includes/class-elephant-btree-index.texinfo
@include includes/class-elephant-indexed-btree.texinfo
@@ -131,42 +143,48 @@
@include includes/fun-elephant-build-btree.texinfo
@include includes/fun-elephant-build-indexed-btree.texinfo
-And every btree needs accessors, these must be implemented for btree,
-indexed-btree and btree-index.
+Most of the user-visible operations over BTrees must be implemented.
+Class indexing functions such as @code{map-class} and
+@code{get-instances-by-value} and related functions are all
+implemented using map-btree and map-index.
+
+@itemize
+@item @ref{Generic-Function elephant:get-value}
+@item @ref{Generic-Function (setf elephant:get-value)}
+@item @ref{Generic-Function elephant:existsp}
+@item @ref{Generic-Function elephant:remove-kv}
+@item @ref{Generic-Function elephant:get-index}
+@item @ref{Generic-Function elephant:remove-index}
+@item @ref{Generic-Function elephant:map-btree}
+@item @ref{Generic-Function elephant:map-index}
+@end itemize
-@include includes/fun-elephant-get-value.texinfo
-@include includes/fun-elephant-setf-get-value.texinfo
-@include includes/fun-elephant-existsp.texinfo
-@include includes/fun-elephant-remove-kv.texinfo
+Mapping over the indices of a btree is important to derived facilities
+such as class indexing and the query subsystem.
@include includes/fun-elephant-map-indices.texinfo
-@include includes/fun-elephant-get-index.texinfo
-@include includes/fun-elephant-remove-index.texinfo
-
-Critical to indexing and queries are the map operators for collections
-
-@include includes/fun-elephant-map-btree.texinfo
-@include includes/fun-elephant-map-index.texinfo
@node DSR Cursors
@comment node-name, next, previous, up
@section Cursors
@cindex Cursors
-@include includes/class-cursor.texinfo
-@c #:cursor-btree
-@c #:cursor-oid
-@c #:cursor-initialized-p
+Data stores must subclass these cursor classes and implement all the
+methods described in @ref{Cursors} except @ref{Macro
+elephant:with-btree-cursor}.
+
+@include includes/class-elephant-cursor.texinfo
+@include includes/class-elephant-secondary-cursor.texinfo
@node DSR Transactions
@comment node-name, next, previous, up
@section Transactions
@cindex Transactions
-These functions must be implemented or stubbed in any
-backend.
+These functions must be implemented or stubbed by all data stores.
@include includes/fun-elephant-backend-execute-transaction.texinfo
+
@include includes/fun-elephant-backend-controller-start-transaction.texinfo
@include includes/fun-elephant-backend-controller-commit-transaction.texinfo
@include includes/fun-elephant-backend-controller-abort-transaction.texinfo
@@ -179,33 +197,77 @@
@include includes/fun-elephant-backend-transaction-store.texinfo
@include includes/fun-elephant-backend-transaction-object.texinfo
+;; Designer considerations:
+;; - with-transaction passes *current-transaction* or the user parameter to execute-transaction
+;; in the parent keyword argument. Backends allowing nested transactions can treat the transaction
+;; as a parent, otherwise they can reuse the current transaction by ignoring it (inheriting the dynamic
+;; value of *current-transaction*) or rebinding the dynamic context (whatever makes coding easier).
+;; - ensure-transaction uses *current-transaction* to determine if there is a current transaction
+;; in progress (not null). If so, it jumps to the body directly. Otherwise it executes the body in a
+;; new transaction by calling ...
+;; - execute-transaction contract:
+;; - Backends must dynamically bind *current-transaction* to a meaningful identifier for the
+;; transaction in progress and execute the provided closure in that context
+;; - All non-local exists result in an abort; only regular return values result in a commit
+;; - If a transaction is aborted due to a deadlock or read conflict, execute-transaction should
+;; automatically retry with an appropriate default amount
+;; - execute-transaction can take any number of backend-defined keywords, although designers should
+;; make sure there are no semantic conflicts if there is a name overlap with existing backends
+;; - A typical design approach is to make sure that the most primitive interfaces to the backend
+;; database look at *current-transaction* to determine whether a transaction is active. Users code can also
+;; access this parameter to check whether a transaction is active.
+
@node DSR Multithreading Considerations
@comment node-name, next, previous, up
@section Multithreading Considerations
@cindex Multithreading
-@c utils locks
-@c utils thread-vars
+Generic locking utility functions
+
+Variable behavior in multithreading situations
@node DSR Handling Serialization
@comment node-name, next, previous, up
@section Handling Serialization
@cindex Serialization
-@c #:deserialize #:serialize
-@c #:serialize-symbol-complete
-@c #:deserialize-from-base64-string
-@c #:serialize-to-base64-string
+Backends must initialize @ref{Class elephant:store-controller} with
+internal serializer functions. Packages @code{elephant-serializer1}
+and @code{elephant-serializer2} contains serialize and deserialize
+methods on buffer-streams as defined in @code{elephant-memutil}. The
+elephant functions @code{serialize} and @code{deserialize} dispatch on
+the appropriate slot values of the store-controller.
+
+@verbatim
+NOTE: This should perhaps become entirely the job of the data store to
+decide how to serialize values and for a specific version, what
+serializer to use. The elphant main package can define serializers
+for use by different backends.
+@end verbatim
+
+@include includes/fun-elephant-backend-serialize.texinfo
+@include includes/fun-elephant-backend-deserialize.texinfo
+
+These utility functions are useful if a data store does not have the
+ability to store variable length binary data. They are based on the
+@code{cl-base64} library.
+
+@include includes/fun-elephant-backend-serialize-to-base64-string.texinfo
+@include includes/fun-elephant-backend-deserialize-from-base64-string.texinfo
-@node DSR Memory utilities
+@node DSR Memory Utilities
@comment node-name, next, previous, up
@section Memory utilities
@cindex Memory utilities
-@node DSR Foreign libraries
+Details about memory utilities here.
+
+@node DSR Foreign Libraries
@comment node-name, next, previous, up
@section Foreign libraries
@cindex Foreign libraries
+How foreign libraries are built and used via UFFI. What functions are
+in the .asd files or main lisp code to build & load libraries?
--- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/30 14:34:34 1.2
+++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/01 14:33:29 1.3
@@ -1,9 +1,37 @@
+@c -*-texinfo-*-
@node Elephant Design
@comment node-name, next, previous, up
-@section Elephant Design
+@chapter Elephant Design
@cindex design
+Elephant's early architecture was tightly coupled to the Berkeley DB
+API. Over time we've moved towards a more modular architecture to
+support easy upgrading, repository migration, shared functionality
+between data stores and general hygene.
+
+The current architecture is modularized into the following pieces:
+
+@verbatim
+[Picture describing] Metaclass, Store controller, persistent objects,
+data stores, serializer & memutils. Derived features such as class
+indexing, migration, query system and root operations can also be
+illustrated?
+@end verbatim
+
+To get a feeling for what is happening inside elephant, it is probably
+most useful to walk through the various major protocols:
+
+@itemize
+@item Initialization of a store controller
+@item Creating a persistent object
+@item Operations on persistent slots
+@item Operations on persistent collections
+@item Implementing @code{with-transaction}
+@end itemize
+
+@section Initializing a store controller
+
When the main elephant @code{open-store} function is called, it calls
@code{get-controller} which grabs an existing store controller if the
spec is identical, or builds a new controller. Building the
@@ -13,4 +41,14 @@
that returns a @code{store-controller} subclass instance specific to
that backend.
-Elephant than calls open-controller
+Elephant than calls open-controller to actually establish a connection
+to or create the files of the data store.
+
+@section Persistent Object Creation
+@section Persistent Slot Protocol
+@section Persistent Slot Protocol
+@section Persistent Collection Protocol
+@section Implementing Transactions
+
+
+
--- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/30 14:34:34 1.6
+++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/01 14:33:29 1.7
@@ -38,6 +38,7 @@
@insertcopying
@end ifnottex
+@ifhtml
@menu
* Table of Contents::
@end menu
@@ -49,7 +50,7 @@
* Tutorial:: A basic ``getting started'' tutorial.
* Installation:: Installation and test-suite procedures.
* User Guide:: In depth discussion of all Elephant facilities and features.
-* Usage scenarios:: Design scenarios for Elephant applications.
+* Usage Scenarios:: Design scenarios for Elephant applications.
* User API Reference:: Function and class documentation of the user API.
* Elephant Design:: An overview of elephant's internal architecture.
* Data Store API Reference:: Function level documentation for data store implementors.
@@ -66,6 +67,8 @@
* Colophon::
@end menu
+@end ifhtml
+
@node Table of Contents
@unnumbered
@comment node-name, next, previous, up
--- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/30 14:34:34 1.5
+++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6
@@ -10,9 +10,9 @@
* Configuring Elephant:: Setting up Elephant and the configuration file.
* Loading Elephant:: Loading Elephant and the data store loading protocol.
* Berkeley DB Data Store:: Installing support for the Berkeley DB data store
-* Berkeley DB Examples:: An example of installing and running the Berkeley DB data store.
+* Berkeley DB Example:: An example of installing and running the Berkeley DB data store.
* CL-SQL Data Store:: Install and connecting to the CL-SQL data store
-* CL-SQL Examples:: An example of using the CL-SQL data store.
+* CL-SQL Example:: An example of using the CL-SQL data store.
* Elephant on Windows:: More details about running Elephant on Windows
* Test Suites:: How to run and interpret the output of the regression test suite
* Documentation:: Building documentation from texinfo sources.
@@ -277,10 +277,6 @@
As of Elephant 0.3, Elephant has been tested to work with both Postgres, and
SQLite 3, thanks do Dan Knapp.
-@node Extension Status
-@comment node-name, next, previous, up
-@section Extension Status
-
As far as is known at this writing, all functionality except nested transaction
support and cursor-puts supported by the BerkeleyDB backend is supported by the
CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested
@@ -302,13 +298,9 @@
the multi-repository version somewhat complicates the underlying
persistent object management.
-@node PostGres Examples
-@comment node-name, next, previous, up
-@section Setting up PostGres
-
-@node Setting up PostGres
+@node CL-SQL Example
@comment node-name, next, previous, up
-@section Setting up PostGres
+@section CL-SQL Example
To set up a PostGres based back end, you should:
@@ -351,6 +343,10 @@
sequences, and so on needed by the Elephant system will be created in the
schema named ``test'' automatically.
+@node Elephant on Windows
+@comment node-name, next, previous, up
+@section Elephant on Windows
+
@node Test Suites
@comment node-name, next, previous, up
@section Test Suites
--- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 23:36:52 1.8
+++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/01 14:33:29 1.9
@@ -22,14 +22,20 @@
@section Store Controllers
@cindex Store Controllers
-Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller.
+Store controllers provide the persistent storage for CLOS objects and
+BTree collections. Any persistent operations must be done in the
+context of a store controller. The default store-controller is stored
+in a global variable.
-@include includes/class-elephant-store-controller.texinfo
@include includes/var-elephant-star-store-controller-star.texinfo
+@c @include includes/class-elephant-store-controller.texinfo
+@ref{Class elephant:store-controller} is associated with the following
+user methods and macros:
+
+@include includes/macro-elephant-with-open-store.texinfo
@include includes/fun-elephant-open-store.texinfo
@include includes/fun-elephant-close-store.texinfo
-@include includes/macro-elephant-with-open-store.texinfo
@include includes/fun-elephant-get-from-root.texinfo
@include includes/fun-elephant-add-to-root.texinfo
@@ -42,12 +48,12 @@
@section Persistent Objects
@cindex Persistent Objects
-@include includes/class-elephant-persistent-metaclass.texinfo
-@include includes/class-elephant-persistent.texinfo
-@include includes/class-elephant-persistent-object.texinfo
+@ref{Class elephant:persistent-metaclass} can be used as the
+:metaclass argument in a defclass form to create a persistent object.
+Slots of the metaclass take the :index and :transient keyword
+arguments and the class accepts the :index keyword argument.
@include includes/macro-elephant-defpclass.texinfo
-
@include includes/fun-elephant-drop-pobject.texinfo
@node Persistent Object Indexing
@@ -94,22 +100,26 @@
interface is provided for reference only, a new system is under
development for the 0.7 release.
-@include includes/fun-elephant-get-query-results.texinfo
-@include includes/fun-elephant-map-class-query.texinfo
+@c @include includes/fun-elephant-get-query-results.texinfo
+@c @include includes/fun-elephant-map-class-query.texinfo
@node Collections
@comment node-name, next, previous, up
@section Collections
@cindex Collections
-@include includes/class-elephant-persistent-collection.texinfo
-@include includes/class-elephant-btree.texinfo
-@include includes/class-elephant-indexed-btree.texinfo
-@include includes/class-elephant-btree-index.texinfo
+Persistent collections inherit from @ref{Class
+elephant:persistent-collection} and consist of the @ref{Class
+elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class
+elephant:btree-index} classes. The following operations are defined
+on most of these classes. More information can be found in @ref{Using
+BTrees} and @ref{Secondary Indices}.
@include includes/fun-elephant-get-value.texinfo
@include includes/fun-elephant-setf-get-value.texinfo
@include includes/fun-elephant-remove-kv.texinfo
+@include includes/fun-elephant-existsp.texinfo
+
@include includes/fun-elephant-map-btree.texinfo
@include includes/fun-elephant-map-index.texinfo
@@ -118,17 +128,17 @@
@include includes/fun-elephant-get-primary-key.texinfo
@include includes/fun-elephant-remove-index.texinfo
-
@node Cursors
@comment node-name, next, previous, up
@section Cursors
@cindex Cursors
-@include includes/class-elephant-cursor.texinfo
-@include includes/class-elephant-secondary-cursor.texinfo
+Cursors are objects of type cursor (@pxref{Class elephant:cursor})
+which provide methods for complex traversals of BTrees.
+
+@include includes/macro-elephant-with-btree-cursor.texinfo
@include includes/fun-elephant-make-cursor.texinfo
@include includes/fun-elephant-cursor-close.texinfo
-@include includes/macro-elephant-with-btree-cursor.texinfo
@include includes/fun-elephant-cursor-current.texinfo
@include includes/fun-elephant-cursor-delete.texinfo
--- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/30 14:34:34 1.10
+++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11
@@ -15,6 +15,7 @@
* Persistent collections:: Keep track of collections of values.
* Indexing Persistent Classes:: Simple way to keep track of persistent instances.
* Using Transactions:: Providing ACID database properties.
+* Advanced Topics:: Additional Elephant features covered in the User Guide.
@end menu
@node Overview
@@ -704,14 +705,14 @@
@end lisp
The @ref{User Guide} contains a descriptions of the advanced features
-of @ref{Class indices} such as ``derived indicies'' that allow you to
+of @ref{Class Indices} such as ``derived indicies'' that allow you to
order classes according to an arbitrary function, a dynamic API for
adding and removing slots and how to set a policy for resolving
conflicts between the code image and a database where the indexing
specification differs.
This same facility is also available for your own use. For more
-information @pxref{Using Indexed BTrees}.
+information @pxref{Secondary Indices}.
@node Using Transactions
@@ -914,8 +915,71 @@
complicated. The best strategy at the beginning is a conservative
one, break things up into the smallest logical sets of primitive
operations and only wrap higher level functions in transactions when
-they absolutely have to commit together. See @ref{Transaction details}
-for the full details and @pxref{Usage scenarios} for more examples of
+they absolutely have to commit together. See @ref{Transaction Details}
+for the full details and @pxref{Usage Scenarios} for more examples of
how systems can be designed and tuned using transactions.
+@node Advanced Topics
+@comment node-name, next, previous, up
+@section Advanced Topics
+
+The tutorial covers the essential topics and concepts for using
+Elephant. Many people will find that these features are the ones that
+are most often needed and used in ordinary applications.
+
+More sophisticated uses of Elephant may require additional features
+that are covered in the user guide. The following is a list of major
+features in the user guide that were not covered in this tutorial.
+
+@itemize
+@item @strong{Class Heirarchies and Queries}
+ There are some subtle issues to take into account when querying
+persistent classes. For example, how do you query a base class of
+type people to get subclass instances such as employee, manager,
+consultant, etc?
+@item @strong{Derived Class Indices}
+ You can create your own indices for classes that are arbitrary
+lisp functions of the persistent object.
+@item @strong{Class Definition/Database Conflict Resolution}
+ When you startup lisp, there are potential conflicts between the
+class definition and the indexing records in the database. There are
+some constraints to account for and some facilities to manage
+how slots, class indices and
+@item @strong{Dynamic Class Index Management}
+ It is possible to add and remove indexes from classes at runtime.
+@item @strong{BTree Cursors}
+ If you need to do more than iterate over a collection, or you need
+to delete elements of the collection as you iterate cursors are an
+important data structure. They implement a variety of operators for
+moving backward and forward over a btree, including ranged operations
+and iterating of duplicate or unique values.
+@item @strong{Indexed BTrees}
+ Indexed BTrees are just like BTrees, except it is possible to add
+indexes which are BTrees who's values are primary keys in the parent
+@code{indexed-btree}. This allows for multiple ordering and groupings
+of the values of a BTree.
+@item @strong{Using the Map Operators}
+ Mapping operators can be very efficient if properly utilized.
+@item @strong{Handling Errors and Conditions}
+ There are a variety of errors that can occur in Elephant that need
+to be dealt with by applications.
+@item @strong{Deadlock Detection in Berkeley DB}
+ Berkeley DB requires an external process to detect deadlock
+conditions among transactions. The :deadlock-detect keyword argument
+to open-store for Berkeley DB specs will launch this process on most
+lisps.
+@item @strong{Using Multiple Stores}
+ Multiple store controllers can be open simultaneously. However it
+does make the code more complex and you need to be careful about how
+you use them to avoid crashes and other unpleasant side effects.
+@item @strong{Custom Transaction Architecture}
+ You can implement your own version of with-transaction using the
+underlying controller methods for starting, aborting and committing
+transactions. You had better know what you are doing, however!
+@item @strong{Using Multiple Threads and Processes}
+ What constraints must be accommodated to use Elephant data stores in
+multiple threads? What capabilities are there to share data stores
+among multiple processes or machines?
+@end itemize
+Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions.
--- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/30 14:34:34 1.4
+++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5
@@ -8,18 +8,19 @@
@menu
* The Store Controller:: Behind the curtain.
* Serialization details:: The devil hides in the details.
-* Reachability:: Determining liveness in a store.
* Persistent objects:: All the dirt on persistent objects.
-* Class indices:: In-depth discussion about indexing persistent indices.
+* Class Indices:: In-depth discussion about indexing persistent indices.
* Querying persistent instances:: Retrieving instances of classes.
* Using BTrees:: Using the native btree.
* Secondary Indices:: Alternative ways to index collections.
* Using Cursors:: Low-level access to BTrees.
-* Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls.
+* Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls.
* Multi-repository Operation:: Specifying repositories.
+* Multi-threaded Applications:: What considerations are required for safe multi-threading
+* Multiple Processes and Distributed Applications:: Can elephant be run on multiple CPUs and multiple machines?
* Repository Migration and Upgrade:: How to move objects from one repository to another.
-* Garbage collection:: How to recover storage and OIDs in long-lived repositories.
-* Performance tuning:: How to get the most from Elephant.
+* Garbage Collection:: How to recover storage and OIDs in long-lived repositories.
+* Performance Tuning:: How to get the most from Elephant.
@end menu
@node Persistent objects
@@ -83,9 +84,15 @@
@code{with-open-controller} macro. Opening and closing a controller
is very expensive.
+@node Serialization details
+@comment node-name, next, previous, up
+@section Serialization details
+
+Empty.
+
@node Class Indices
@comment node-name, next, previous, up
-@section Class Indicies
+@section Class Indices
You can enable/disable class indexing for an entire class. When you disable
indexing all references to instances of that class are lost. If you re-enable
@@ -130,6 +137,10 @@
somewhat user customizable; documentation for this exists in the source
file referenced above.
+@node Querying persistent instances
+@comment node-name, next, previous, up
+@section Querying persistent instances
+
@node Using BTrees
@comment node-name, next, previous, up
@section Using BTrees
@@ -147,10 +158,127 @@
blocks of data is relatively inexpensive after a seek and comparisons
on objects that are stored in memory is cheap.
+@node Secondary Indices
+@comment node-name, next, previous, up
+@section Secondary Indices
+
+Empty.
+
+@node Using Cursors
+@comment node-name, next, previous, up
+@section Using Cursors
+
+Empty.
-@node Repository Migration
+@node Transaction Details
@comment node-name, next, previous, up
-@section Repository Migration
+@section Transaction Details
+
+;; Transaction architecture:
+;;
+;; User and designer considerations:
+;; - *current-transaction* is reserved for use by dynamic transaction context. The default global
+;; value must always be null (no transaction). Each backend can set it to a different parameter
+;; within the dynamic context of an execute-transaction.
+;; - Any closures returned from within a transaction cannot bind *current-transaction*
+;; - Only a normal return value will result in the transaction being committed, any non-local exit
+;; results in a transaction abort. If you want to do something more sophisticated, roll your own
+;; using controller-start-transaction, etc.
+;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc)
+;; knowing that the transaction will be aborted
+;;
+
+@node Multi-threaded Applications
+@comment node-name, next, previous, up
+@section Multi-threaded Applications
+
+Sleepycat plays well with threads and processes. The store controller
+is thread-safe by default, that is, can be shared amongst threads.
+This is set by the @code{:thread} key. Transactions may not be shared
+amongst threads except serially. One thing which is NOT thread and
+process safe is recovery, which should be run when no one is else is
+talking to the database environment. Consult the Sleepycat docs for
+more information.
+
+Elephant uses some specials to hold parameters and buffers. If you're
+using a natively threaded lisp, you can initialize these specials to
+thread-local storage by using the @code{run-elephant-thread} function,
+assuming your lisp creates thread-local storage for let-bound
+specials. (This functionality is currently broken)
+
+Persisting ordinary aggregate types (e.g. NOT persistent classes or
+btrees) suffers from something called "merge-conflicts." Since
+updating one value of an aggregate object requires the entire object
+to be written to the database, in heavily threaded situations you may
+overwrite changes another thread or process has committed. This is
+not protected by transactions!
+
+Consider two processes operating on the same cons:
+
+@code{
+-----start---read--update-car--write--commit----------------
+-start------read--update-cdr-----------------write--commit--
+}
+
+Although the first process successfully committed its transaction, its
+work (writing to the car) will be erased by the second transaction
+(which writes both the car and cdr.)
+
+Persistent classes and persistent collections do not suffer from
+merge-conflicts, since each slot / entry is a separate database entry.
+
+@node Multi-repository Operation
+@comment node-name, next, previous, up
+@section Multi-repository Operation
+
+Elephant now keeps a small hashtables that maps ``database specifications'' into
+actual database connections.
+
+If a database spec is a string, it is assumed to be a BerkeleyDB path.
+If it is a list, it is a assumed to be a CL-SQL connection specification.
+For example:
+@lisp
+ELE-TESTS> *testdb-path*
+"/home/read/projects/elephant/elephant/tests/testdb/"
+ELE-TESTS> *testpg-path*
+(:postgresql "localhost.localdomain" "test" "postgres" "")
+ELE-TESTS>
+@end lisp
+
+The tests now have a function @code{do-all-tests-spec} that take a spec and
+based on its type attempt to open the correct kind of store controller and
+perform the tests.
+
+The routine @code{get-controller} takes this specifiation.
+
+The basic strategy is that the ``database specification'' object is stored in
+every persistent object and collection so that the repository can be found.
+
+In this way, objects that reside in different repositories can coexist within
+the LISP object space, allowing data migration.
+
+;; Multiple stores
+
+;; Multiple store considerations:
+;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to
+;; work around: how to avoid writing one store with a transaction created in the context of another.
+;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct;
+;; this requirement may relax in the future
+;; - The following macros accomodate multiple stores by requiring that execute-transaction return a
+;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller
+;; is the store instance it is associated with. A nested or ensured transaction is only indicated
+;; in the call to execute transaction if the store controllers match, otherwise a new transaction
+;; for that store is created
+
+@node Multiple Processes and Distributed Applications
+@comment node-name, next, previous, up
+@section Multiple Processes and Distributed Applications
+
+Can we do this? What do we have to say about this?
+
+@node Repository Migration and Upgrade
+@comment node-name, next, previous, up
+@section Repository Migration and Upgrade
This version of Elephant supports migration between store controllers of
any backend type.
@@ -227,49 +355,17 @@
@code{*inhibit-slot-writes*} in your user method using
@code{with-inhibited-slot-copy} a convenience macro.
-
-@node Threading
+@node Garbage Collection
@comment node-name, next, previous, up
-@section Threading
-
-Sleepycat plays well with threads and processes. The store controller
-is thread-safe by default, that is, can be shared amongst threads.
-This is set by the @code{:thread} key. Transactions may not be shared
-amongst threads except serially. One thing which is NOT thread and
-process safe is recovery, which should be run when no one is else is
-talking to the database environment. Consult the Sleepycat docs for
-more information.
-
-Elephant uses some specials to hold parameters and buffers. If you're
-using a natively threaded lisp, you can initialize these specials to
-thread-local storage by using the @code{run-elephant-thread} function,
-assuming your lisp creates thread-local storage for let-bound
-specials. (This functionality is currently broken)
-
-Persisting ordinary aggregate types (e.g. NOT persistent classes or
-btrees) suffers from something called "merge-conflicts." Since
-updating one value of an aggregate object requires the entire object
-to be written to the database, in heavily threaded situations you may
-overwrite changes another thread or process has committed. This is
-not protected by transactions!
-
-Consider two processes operating on the same cons:
-
-@code{
------start---read--update-car--write--commit----------------
--start------read--update-cdr-----------------write--commit--
-}
-
-Although the first process successfully committed its transaction, its
-work (writing to the car) will be erased by the second transaction
-(which writes both the car and cdr.)
+@section Garbage Collection
-Persistent classes and persistent collections do not suffer from
-merge-conflicts, since each slot / entry is a separate database entry.
+GC is not implemented, but migration (@pxref{Repository Migration and
+Upgrade}) will consolidate storage and recover OIDs which emulates GC.
+No online solution is currently supported.
-@node Performance Tips
+@node Performance Tuning
@comment node-name, next, previous, up
-@section Performance Tips
+@section Performance
Performance is usually measured in transactions per second. Database
reads are cheap. To get more transactions throughput, consider
@@ -290,8 +386,8 @@
things. It is fast but consing with floats and doubles. YMMV with
other values, though I've tried to make them fast.
-Using @code{*auto-commit*} and not @code{with-transactions} is a great
-way to have a huge number of transactions. You'll find that
+Use @code{with-transactions} to avoid many automatic transactions, for
+example you'll find that this construct
@lisp
(dotimes (i 1000) (add-to-root "key" "value"))
@@ -300,9 +396,8 @@
is much slower than
@lisp
-(let ((*auto-commit* nil))
- (with-transaction ()
- (dotimes (i 1000) (add-to-root "key" "value"))))
+(with-transaction ()
+ (dotimes (i 1000) (add-to-root "key" "value"))))
@end lisp
since there's only 1 transaction in the latter. However storing
@@ -319,32 +414,3 @@
mode is not suitable for use in web servers or other typically
multi-threaded applications.
-@node Multi-repository Operation
-@comment node-name, next, previous, up
-@section Multi-repository Operation
-
-Elephant now keeps a small hashtables that maps ``database specifications'' into
-actual database connections.
-
-If a database spec is a string, it is assumed to be a BerkeleyDB path.
-If it is a list, it is a assumed to be a CL-SQL connection specification.
-For example:
-@lisp
-ELE-TESTS> *testdb-path*
-"/home/read/projects/elephant/elephant/tests/testdb/"
-ELE-TESTS> *testpg-path*
-(:postgresql "localhost.localdomain" "test" "postgres" "")
-ELE-TESTS>
-@end lisp
-
-The tests now have a function @code{do-all-tests-spec} that take a spec and
-based on its type attempt to open the correct kind of store controller and
-perform the tests.
-
-The routine @code{get-controller} takes this specifiation.
-
-The basic strategy is that the ``database specification'' object is stored in
-every persistent object and collection so that the repository can be found.
-
-In this way, objects that reside in different repositories can coexist within
-the LISP object space, allowing data migration.
1
0
Update of /project/elephant/cvsroot/elephant
In directory clnet:/tmp/cvs-serv29083
Modified Files:
TODO elephant.asd
Log Message:
Latest documentation changes
--- /project/elephant/cvsroot/elephant/TODO 2007/03/30 14:34:34 1.76
+++ /project/elephant/cvsroot/elephant/TODO 2007/04/01 14:33:23 1.77
@@ -18,7 +18,9 @@
- Verify db_deadlock for other lisps (launch and kill background program I/F)
Bugs:
+- Fix awkward serializer API
- Support for asdf-install?
+- Edi's patches & suggestions for windows
Test coverage:
- Clean up interface to tests
@@ -26,14 +28,13 @@
- Multi-threading stress tests? Ensure that there are conflicts and lots of serialization
happening concurrently to make sure that multi-threading is in good shape (Henrik's code)
- Unicode tests
- - Ensure that variable length UTF-8 reps are automatically stored as UTF-16
+ - Ensure that variable length UTF-8 reps are automatically stored as UTF-16?
- Class / DB sychronization tests
Documentation:
~ License and copyright file headers
-- Redo tutorial
-- Proper user guide
- Update install, build and test procedures
+- Proper user guide
- Upgrade, migration and other system level issues
- Performance and design issues
- More notes about transaction performance
@@ -42,7 +43,6 @@
- Add notes about optimize-storage
- Add notes about deadlock-detect
- Add notes about checkpoint (null in SQL?)
-- Add document section about backend interface & developer decisions
0.6.1 - Features COMPLETED to date
----------------------------------
--- /project/elephant/cvsroot/elephant/elephant.asd 2007/03/30 23:36:52 1.40
+++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/01 14:33:23 1.41
@@ -304,6 +304,7 @@
(:file "serializer2") ;; 0.6.1 db's
(:file "unicode2")
(:file "migrate")
+ (:file "query")
(:file "backend"))
:serial t
:depends-on (memutil utils)))))
1
0
Update of /project/elephant/cvsroot/elephant/src/elephant
In directory clnet:/tmp/cvs-serv17159/src/elephant
Modified Files:
controller.lisp
Log Message:
Fixed symbol mgmt bug in serializer2 init
--- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:36:53 1.43
+++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:42:35 1.44
@@ -290,7 +290,7 @@
(setf (controller-serialize sc)
(intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER2)))
(setf (controller-deserialize sc)
- (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER2))))))
+ (intern "DESERIALIZE" (find-package :ELEPHANT-SERIALIZER2))))))
;;
;; Handling package changes in legacy databases
1
0
Update of /project/elephant/cvsroot/elephant/src/elephant
In directory clnet:/tmp/cvs-serv15195/src/elephant
Modified Files:
backend.lisp classes.lisp collections.lisp controller.lisp
metaclasses.lisp serializer2.lisp variables.lisp
Log Message:
Sanitize class indexing option; more documentation stuff
--- /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 17:46:14 1.14
+++ /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 23:36:53 1.15
@@ -76,6 +76,9 @@
#:transaction-store
#:transaction-object
#:execute-transaction
+ #:controller-start-transaction
+ #:controller-abort-transaction
+ #:controller-commit-transaction
;; Registration
#:register-backend-con-init
--- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/24 12:16:03 1.24
+++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/30 23:36:53 1.25
@@ -47,30 +47,16 @@
;; METACLASS INITIALIZATION AND CHANGES
;; ================================================
-(defmethod ensure-class-using-class :around ((class null) name &rest args &key index)
- "Support the :index class option"
- (let ((result (apply #'call-next-method class name (remove-keywords '(:index) args))))
- (when (and index (subtypep (type-of result) 'persistent-metaclass))
- (update-indexed-record result nil :class-indexed t))
- result))
-
-(defmethod ensure-class-using-class ((class persistent-metaclass) name &rest args &key index)
- "Support the :index class option on redefinition"
- (let ((result (apply #'call-next-method class name (remove-keywords '(:index) args))))
- (when index
- (update-indexed-record result nil :class-indexed t))
- result))
-
-(defmethod shared-initialize :around ((class persistent-metaclass) slot-names &rest args &key direct-superclasses)
+(defmethod shared-initialize :around ((class persistent-metaclass) slot-names &rest args &key direct-superclasses index)
"Ensures we inherit from persistent-object."
(let* ((persistent-metaclass (find-class 'persistent-metaclass))
(persistent-object (find-class 'persistent-object))
(not-already-persistent (loop for superclass in direct-superclasses
never (eq (class-of superclass) persistent-metaclass))))
+ (when index
+ (update-indexed-record class nil :class-indexed t))
(if (and (not (eq class persistent-object)) not-already-persistent)
(apply #'call-next-method class slot-names
-;; :direct-superclasses (cons persistent-object
-;; direct-superclasses) args)
:direct-superclasses (append direct-superclasses (list persistent-object)) args)
(call-next-method))))
--- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/03/25 14:57:49 1.19
+++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/03/30 23:36:53 1.20
@@ -339,15 +339,18 @@
(defun lisp-compare-equal (a b)
(equal a b))
+(defgeneric map-btree (fn btree &rest args &key start end value)
+ (:documentation "Map btree maps over a btree from the value start to the value of end.
+ If values are not provided, then it maps over all values. BTrees
+ do not have duplicates, but map-btree can also be used with indices
+ in the case where you don't want access to the primary key so we
+ require a value argument as well for mapping duplicate value sets."))
+
;; NOTE: the use of nil for the last element in a btree only works because the C comparison
;; function orders by type tag and nil is the highest valued type tag so nils are the last
;; possible element in a btree ordered by value.
+
(defmethod map-btree (fn (btree btree) &rest args &key start end (value nil value-set-p))
- "Map btree maps over a btree from the value start to the value of end.
- If values are not provided, then it maps over all values. BTrees
- do not have duplicates, but map-btree can also be used with indices
- in the case where you don't want access to the primary key so we
- require a value argument as well for mapping duplicate value sets."
(let ((end (if value-set-p value end)))
(ensure-transaction (:store-controller (get-con btree))
(with-btree-cursor (curs btree)
@@ -368,8 +371,8 @@
(funcall fn k v)
(return nil)))))))))
-(defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p))
- "Map-index is like map-btree but for secondary indices, it
+(defgeneric map-index (fn btree &rest args &key start end value)
+ (:documentation "Map-index is like map-btree but for secondary indices, it
takes a function of three arguments: key, value and primary
key. As with map-btree the keyword arguments start and end
determine the starting element and ending element, inclusive.
@@ -377,7 +380,9 @@
the last element in the index. If you want to traverse only a
set of identical key values, for example all nil values, then
use the value keyword which will override any values of start
- and end."
+ and end."))
+
+(defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p))
(declare (dynamic-extent args)
(ignorable args))
(let ((sc (get-con index))
--- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 14:34:35 1.42
+++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:36:53 1.43
@@ -250,6 +250,12 @@
(when (member ver (rest row) :test #'equal)) t)
nil))
+(defgeneric upgrade (sc spec)
+ (:documentation "Given an open store controller from a prior version,
+ open a new store specified by spec and migrate the
+ data from the original store to the new one, upgrading
+ it to the latest version"))
+
(defmethod upgrade ((sc store-controller) target-spec)
(unless (upgradable-p sc)
(error "Cannot upgrade ~A from version ~A to version ~A~%Valid upgrades are:~%~A"
@@ -275,12 +281,16 @@
associated with the database version that is opened."
(cond ((prior-version-p (database-version sc) '(0 6 0))
(setf (controller-serializer-version sc) 1)
- (setf (controller-serialize sc) 'elephant-serializer1::serialize)
- (setf (controller-deserialize sc) 'elephant-serializer1::deserialize))
+ (setf (controller-serialize sc)
+ (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER1)))
+ (setf (controller-deserialize sc)
+ (intern "DESERIALIZE" (find-package :ELEPHANT-SERIALIZER1))))
(t
(setf (controller-serializer-version sc) 2)
- (setf (controller-serialize sc) 'elephant-serializer2::serialize)
- (setf (controller-deserialize sc) 'elephant-serializer2::deserialize))))
+ (setf (controller-serialize sc)
+ (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER2)))
+ (setf (controller-deserialize sc)
+ (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER2))))))
;;
;; Handling package changes in legacy databases
--- /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/03/23 16:08:10 1.13
+++ /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/03/30 23:36:53 1.14
@@ -23,8 +23,11 @@
(declaim #-elephant-without-optimize (optimize (speed 3) (safety 1)))
(defclass persistent ()
- ((%oid :accessor oid :initarg :from-oid)
- (dbconnection-spec-pst :type (or list string) :accessor dbcn-spc-pst :initarg :dbconnection-spec-pst))
+ ((%oid :accessor oid :initarg :from-oid
+ :documentation "All persistent objects have an oid")
+ (dbconnection-spec-pst :type (or list string) :accessor dbcn-spc-pst :initarg :dbconnection-spec-pst
+ :documentation "Persistent objects use a spec pointer to identify which store
+ they are connected to"))
(:documentation "Abstract superclass for all persistent classes (common
to user-defined classes and collections.)"))
--- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/03/30 14:34:35 1.34
+++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/03/30 23:36:53 1.35
@@ -24,8 +24,6 @@
(:import-from :elephant
*circularity-initial-hash-size*
get-cached-instance
- controller-symbol-cache
- controller-symbol-id-cache
slot-definition-allocation
slot-definition-name
compute-slots
--- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/03/30 17:45:41 1.13
+++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/03/30 23:36:53 1.14
@@ -64,10 +64,11 @@
;; properly load in asdf due to some circular dependencies
;; between lisp files
-(eval-when (load-toplevel compile-toplevel)
+(eval-when (:compile-toplevel :load-toplevel)
(mapcar (lambda (symbol)
(intern symbol :elephant))
- '(get-cached-instance)))
+ '("GET-CACHED-INSTANCE"
+ "SET-DB-SYNCH")))
1
0
Update of /project/elephant/cvsroot/elephant/doc
In directory clnet:/tmp/cvs-serv15195/doc
Modified Files:
Makefile data-store-reference.texinfo make-ref.lisp
reference.texinfo
Log Message:
Sanitize class indexing option; more documentation stuff
--- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 13:55:15 1.4
+++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/30 23:36:52 1.5
@@ -8,3 +8,5 @@
makeinfo -v --html --css-include=style.css --force elephant.texinfo
makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo
+pdf: includes-stuff
+ texi2dvi --pdf elphant.texinfo
--- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 17:28:50 1.3
+++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 23:36:52 1.4
@@ -38,10 +38,10 @@
* Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries
@end menu
-@node Registration
+@node DSR Registration
@comment node-name, next, previous, up
@section Registration
-@cindex Registration
+@cindex Registration and Initialization
Elephant looks at the first element of the specification list to
determine which backend code base to use. The master table for this
@@ -55,7 +55,13 @@
@include includes/fun-elephant-backend-register-backend-con-init.texinfo
-@node Store Controllers
+If the backend requires any special user-specified configuration,
+augment the key types in config.sexp with what you need and use the
+following function to access.
+
+@include includes/fun-elephant-backend-get-user-configuration-parameter.texinfo
+
+@node DSR Store Controllers
@comment node-name, next, previous, up
@section Store Controllers
@cindex Store Controllers
@@ -63,9 +69,10 @@
Subclass store-controller and implement store and close controller
which are called by open-store and close-store respectively.
-@include includes/class-elephant-backend-store-controller.texinfo
+@include includes/class-elephant-store-controller.texinfo
@include includes/fun-elephant-backend-open-controller.texinfo
@include includes/fun-elephant-backend-close-controller.texinfo
+@include includes/fun-elephant-backend-connection-is-indeed-open.texinfo
For upgrading and opening legacy databases it is important that a
store be able to indicate which version of elephant was used to create
@@ -79,21 +86,24 @@
There are some utilities for serializing simple data without a
serializer using the memutil package.
-@include
-
-These functions are useful utilities for implementing
-store-controllers.
+@include includes/fun-elephant-backend-serialize-database-version-key.texinfo
+@include includes/fun-elephant-backend-serialize-database-version-value.texinfo
+@include includes/fun-elephant-backend-deserialize-database-version-value.texinfo
-@include includes/fun-elephant-backend-get-con.texinfo
-@include includes/fun-elephant-backend-oid.texinfo
-@include includes/fun-elephant-backend-next-oid.texinfo
-@include includes/fun-elephant-backend-connection-is-indeed-open.texinfo
-@include includes/fun-elephant-get-user-configuration-parameter.texinfo
-@node Slot Access
+@node DSR Persistent Objects and Slot Access
@comment node-name, next, previous, up
@section Slot Access
-@cindex Slot Access
+@cindex Persistent Objects and Slot Access
+
+@include includes/class-elephant-persistent.texinfo
+@include includes/fun-elephant-backend-get-con.texinfo
+@c @include includes/fun-elephant-backend-oid.texinfo
+
+All objects require a unique id. During new object creation the
+backend is asked to produce a unique id.
+
+@include includes/fun-elephant-backend-next-oid.texinfo
These functions are called by the metaclass protocol to support
operations on persistent class slots.
@@ -103,7 +113,7 @@
@include includes/fun-elephant-backend-persistent-slot-boundp.texinfo
@include includes/fun-elephant-backend-persistent-slot-makunbound.texinfo
-@node Collections
+@node DSR Collections
@comment node-name, next, previous, up
@section Collections
@cindex Collections
@@ -133,7 +143,12 @@
@include includes/fun-elephant-get-index.texinfo
@include includes/fun-elephant-remove-index.texinfo
-@node Cursors
+Critical to indexing and queries are the map operators for collections
+
+@include includes/fun-elephant-map-btree.texinfo
+@include includes/fun-elephant-map-index.texinfo
+
+@node DSR Cursors
@comment node-name, next, previous, up
@section Cursors
@cindex Cursors
@@ -143,22 +158,28 @@
@c #:cursor-oid
@c #:cursor-initialized-p
-@node Transactions
+@node DSR Transactions
@comment node-name, next, previous, up
@section Transactions
@cindex Transactions
-@c #:*current-transaction*
-@c #:make-transaction-record
-@c #:transaction-store
-@c #:transaction-object
-
-@c #:execute-transaction
-@c #:controller-start-transaction
-@c #:controller-commit-transaction
-@c #:controller-abort-transaction
+These functions must be implemented or stubbed in any
+backend.
+
+@include includes/fun-elephant-backend-execute-transaction.texinfo
+@include includes/fun-elephant-backend-controller-start-transaction.texinfo
+@include includes/fun-elephant-backend-controller-commit-transaction.texinfo
+@include includes/fun-elephant-backend-controller-abort-transaction.texinfo
+
+These are supporting functions and variables for implementing
+transactions.
+
+@include includes/var-elephant-backend-star-current-transaction-star.texinfo
+@include includes/fun-elephant-backend-make-transaction-record.texinfo
+@include includes/fun-elephant-backend-transaction-store.texinfo
+@include includes/fun-elephant-backend-transaction-object.texinfo
-@node Multithreading Considerations
+@node DSR Multithreading Considerations
@comment node-name, next, previous, up
@section Multithreading Considerations
@cindex Multithreading
@@ -166,7 +187,7 @@
@c utils locks
@c utils thread-vars
-@node Handling Serialization
+@node DSR Handling Serialization
@comment node-name, next, previous, up
@section Handling Serialization
@cindex Serialization
@@ -176,12 +197,12 @@
@c #:deserialize-from-base64-string
@c #:serialize-to-base64-string
-@node Memory utilities
+@node DSR Memory utilities
@comment node-name, next, previous, up
@section Memory utilities
@cindex Memory utilities
-@node Foreign libraries
+@node DSR Foreign libraries
@comment node-name, next, previous, up
@section Foreign libraries
@cindex Foreign libraries
--- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/30 14:34:34 1.5
+++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/30 23:36:52 1.6
@@ -1,29 +1,39 @@
(require 'asdf)
-(asdf:operate 'asdf:load-op 'elephant-tests)
+(asdf:operate 'asdf:load-op 'elephant)
+(load (merge-pathnames
+ #p"src/elephant/query"
+ (asdf:component-pathname (asdf:find-system 'elephant))))
+
(defparameter include-dir-path
(namestring
(merge-pathnames
#p"doc/includes/"
- (asdf:component-pathname (asdf:find-system 'elephant-tests)))))
+ (asdf:component-pathname (asdf:find-system 'elephant)))))
(defparameter docstrings-path
(namestring
(merge-pathnames
#p"doc/docstrings.lisp"
- (asdf:component-pathname (asdf:find-system 'elephant-tests)))))
+ (asdf:component-pathname (asdf:find-system 'elephant)))))
(sb-posix:chdir include-dir-path)
(load docstrings-path)
+(in-package :elephant)
+
+(defclass simple-store-controller (store-controller)
+ ())
+
(defun make-docs ()
- (elephant:open-store elephant-tests::*testbdb-spec*)
- (make-instance 'elephant::persistent-collection)
- (make-instance 'elephant::secondary-cursor)
- (make-instance 'elephant::indexed-btree)
- (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/"
- (find-package :elephant)
- (find-package :elephant-backend)
- (find-package :elephant-memutil)
- (find-package :elephant-system)))
+ (let ((sc (make-instance 'simple-store-controller)))
+ (setf (controller-spec sc) nil)
+ (make-instance 'elephant::persistent-collection :sc sc :from-oid 10)
+ (make-instance 'elephant::secondary-cursor)
+ (make-instance 'elephant::indexed-btree :sc sc :from-oid 10)
+ (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/"
+ (find-package :elephant)
+ (find-package :elephant-backend)
+ (find-package :elephant-memutil)
+ (find-package :elephant-system))))
(make-docs)
--- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 14:34:34 1.7
+++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 23:36:52 1.8
@@ -110,12 +110,15 @@
@include includes/fun-elephant-get-value.texinfo
@include includes/fun-elephant-setf-get-value.texinfo
@include includes/fun-elephant-remove-kv.texinfo
+@include includes/fun-elephant-map-btree.texinfo
+@include includes/fun-elephant-map-index.texinfo
@include includes/fun-elephant-add-index.texinfo
@include includes/fun-elephant-get-index.texinfo
@include includes/fun-elephant-get-primary-key.texinfo
@include includes/fun-elephant-remove-index.texinfo
+
@node Cursors
@comment node-name, next, previous, up
@section Cursors
@@ -125,7 +128,6 @@
@include includes/class-elephant-secondary-cursor.texinfo
@include includes/fun-elephant-make-cursor.texinfo
@include includes/fun-elephant-cursor-close.texinfo
-@include includes/fun-elephant-map-btree.texinfo
@include includes/macro-elephant-with-btree-cursor.texinfo
@include includes/fun-elephant-cursor-current.texinfo
@@ -163,15 +165,10 @@
@include includes/macro-elephant-with-transaction.texinfo
-@include includes/var-elephant-star-auto-commit-star.texinfo
-@include includes/var-elephant-star-current-transaction-star.texinfo
-@include includes/fun-elephant-start-ele-transaction.texinfo
-@include includes/fun-elephant-commit-transaction.texinfo
-@include includes/fun-elephant-abort-transaction.texinfo
-
@node Migration and Upgrading
@comment node-name, next, previous, up
@section Migration and Upgrading
@cindex Migration and Upgrading
@include includes/fun-elephant-migrate.texinfo
+@include includes/fun-elephant-upgrade.texinfo
1
0