- elephant-cvs@lists.common-lisp.net - mailman3.common-lisp.net

[elephant-cvs] CVS elephant/src/db-clsql
by rread 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv8188 Modified Files: package.lisp Log Message: Not sure what happened here but this file is definitely needed for the test on the sql side. --- /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/02/07 22:54:12 1.2 +++ /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/04/01 22:01:51 1.3 @@ -21,7 +21,7 @@ (defpackage db-clsql (:use :common-lisp :uffi :cl-base64 :elephant :elephant-memutil :elephant-backend -;; :elephant-utils + :elephant-utils #+sbcl :sb-thread ))

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv27196/doc Modified Files: user-guide.texinfo Log Message: Fixed package bug. Minor tweaks elsewhere --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:22:24 1.6 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:56:19 1.7 @@ -141,8 +141,6 @@ @comment node-name, next, previous, up @section Querying persistent instances - - A SQL select-like interface is in the works, but for now queries are limited to manual mapping over class instances or doing small queries with @code{get-instances-*} functions. One advantage of this is that

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv27196/src/elephant Modified Files: backend.lisp package.lisp query-example.lisp query.lisp Log Message: Fixed package bug. Minor tweaks elsewhere --- /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 23:36:53 1.15 +++ /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/04/01 20:56:19 1.16 @@ -56,6 +56,10 @@ #:controller-deserialize #:root #:spec #:class-root + ;; Collections + #:build-btree + #:build-indexed-btree + ;; Serializer tools/api's #:serialize #:deserialize #:deserialize-from-base64-string --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/03/30 14:55:54 1.27 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/01 20:56:19 1.28 @@ -206,9 +206,9 @@ #:persistent #:persistent-object #:persistent-metaclass #:defpclass #:persistent-collection #:drop-pobject - #:btree #:build-btree + #:btree #:make-btree #:get-value #:remove-kv #:existsp - #:indexed-btree #:build-indexed-btree + #:indexed-btree #:make-indexed-btree #:btree-index #:add-index #:get-index #:remove-index #:map-indices #:get-primary-key #:primary #:key-form #:key-fn @@ -260,6 +260,9 @@ #:lookup-persistent-symbol-id #:struct-constructor + + #:map-class-query + #:get-query-instances ) ) --- /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/03/01 02:45:45 1.1 +++ /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/04/01 20:56:19 1.2 @@ -61,7 +61,7 @@ (slot-value (slot-value person 'department) 'name))) (defun example-query1 () - "Performs a query against a single class. Trivial string & integer matchingA" + "Performs a query against a single class. Trivial string & integer matching" (map-class-query #'print-person '((person name = "Greg") (person salary >= 100000)))) --- /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/03/30 14:34:35 1.2 +++ /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/04/01 20:56:19 1.3 @@ -41,6 +41,7 @@ (defun get-query-instances (constraints) "Get a list of instances according to the query constraints" + (declare (dynamic-extent constraints)) (let ((list nil)) (flet ((collect (inst) (push inst list))) @@ -52,6 +53,7 @@ slot indices (for last query) and stack allocated test closures. This is a minimally optimizing version that uses the first index it finds, and then does a nested loop join on the rest of the parameters." + (declare (dynamic-extent constraints)) (assert (not (null constraints))) (destructuring-bind (class slot relation &rest values) (first constraints) (flet ((filter-by-relation (inst)

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv22287 Modified Files: tutorial.texinfo user-guide.texinfo Log Message: Documentation changes, mostly to transaction section of tutorial --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 20:22:24 1.12 @@ -732,6 +732,8 @@ transaction that performs all the updates atomically and thus enforcing consistency. +@subsection Why do we need Transactions? + Most real applications will need to use explicit transactions rather than relying on the primitives alone because you will want multiple read-modify-update operations act as an atomic unit. A good example @@ -815,6 +817,8 @@ And presto, we have an ACID compliant, thread-safe, persistent banking system! +@subsection Using @code{with-transaction} + What is @code{with-transaction} really doing for us? It first starts a new transaction, attempts to execute the body, and if successful commit the transaction. If anywhere along the way there is a deadlock @@ -823,14 +827,145 @@ to retry the transaction a fixed number of times by re-executing the whole body. -The other value transactions provide is the capability to delay -flushing dirty data to disk. The most time-intensive part of -persistent operations is flushing newly written data to disk. Using -the default auto-commit behavior requires a flush on every operation -which can become very expensive. Because a transaction caches values, -all the values read or written are cached in memory until the -transaction completes, dramatically decreasing the number of flushes -and the total time taken. +And this brings us to two important caveats: nested transactions and +idempotent side-effects. + +@subsection Nesting Transactions + +In general, you want to avoid nesting @code{with-transaction} +statements. Nested transactions are valid for some data stores +(namely Berkeley DB), but typically only a single transaction can be +active at a time. The purpose of a nested transaction in data stores +that provide it, is break a long transaction into chunks. This way if +there is contention on a given subset of variables, only the inner +transaction is restarted while the larger transaction can continue. +When commit their results, those results become part of the outer +transaction until it in turn commits. + +If you have transaction protected primitive operations (such as +@code{deposit} and @code{withdraw}) and you want to perform a group of +such transactions, for example a transfer between accounts, you can +use the macro @code{ensure-transaction} instead of @code{with-transaction}. + +@lisp +(defun deposit (account amount) + "Wrap the balance read and the setf with the new balance" + (ensure-transaction () + (let ((balance (balance account))) + (setf (balance account) + (+ balance amount))))) + +(defun deposit (account amount) + "A more concise version with decf doing both read and write" + (ensure-transaction () + (decf (balance account) amount))) + +(defun withdraw (account amount) + (ensure-transaction () + (decf (balance account) amount))) + +(defun transfer (src dst amount) + "There are four primitive read/write operations + grouped together in this transaction" + (with-transaction () + (withdraw src amount) + (deposit dst amount))) +@end lisp + +@code{ensure-transaction} is exactly like @code{with-transaction} +except it will reuse an existing transaction, if there is one, or +create a new one. There is no harm, in fact, in using this macro all +the time. + +Notice the use of @code{decf} and @code{incf} above. The primary +reason to use Lisp is that it is good at hiding complexity using +shorthand constructs just like this. This also means it is also going +to be good at hiding data dependencies that should be captured in a +transaction! + +@subsection Idempotent Side Effects + +Within the body of a with-transaction, any non database operations +need to be @emph{idempotent}. That is the side effects of the body +must be the same no matter how many times the body is executed. This +is done automatically for side effects on the database, but not for +side effects like pushing a value on a lisp list, or creating a new +standard object. + +@lisp +(defparameter *transient-objects* nil) + +(defun load-transients (n) + "This is the wrong way!" + (with-transaction () + (loop for i from 0 upto n do + (push (get-from-root i) *transient-objects*)))) +@end lisp + +In this contrived example we are pulling a set of standard objects +from the database using an integer key and pushing them onto a list +for later use. However, if there is a conflict where some other +process writes a key-value pair to a matching key, the whole +transaction will abort and the loop will be run again. In a heavily +contended system you might see results like the following. + +@lisp +(defun test-list () + (setf *transient-objects* nil) + (load-transients) + (length *transient-objects*)) + +(test-list) +=> 3 + +(test-list) +=> 5 + +(test-list) +=> 4 +@end lisp + +So the solution is to make sure that the operation on the lisp +parameters is atomic if the transaction completes. + +@lisp +(defun load-transients () + "This is a better way" + (setq *transient-objects* + (with-transaction () + (loop for i from 0 upto 3 collect + (get-from-root i))))) +@end lisp + +Of course we would need to use @code{nreverse} if we cared about the +order of instances in @code{*transient-objects*}. The best rule of +thumb is that transaction bodies should be purely functional as above, +except for side effects to the persistent store such as persistent +slot writes, adding to btrees, etc). + +If you do need side effects to lisp memory, such as writes to +transient slots, make sure they are idempotent and that other +processes will not be reading the written values until the transaction +completes. + +@subsection Transactions and Performance + +By now transactions almost look like more work than they are worth! +Well there are still some significant benefits to be had. Part of how +transactions are implemented is that they gather together all the +writes that are supposed to made to the database and store them until +the transaction commits, and then writes them atomically. + +The most time-intensive part of persistent operations is flushing +newly written data to disk. Using the default auto-committing +behavior requires a flush for every primitive write operation. This +can become very expensive! Because all the values read or written are +cached in memory until the transaction completes, the number of +flushes can be dramatically reduced. + +But don't take my word for it, run the following statements and see +for yourself the visceral impact transactions can have on system +performance. @lisp (defpclass test () @@ -872,52 +1007,42 @@ thumb is to keep the number of objects touched in a transaction well under 1000. -And this brings us to the last caveat we'll introduce in this -introductory tutorial: nested transactions. - -In general, avoid nesting transactions. Nested transactions are valid -for some data stores (namely Berkeley DB), but typically only a single -transaction is valid at a time. The purpose of a nested transaction -is to allow a long transaction to be broken up into chunks. This way -if there is contention on a given subset of variables, only the -subtransaction is restarted while the larger transaction can continue. -Subtransactions commit their results and they become part of the -outer transaction until it in turn commits. - -If you have transaction protected primitive operations (such as -@code{deposit} and @code{withdraw}) and you want to perform a group of -such transactions, for example a transfer between accounts, you can -use the macro @code{ensure-transaction} instead of @code{with-transaction}. - -@lisp -(defun deposit (account amount) - (ensure-transaction () - (let ((balance (balance account))) - (setf (balance account) - (+ balance amount))))) - -(defun withdraw (account amount) - (ensure-transaction () - (decf (balance account) amount))) - -(defun transfer (src dst amount) - (with-transaction () - (withdraw src amount) - (deposit dst amount))) -@end lisp - -@code{ensure-transaction} is exactly like @code{with-transaction} -except it will reuse an existing transaction, if there is one, or -create a new one. There is no harm, in fact, in using this macro all -the time. +@subsection Transactions and Applications Designing and tuning a transactional architecture can become quite -complicated. The best strategy at the beginning is a conservative -one, break things up into the smallest logical sets of primitive -operations and only wrap higher level functions in transactions when -they absolutely have to commit together. See @ref{Transaction Details} -for the full details and @pxref{Usage Scenarios} for more examples of -how systems can be designed and tuned using transactions. +complex. Moreover, bugs in your system can be very difficult to find +as they only show up when transactions are interleaved within a +larger, multi-threaded application. + +In many cases, however, you can ignore transactions. For example, +when you don't have any other concurrent processes running. In this +case all operations are sequential and there is no chance of +conflicts. You would only want to use transactions for write +performance. + +You can also ignore transactions if your application can guarantee +that concurrency won't generate any conflicts. For example, a web app +that guarantees only one thread will write to objects in a particular +session can avoid transactions altogether. However, it is good to be +careful about making these assumptions. In the above example, a +reporting function that iterates over sessions, users or other objects +may still see partial updates (i.e. a user's id was written prior to +the query, but not the name). However, if you don't care about these +infrequent glitches, this case would still hold. + +If these cases don't apply to your application, or you aren't sure, +you will fare best by programming defensively. Break your system into +the smallest logical sets of primitive operations +(i.e. @code{withdraw} and @code{deposit}) using +@code{ensure-transaction} and then wrap the highest level calls made +to your system in with-transaction when the operations absolutely have +to commit together or you need the extra performance. Try not to have +more than two levels of transactional accesses with the top using +with-transaction and the bottom using ensure-transaction. + +@xref{Transaction Details} for more details and @pxref{Usage +Scenarios} for examples of how systems can be designed and tuned using +transactions. @node Advanced Topics @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:22:24 1.6 @@ -23,26 +23,6 @@ * Performance Tuning:: How to get the most from Elephant. @end menu -@node Persistent objects -@comment node-name, next, previous, up -@section Persistent Objects - -Finally, if you for some reason make an instance with a specified OID -which already exists in the database, @code{initargs} take precedence -over values in the database, which take precedences over -@code{initforms}. - -Also currently there is a bug where -@code{initforms} are always evaluated, so beware. -(What is the current model here?) - -Readers, writers, accessors, and @code{slot-value-using-class} are -employed in redirecting slot accesses to the database, so override -these with care. Because @code{slot-value, slot-boundp, -slot-makunbound} are not generic functions, they are not guaranteed by -the specification to work properly with persistent slots. However the -proper behavior has been verified on SBCL, Allegro and Lispworks. - @node The Store Controller @comment node-name, next, previous, up @section The Store Controller @@ -90,6 +70,26 @@ Empty. +@node Persistent objects +@comment node-name, next, previous, up +@section Persistent Objects + +Finally, if you for some reason make an instance with a specified OID +which already exists in the database, @code{initargs} take precedence +over values in the database, which take precedences over +@code{initforms}. + +Also currently there is a bug where +@code{initforms} are always evaluated, so beware. +(What is the current model here?) + +Readers, writers, accessors, and @code{slot-value-using-class} are +employed in redirecting slot accesses to the database, so override +these with care. Because @code{slot-value, slot-boundp, +slot-makunbound} are not generic functions, they are not guaranteed by +the specification to work properly with persistent slots. However the +proper behavior has been verified on SBCL, Allegro and Lispworks. + @node Class Indices @comment node-name, next, previous, up @section Class Indices @@ -141,6 +141,111 @@ @comment node-name, next, previous, up @section Querying persistent instances + + +A SQL select-like interface is in the works, but for now queries are +limited to manual mapping over class instances or doing small queries +with @code{get-instances-*} functions. One advantage of this is that +it is easy to estimate the performance costs of your queries and to +choose standard and derived indices that give you the ordering and +performance you want. + +There is, however, a quick and dirty query API example that is not +officially supported in the release but is intended to invite comment. +This is an example of a full query system that would automatically +perform joins, use the appropriate indices and perhaps even adaptively +suggest or add indices to facilitate better performance on common +queries. + +There are two functions @ref{Function elephant:get-query-instances} +and @ref{Function elephant:map-class-query} which accept a set of +constraints instead of the familiar value or range arguments. + +We'll use the classes @code{person} and @code{department} to +illustrate how to perform queries over a set of objects that may be +constrainted by their relationships to other objects. + +@lisp +(defpclass person () + ((name :initarg :name :index t) + (salary :initarg :salary :index t) + (department :initarg :dept))) + +(defmethod print-object ((p person) stream) + (format stream "#<PERS: ~A>" (slot-value p 'name))) + +(defun print-name (inst) + (format t "Name: ~A~%" (slot-value inst 'name))) + +(defpclass department () + ((name :initarg :name) + (manager :initarg :manager))) + +(defmethod print-object ((d department) stream) + (format stream "#<DEPT ~A, mgr = ~A>" + (slot-value d 'name) + (when (slot-boundp d 'manager) + (slot-value (slot-value d 'manager) 'name)))) +@end lisp + +Here we have a simple employee database with managers (also of type +person) and departments. This simple system will provide fodder for +some reasonably complex constraints. Let's create a few departments. + +@lisp +(setf marketing (make-instance 'department :name "Marketing")) +(setf engineering (make-instance 'department :name "Engineering")) +(setf sales (make-instance 'department :name "Sales")) +@end lisp + +And manager @code{people} for the departments. + +@lisp +(make-instance 'person :name "George" :salary 140000 :department marketing) +(setf (slot-value marketing 'manager) *) + +(make-instance 'person :name "Sally" :salary 140000 :department engineering) +(setf (slot-value engineering 'manager) *) + +(make-instance 'person :name "Freddy" :salary 180000 :department sales) +(setf (slot-value sales 'manager) *) +@end lisp + +And of course we need some folks to manage + +@lisp +(defparameter *names* + '("Jacob" "Emily" "Michael" "Joshua" "Andrew" "Olivia" "Hannah" "Christopher")) + +(defun random-element (list) + "Choose a random element from the list and return it" + (nth (random (length list)) list)) + +(with-transaction () + (loop for i from 0 upto 40 do + (make-instance 'person + :name (format nil "~A~A" (random-elephant *names*) i) + :salary (floor (+ (* (random 1000) 100) 30000)) + :department (case (random 3) + (0 marketing) + (1 engineering) + (2 sales))))) +@end lisp + +Due to the random allocation of +In the follwoing examples below, the results will be different due to the random +allocation of employee names, etc. However, these examples are +illustrative of what you should see if you run the same code. + + + +For those familiar with SQL, if an instance of @code{person} has a +pointer to an instance of @code{department} then that relation can be +used to perform a join. Of course joins in the object world won't +return a table, instead they will return conjunctions of objects that +satisfy a mutual set of constraints. + + @node Using BTrees @comment node-name, next, previous, up @section Using BTrees @@ -174,6 +279,14 @@ @comment node-name, next, previous, up @section Transaction Details +You can trace @code{elephant::execute-transaction} to see the sequence +of calls to @code{execute-transaction} that occur dynamically and +detect where transactions are and are not happening. We may add some +transaction diagnosis and tracing tools in the future, such as +throwing a condition when @code{with-transaction} forms are nested +dynamically. + + ;; Transaction architecture: ;; ;; User and designer considerations:

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv29083/src/elephant Modified Files: classes.lisp controller.lisp metaclasses.lisp serializer.lisp Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/30 23:36:53 1.25 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/01 14:33:34 1.26 @@ -39,9 +39,11 @@ (defclass persistent-object (persistent) () (:metaclass persistent-metaclass) (:documentation - "Superclass of all user-defined persistent classes. This is + "Superclass for all user-defined persistent classes. This is automatically inherited if you use the persistent-metaclass - metaclass.")) + metaclass. This allows specialization of functions for user + objects that would not be appropriate for Elephant objects + such as persistent collections")) ;; ================================================ ;; METACLASS INITIALIZATION AND CHANGES --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:42:35 1.44 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/01 14:33:34 1.45 @@ -48,11 +48,13 @@ (defvar *dbconnection-spec* (make-hash-table :test 'equal)) (defvar *dbconnection-lock* (ele-make-lock)) -(defmethod get-con ((instance persistent) &optional (sc *store-controller*)) - "This is used to find and validate the connection spec +(defgeneric get-con (instance &optional sc) + (:documentation "This is used to find and validate the connection spec maintained for in-memory persistent objects. Should we re-open the controller from the spec if it's not - cached? That might be dangerous so for now we error" + cached? That might be dangerous so for now we error")) + +(defmethod get-con ((instance persistent) &optional (sc *store-controller*)) (declare (ignore sc)) (let ((con (gethash (dbcn-spc-pst instance) *dbconnection-spec*))) (cond ((not con) --- /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/03/30 23:36:53 1.14 +++ /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/04/01 14:33:34 1.15 @@ -29,7 +29,7 @@ :documentation "Persistent objects use a spec pointer to identify which store they are connected to")) (:documentation "Abstract superclass for all persistent classes (common - to user-defined classes and collections.)")) + to both user-defined classes and Elephant-defined objects such as collections.)")) (defclass persistent-metaclass (standard-class) ((%persistent-slots :accessor %persistent-slots) --- /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/03/30 14:34:35 1.26 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/04/01 14:33:34 1.27 @@ -47,6 +47,8 @@ ;; (defun serialize-to-base64-string (x sc) + "Encode object using the store controller's serializer format, + but encoded in a base64" (with-buffer-streams (out-buf) (cl-base64::usb8-array-to-base64-string (elephant-memutil::buffer-read-byte-vector @@ -62,6 +64,7 @@ ) (defun deserialize-from-base64-string (x sc) + "Decode a base64-string using the store controller's deserialize method" (with-buffer-streams (other) (deserialize (elephant-memutil::buffer-write-byte-vector

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv29083/doc Modified Files: Makefile data-store-reference.texinfo elephant-design.texinfo elephant.texinfo installation.texinfo reference.texinfo tutorial.texinfo user-guide.texinfo Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/30 23:36:52 1.5 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/01 14:33:24 1.6 @@ -9,4 +9,4 @@ makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo pdf: includes-stuff - texi2dvi --pdf elphant.texinfo + texi2dvi --pdf elephant.texinfo --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 23:36:52 1.4 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5 @@ -3,9 +3,8 @@ @node Data Store API Reference @comment node-name, next, previous, up @chapter Data Store API Reference -@cindex Data Store API Reference @cindex Data Store -@cindex API Reference +@cindex API This reference includes functions that need to be overridden, classes inherited from or other action taken to implement support for a new @@ -26,16 +25,16 @@ relevant to them. @menu -* Registration:: Register the backend to parse controller specifications. -* Store Controllers:: Subclassing the store controller. -* Handling Serialization:: Available facilities for serializing objects. -* C Utilities:: Writing primitive C types. -* Slot access:: Support for metaprotocol slot access. -* Collections:: BTrees and indices. -* Cursors:: Traversing BTrees. -* Transactions:: Transaction implementation. -* Multithreading:: Multithreading considerations. -* Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries +* Registration: DSR Registration. Register the backend to parse controller specifications. +* Store Controllers: DSR Store Controllers. Subclassing the store controller. +* Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects. +* Memory Utilities: DSR Memory Utilities. Writing primitive C types. +* Persistent Objects and Slot access: DSR Persistent Objects and Slot Access. Support for metaprotocol slot access. +* Collections: DSR Collections. BTrees and indices. +* Cursors: DSR Cursors. Traversing BTrees. +* Transactions: DSR Transactions. Transaction implementation. +* Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations. +* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries @end menu @node DSR Registration @@ -96,17 +95,28 @@ @section Slot Access @cindex Persistent Objects and Slot Access +Persistence is implement with a metaclass and several required base +classes. + +@include includes/class-elephant-persistent-metaclass.texinfo @include includes/class-elephant-persistent.texinfo +@include includes/class-elephant-persistent-object.texinfo + +Persistent objects can be queries for their home store controller so +that functions such as map-btree do not need a store-controller +argument. (NOTE: Should this function be user visible?) + @include includes/fun-elephant-backend-get-con.texinfo -@c @include includes/fun-elephant-backend-oid.texinfo -All objects require a unique id. During new object creation the -backend is asked to produce a unique id. +All objects require a unique object identifier. During new object +creation the backend is asked to produce a unique id. @include includes/fun-elephant-backend-next-oid.texinfo -These functions are called by the metaclass protocol to support -operations on persistent class slots. +These functions are called by the metaclass protocol to implement the +appropriate operations on persistent class slots. Unless protected by +a transaction, the side effects of these functions should be atomic, +persistent and visible to other threads on completion. @include includes/fun-elephant-backend-persistent-slot-writer.texinfo @include includes/fun-elephant-backend-persistent-slot-reader.texinfo @@ -121,7 +131,9 @@ To support collections, the data store must subclass the following classes. -@include includes/class-elephant-btree.texinfo.texinfo +@include includes/class-elephant-persistent-collection.texinfo + +@include includes/class-elephant-btree.texinfo @include includes/class-elephant-btree-index.texinfo @include includes/class-elephant-indexed-btree.texinfo @@ -131,42 +143,48 @@ @include includes/fun-elephant-build-btree.texinfo @include includes/fun-elephant-build-indexed-btree.texinfo -And every btree needs accessors, these must be implemented for btree, -indexed-btree and btree-index. +Most of the user-visible operations over BTrees must be implemented. +Class indexing functions such as @code{map-class} and +@code{get-instances-by-value} and related functions are all +implemented using map-btree and map-index. + +@itemize +@item @ref{Generic-Function elephant:get-value} +@item @ref{Generic-Function (setf elephant:get-value)} +@item @ref{Generic-Function elephant:existsp} +@item @ref{Generic-Function elephant:remove-kv} +@item @ref{Generic-Function elephant:get-index} +@item @ref{Generic-Function elephant:remove-index} +@item @ref{Generic-Function elephant:map-btree} +@item @ref{Generic-Function elephant:map-index} +@end itemize -@include includes/fun-elephant-get-value.texinfo -@include includes/fun-elephant-setf-get-value.texinfo -@include includes/fun-elephant-existsp.texinfo -@include includes/fun-elephant-remove-kv.texinfo +Mapping over the indices of a btree is important to derived facilities +such as class indexing and the query subsystem. @include includes/fun-elephant-map-indices.texinfo -@include includes/fun-elephant-get-index.texinfo -@include includes/fun-elephant-remove-index.texinfo - -Critical to indexing and queries are the map operators for collections - -@include includes/fun-elephant-map-btree.texinfo -@include includes/fun-elephant-map-index.texinfo @node DSR Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors -@include includes/class-cursor.texinfo -@c #:cursor-btree -@c #:cursor-oid -@c #:cursor-initialized-p +Data stores must subclass these cursor classes and implement all the +methods described in @ref{Cursors} except @ref{Macro +elephant:with-btree-cursor}. + +@include includes/class-elephant-cursor.texinfo +@include includes/class-elephant-secondary-cursor.texinfo @node DSR Transactions @comment node-name, next, previous, up @section Transactions @cindex Transactions -These functions must be implemented or stubbed in any -backend. +These functions must be implemented or stubbed by all data stores. @include includes/fun-elephant-backend-execute-transaction.texinfo + @include includes/fun-elephant-backend-controller-start-transaction.texinfo @include includes/fun-elephant-backend-controller-commit-transaction.texinfo @include includes/fun-elephant-backend-controller-abort-transaction.texinfo @@ -179,33 +197,77 @@ @include includes/fun-elephant-backend-transaction-store.texinfo @include includes/fun-elephant-backend-transaction-object.texinfo +;; Designer considerations: +;; - with-transaction passes *current-transaction* or the user parameter to execute-transaction +;; in the parent keyword argument. Backends allowing nested transactions can treat the transaction +;; as a parent, otherwise they can reuse the current transaction by ignoring it (inheriting the dynamic +;; value of *current-transaction*) or rebinding the dynamic context (whatever makes coding easier). +;; - ensure-transaction uses *current-transaction* to determine if there is a current transaction +;; in progress (not null). If so, it jumps to the body directly. Otherwise it executes the body in a +;; new transaction by calling ... +;; - execute-transaction contract: +;; - Backends must dynamically bind *current-transaction* to a meaningful identifier for the +;; transaction in progress and execute the provided closure in that context +;; - All non-local exists result in an abort; only regular return values result in a commit +;; - If a transaction is aborted due to a deadlock or read conflict, execute-transaction should +;; automatically retry with an appropriate default amount +;; - execute-transaction can take any number of backend-defined keywords, although designers should +;; make sure there are no semantic conflicts if there is a name overlap with existing backends +;; - A typical design approach is to make sure that the most primitive interfaces to the backend +;; database look at *current-transaction* to determine whether a transaction is active. Users code can also +;; access this parameter to check whether a transaction is active. + @node DSR Multithreading Considerations @comment node-name, next, previous, up @section Multithreading Considerations @cindex Multithreading -@c utils locks -@c utils thread-vars +Generic locking utility functions + +Variable behavior in multithreading situations @node DSR Handling Serialization @comment node-name, next, previous, up @section Handling Serialization @cindex Serialization -@c #:deserialize #:serialize -@c #:serialize-symbol-complete -@c #:deserialize-from-base64-string -@c #:serialize-to-base64-string +Backends must initialize @ref{Class elephant:store-controller} with +internal serializer functions. Packages @code{elephant-serializer1} +and @code{elephant-serializer2} contains serialize and deserialize +methods on buffer-streams as defined in @code{elephant-memutil}. The +elephant functions @code{serialize} and @code{deserialize} dispatch on +the appropriate slot values of the store-controller. + +@verbatim +NOTE: This should perhaps become entirely the job of the data store to +decide how to serialize values and for a specific version, what +serializer to use. The elphant main package can define serializers +for use by different backends. +@end verbatim + +@include includes/fun-elephant-backend-serialize.texinfo +@include includes/fun-elephant-backend-deserialize.texinfo + +These utility functions are useful if a data store does not have the +ability to store variable length binary data. They are based on the +@code{cl-base64} library. + +@include includes/fun-elephant-backend-serialize-to-base64-string.texinfo +@include includes/fun-elephant-backend-deserialize-from-base64-string.texinfo -@node DSR Memory utilities +@node DSR Memory Utilities @comment node-name, next, previous, up @section Memory utilities @cindex Memory utilities -@node DSR Foreign libraries +Details about memory utilities here. + +@node DSR Foreign Libraries @comment node-name, next, previous, up @section Foreign libraries @cindex Foreign libraries +How foreign libraries are built and used via UFFI. What functions are +in the .asd files or main lisp code to build & load libraries? --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/30 14:34:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/01 14:33:29 1.3 @@ -1,9 +1,37 @@ +@c -*-texinfo-*- @node Elephant Design @comment node-name, next, previous, up -@section Elephant Design +@chapter Elephant Design @cindex design +Elephant's early architecture was tightly coupled to the Berkeley DB +API. Over time we've moved towards a more modular architecture to +support easy upgrading, repository migration, shared functionality +between data stores and general hygene. + +The current architecture is modularized into the following pieces: + +@verbatim +[Picture describing] Metaclass, Store controller, persistent objects, +data stores, serializer & memutils. Derived features such as class +indexing, migration, query system and root operations can also be +illustrated? +@end verbatim + +To get a feeling for what is happening inside elephant, it is probably +most useful to walk through the various major protocols: + +@itemize +@item Initialization of a store controller +@item Creating a persistent object +@item Operations on persistent slots +@item Operations on persistent collections +@item Implementing @code{with-transaction} +@end itemize + +@section Initializing a store controller + When the main elephant @code{open-store} function is called, it calls @code{get-controller} which grabs an existing store controller if the spec is identical, or builds a new controller. Building the @@ -13,4 +41,14 @@ that returns a @code{store-controller} subclass instance specific to that backend. -Elephant than calls open-controller +Elephant than calls open-controller to actually establish a connection +to or create the files of the data store. + +@section Persistent Object Creation +@section Persistent Slot Protocol +@section Persistent Slot Protocol +@section Persistent Collection Protocol +@section Implementing Transactions + + + --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/30 14:34:34 1.6 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/01 14:33:29 1.7 @@ -38,6 +38,7 @@ @insertcopying @end ifnottex +@ifhtml @menu * Table of Contents:: @end menu @@ -49,7 +50,7 @@ * Tutorial:: A basic ``getting started'' tutorial. * Installation:: Installation and test-suite procedures. * User Guide:: In depth discussion of all Elephant facilities and features. -* Usage scenarios:: Design scenarios for Elephant applications. +* Usage Scenarios:: Design scenarios for Elephant applications. * User API Reference:: Function and class documentation of the user API. * Elephant Design:: An overview of elephant's internal architecture. * Data Store API Reference:: Function level documentation for data store implementors. @@ -66,6 +67,8 @@ * Colophon:: @end menu +@end ifhtml + @node Table of Contents @unnumbered @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/30 14:34:34 1.5 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6 @@ -10,9 +10,9 @@ * Configuring Elephant:: Setting up Elephant and the configuration file. * Loading Elephant:: Loading Elephant and the data store loading protocol. * Berkeley DB Data Store:: Installing support for the Berkeley DB data store -* Berkeley DB Examples:: An example of installing and running the Berkeley DB data store. +* Berkeley DB Example:: An example of installing and running the Berkeley DB data store. * CL-SQL Data Store:: Install and connecting to the CL-SQL data store -* CL-SQL Examples:: An example of using the CL-SQL data store. +* CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows * Test Suites:: How to run and interpret the output of the regression test suite * Documentation:: Building documentation from texinfo sources. @@ -277,10 +277,6 @@ As of Elephant 0.3, Elephant has been tested to work with both Postgres, and SQLite 3, thanks do Dan Knapp. -@node Extension Status -@comment node-name, next, previous, up -@section Extension Status - As far as is known at this writing, all functionality except nested transaction support and cursor-puts supported by the BerkeleyDB backend is supported by the CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested @@ -302,13 +298,9 @@ the multi-repository version somewhat complicates the underlying persistent object management. -@node PostGres Examples -@comment node-name, next, previous, up -@section Setting up PostGres - -@node Setting up PostGres +@node CL-SQL Example @comment node-name, next, previous, up -@section Setting up PostGres +@section CL-SQL Example To set up a PostGres based back end, you should: @@ -351,6 +343,10 @@ sequences, and so on needed by the Elephant system will be created in the schema named ``test'' automatically. +@node Elephant on Windows +@comment node-name, next, previous, up +@section Elephant on Windows + @node Test Suites @comment node-name, next, previous, up @section Test Suites --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 23:36:52 1.8 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/01 14:33:29 1.9 @@ -22,14 +22,20 @@ @section Store Controllers @cindex Store Controllers -Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller. +Store controllers provide the persistent storage for CLOS objects and +BTree collections. Any persistent operations must be done in the +context of a store controller. The default store-controller is stored +in a global variable. -@include includes/class-elephant-store-controller.texinfo @include includes/var-elephant-star-store-controller-star.texinfo +@c @include includes/class-elephant-store-controller.texinfo +@ref{Class elephant:store-controller} is associated with the following +user methods and macros: + +@include includes/macro-elephant-with-open-store.texinfo @include includes/fun-elephant-open-store.texinfo @include includes/fun-elephant-close-store.texinfo -@include includes/macro-elephant-with-open-store.texinfo @include includes/fun-elephant-get-from-root.texinfo @include includes/fun-elephant-add-to-root.texinfo @@ -42,12 +48,12 @@ @section Persistent Objects @cindex Persistent Objects -@include includes/class-elephant-persistent-metaclass.texinfo -@include includes/class-elephant-persistent.texinfo -@include includes/class-elephant-persistent-object.texinfo +@ref{Class elephant:persistent-metaclass} can be used as the +:metaclass argument in a defclass form to create a persistent object. +Slots of the metaclass take the :index and :transient keyword +arguments and the class accepts the :index keyword argument. @include includes/macro-elephant-defpclass.texinfo - @include includes/fun-elephant-drop-pobject.texinfo @node Persistent Object Indexing @@ -94,22 +100,26 @@ interface is provided for reference only, a new system is under development for the 0.7 release. -@include includes/fun-elephant-get-query-results.texinfo -@include includes/fun-elephant-map-class-query.texinfo +@c @include includes/fun-elephant-get-query-results.texinfo +@c @include includes/fun-elephant-map-class-query.texinfo @node Collections @comment node-name, next, previous, up @section Collections @cindex Collections -@include includes/class-elephant-persistent-collection.texinfo -@include includes/class-elephant-btree.texinfo -@include includes/class-elephant-indexed-btree.texinfo -@include includes/class-elephant-btree-index.texinfo +Persistent collections inherit from @ref{Class +elephant:persistent-collection} and consist of the @ref{Class +elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class +elephant:btree-index} classes. The following operations are defined +on most of these classes. More information can be found in @ref{Using +BTrees} and @ref{Secondary Indices}. @include includes/fun-elephant-get-value.texinfo @include includes/fun-elephant-setf-get-value.texinfo @include includes/fun-elephant-remove-kv.texinfo +@include includes/fun-elephant-existsp.texinfo + @include includes/fun-elephant-map-btree.texinfo @include includes/fun-elephant-map-index.texinfo @@ -118,17 +128,17 @@ @include includes/fun-elephant-get-primary-key.texinfo @include includes/fun-elephant-remove-index.texinfo - @node Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors -@include includes/class-elephant-cursor.texinfo -@include includes/class-elephant-secondary-cursor.texinfo +Cursors are objects of type cursor (@pxref{Class elephant:cursor}) +which provide methods for complex traversals of BTrees. + +@include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo -@include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-cursor-current.texinfo @include includes/fun-elephant-cursor-delete.texinfo --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/30 14:34:34 1.10 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11 @@ -15,6 +15,7 @@ * Persistent collections:: Keep track of collections of values. * Indexing Persistent Classes:: Simple way to keep track of persistent instances. * Using Transactions:: Providing ACID database properties. +* Advanced Topics:: Additional Elephant features covered in the User Guide. @end menu @node Overview @@ -704,14 +705,14 @@ @end lisp The @ref{User Guide} contains a descriptions of the advanced features -of @ref{Class indices} such as ``derived indicies'' that allow you to +of @ref{Class Indices} such as ``derived indicies'' that allow you to order classes according to an arbitrary function, a dynamic API for adding and removing slots and how to set a policy for resolving conflicts between the code image and a database where the indexing specification differs. This same facility is also available for your own use. For more -information @pxref{Using Indexed BTrees}. +information @pxref{Secondary Indices}. @node Using Transactions @@ -914,8 +915,71 @@ complicated. The best strategy at the beginning is a conservative one, break things up into the smallest logical sets of primitive operations and only wrap higher level functions in transactions when -they absolutely have to commit together. See @ref{Transaction details} -for the full details and @pxref{Usage scenarios} for more examples of +they absolutely have to commit together. See @ref{Transaction Details} +for the full details and @pxref{Usage Scenarios} for more examples of how systems can be designed and tuned using transactions. +@node Advanced Topics +@comment node-name, next, previous, up +@section Advanced Topics + +The tutorial covers the essential topics and concepts for using +Elephant. Many people will find that these features are the ones that +are most often needed and used in ordinary applications. + +More sophisticated uses of Elephant may require additional features +that are covered in the user guide. The following is a list of major +features in the user guide that were not covered in this tutorial. + +@itemize +@item @strong{Class Heirarchies and Queries} + There are some subtle issues to take into account when querying +persistent classes. For example, how do you query a base class of +type people to get subclass instances such as employee, manager, +consultant, etc? +@item @strong{Derived Class Indices} + You can create your own indices for classes that are arbitrary +lisp functions of the persistent object. +@item @strong{Class Definition/Database Conflict Resolution} + When you startup lisp, there are potential conflicts between the +class definition and the indexing records in the database. There are +some constraints to account for and some facilities to manage +how slots, class indices and +@item @strong{Dynamic Class Index Management} + It is possible to add and remove indexes from classes at runtime. +@item @strong{BTree Cursors} + If you need to do more than iterate over a collection, or you need +to delete elements of the collection as you iterate cursors are an +important data structure. They implement a variety of operators for +moving backward and forward over a btree, including ranged operations +and iterating of duplicate or unique values. +@item @strong{Indexed BTrees} + Indexed BTrees are just like BTrees, except it is possible to add +indexes which are BTrees who's values are primary keys in the parent +@code{indexed-btree}. This allows for multiple ordering and groupings +of the values of a BTree. +@item @strong{Using the Map Operators} + Mapping operators can be very efficient if properly utilized. +@item @strong{Handling Errors and Conditions} + There are a variety of errors that can occur in Elephant that need +to be dealt with by applications. +@item @strong{Deadlock Detection in Berkeley DB} + Berkeley DB requires an external process to detect deadlock +conditions among transactions. The :deadlock-detect keyword argument +to open-store for Berkeley DB specs will launch this process on most +lisps. +@item @strong{Using Multiple Stores} + Multiple store controllers can be open simultaneously. However it +does make the code more complex and you need to be careful about how +you use them to avoid crashes and other unpleasant side effects. +@item @strong{Custom Transaction Architecture} + You can implement your own version of with-transaction using the +underlying controller methods for starting, aborting and committing +transactions. You had better know what you are doing, however! +@item @strong{Using Multiple Threads and Processes} + What constraints must be accommodated to use Elephant data stores in +multiple threads? What capabilities are there to share data stores +among multiple processes or machines? +@end itemize +Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/30 14:34:34 1.4 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5 @@ -8,18 +8,19 @@ @menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. -* Reachability:: Determining liveness in a store. * Persistent objects:: All the dirt on persistent objects. -* Class indices:: In-depth discussion about indexing persistent indices. +* Class Indices:: In-depth discussion about indexing persistent indices. * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. -* Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls. +* Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Multi-repository Operation:: Specifying repositories. +* Multi-threaded Applications:: What considerations are required for safe multi-threading +* Multiple Processes and Distributed Applications:: Can elephant be run on multiple CPUs and multiple machines? * Repository Migration and Upgrade:: How to move objects from one repository to another. -* Garbage collection:: How to recover storage and OIDs in long-lived repositories. -* Performance tuning:: How to get the most from Elephant. +* Garbage Collection:: How to recover storage and OIDs in long-lived repositories. +* Performance Tuning:: How to get the most from Elephant. @end menu @node Persistent objects @@ -83,9 +84,15 @@ @code{with-open-controller} macro. Opening and closing a controller is very expensive. +@node Serialization details +@comment node-name, next, previous, up +@section Serialization details + +Empty. + @node Class Indices @comment node-name, next, previous, up -@section Class Indicies +@section Class Indices You can enable/disable class indexing for an entire class. When you disable indexing all references to instances of that class are lost. If you re-enable @@ -130,6 +137,10 @@ somewhat user customizable; documentation for this exists in the source file referenced above. +@node Querying persistent instances +@comment node-name, next, previous, up +@section Querying persistent instances + @node Using BTrees @comment node-name, next, previous, up @section Using BTrees @@ -147,10 +158,127 @@ blocks of data is relatively inexpensive after a seek and comparisons on objects that are stored in memory is cheap. +@node Secondary Indices +@comment node-name, next, previous, up +@section Secondary Indices + +Empty. + +@node Using Cursors +@comment node-name, next, previous, up +@section Using Cursors + +Empty. -@node Repository Migration +@node Transaction Details @comment node-name, next, previous, up -@section Repository Migration +@section Transaction Details + +;; Transaction architecture: +;; +;; User and designer considerations: +;; - *current-transaction* is reserved for use by dynamic transaction context. The default global +;; value must always be null (no transaction). Each backend can set it to a different parameter +;; within the dynamic context of an execute-transaction. +;; - Any closures returned from within a transaction cannot bind *current-transaction* +;; - Only a normal return value will result in the transaction being committed, any non-local exit +;; results in a transaction abort. If you want to do something more sophisticated, roll your own +;; using controller-start-transaction, etc. +;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc) +;; knowing that the transaction will be aborted +;; + +@node Multi-threaded Applications +@comment node-name, next, previous, up +@section Multi-threaded Applications + +Sleepycat plays well with threads and processes. The store controller +is thread-safe by default, that is, can be shared amongst threads. +This is set by the @code{:thread} key. Transactions may not be shared +amongst threads except serially. One thing which is NOT thread and +process safe is recovery, which should be run when no one is else is +talking to the database environment. Consult the Sleepycat docs for +more information. + +Elephant uses some specials to hold parameters and buffers. If you're +using a natively threaded lisp, you can initialize these specials to +thread-local storage by using the @code{run-elephant-thread} function, +assuming your lisp creates thread-local storage for let-bound +specials. (This functionality is currently broken) + +Persisting ordinary aggregate types (e.g. NOT persistent classes or +btrees) suffers from something called "merge-conflicts." Since +updating one value of an aggregate object requires the entire object +to be written to the database, in heavily threaded situations you may +overwrite changes another thread or process has committed. This is +not protected by transactions! + +Consider two processes operating on the same cons: + +@code{ +-----start---read--update-car--write--commit---------------- +-start------read--update-cdr-----------------write--commit-- +} + +Although the first process successfully committed its transaction, its +work (writing to the car) will be erased by the second transaction +(which writes both the car and cdr.) + +Persistent classes and persistent collections do not suffer from +merge-conflicts, since each slot / entry is a separate database entry. + +@node Multi-repository Operation +@comment node-name, next, previous, up +@section Multi-repository Operation + +Elephant now keeps a small hashtables that maps ``database specifications'' into +actual database connections. + +If a database spec is a string, it is assumed to be a BerkeleyDB path. +If it is a list, it is a assumed to be a CL-SQL connection specification. +For example: +@lisp +ELE-TESTS> *testdb-path* +"/home/read/projects/elephant/elephant/tests/testdb/" +ELE-TESTS> *testpg-path* +(:postgresql "localhost.localdomain" "test" "postgres" "") +ELE-TESTS> +@end lisp + +The tests now have a function @code{do-all-tests-spec} that take a spec and +based on its type attempt to open the correct kind of store controller and +perform the tests. + +The routine @code{get-controller} takes this specifiation. + +The basic strategy is that the ``database specification'' object is stored in +every persistent object and collection so that the repository can be found. + +In this way, objects that reside in different repositories can coexist within +the LISP object space, allowing data migration. + +;; Multiple stores + +;; Multiple store considerations: +;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to +;; work around: how to avoid writing one store with a transaction created in the context of another. +;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct; +;; this requirement may relax in the future +;; - The following macros accomodate multiple stores by requiring that execute-transaction return a +;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller +;; is the store instance it is associated with. A nested or ensured transaction is only indicated +;; in the call to execute transaction if the store controllers match, otherwise a new transaction +;; for that store is created + +@node Multiple Processes and Distributed Applications +@comment node-name, next, previous, up +@section Multiple Processes and Distributed Applications + +Can we do this? What do we have to say about this? + +@node Repository Migration and Upgrade +@comment node-name, next, previous, up +@section Repository Migration and Upgrade This version of Elephant supports migration between store controllers of any backend type. @@ -227,49 +355,17 @@ @code{*inhibit-slot-writes*} in your user method using @code{with-inhibited-slot-copy} a convenience macro. - -@node Threading +@node Garbage Collection @comment node-name, next, previous, up -@section Threading - -Sleepycat plays well with threads and processes. The store controller -is thread-safe by default, that is, can be shared amongst threads. -This is set by the @code{:thread} key. Transactions may not be shared -amongst threads except serially. One thing which is NOT thread and -process safe is recovery, which should be run when no one is else is -talking to the database environment. Consult the Sleepycat docs for -more information. - -Elephant uses some specials to hold parameters and buffers. If you're -using a natively threaded lisp, you can initialize these specials to -thread-local storage by using the @code{run-elephant-thread} function, -assuming your lisp creates thread-local storage for let-bound -specials. (This functionality is currently broken) - -Persisting ordinary aggregate types (e.g. NOT persistent classes or -btrees) suffers from something called "merge-conflicts." Since -updating one value of an aggregate object requires the entire object -to be written to the database, in heavily threaded situations you may -overwrite changes another thread or process has committed. This is -not protected by transactions! - -Consider two processes operating on the same cons: - -@code{ ------start---read--update-car--write--commit---------------- --start------read--update-cdr-----------------write--commit-- -} - -Although the first process successfully committed its transaction, its -work (writing to the car) will be erased by the second transaction -(which writes both the car and cdr.) +@section Garbage Collection -Persistent classes and persistent collections do not suffer from -merge-conflicts, since each slot / entry is a separate database entry. +GC is not implemented, but migration (@pxref{Repository Migration and +Upgrade}) will consolidate storage and recover OIDs which emulates GC. +No online solution is currently supported. -@node Performance Tips +@node Performance Tuning @comment node-name, next, previous, up -@section Performance Tips +@section Performance Performance is usually measured in transactions per second. Database reads are cheap. To get more transactions throughput, consider @@ -290,8 +386,8 @@ things. It is fast but consing with floats and doubles. YMMV with other values, though I've tried to make them fast. -Using @code{*auto-commit*} and not @code{with-transactions} is a great -way to have a huge number of transactions. You'll find that +Use @code{with-transactions} to avoid many automatic transactions, for +example you'll find that this construct @lisp (dotimes (i 1000) (add-to-root "key" "value")) @@ -300,9 +396,8 @@ is much slower than @lisp -(let ((*auto-commit* nil)) - (with-transaction () - (dotimes (i 1000) (add-to-root "key" "value")))) +(with-transaction () + (dotimes (i 1000) (add-to-root "key" "value")))) @end lisp since there's only 1 transaction in the latter. However storing @@ -319,32 +414,3 @@ mode is not suitable for use in web servers or other typically multi-threaded applications. -@node Multi-repository Operation -@comment node-name, next, previous, up -@section Multi-repository Operation - -Elephant now keeps a small hashtables that maps ``database specifications'' into -actual database connections. - -If a database spec is a string, it is assumed to be a BerkeleyDB path. -If it is a list, it is a assumed to be a CL-SQL connection specification. -For example: -@lisp -ELE-TESTS> *testdb-path* -"/home/read/projects/elephant/elephant/tests/testdb/" -ELE-TESTS> *testpg-path* -(:postgresql "localhost.localdomain" "test" "postgres" "") -ELE-TESTS> -@end lisp - -The tests now have a function @code{do-all-tests-spec} that take a spec and -based on its type attempt to open the correct kind of store controller and -perform the tests. - -The routine @code{get-controller} takes this specifiation. - -The basic strategy is that the ``database specification'' object is stored in -every persistent object and collection so that the repository can be found. - -In this way, objects that reside in different repositories can coexist within -the LISP object space, allowing data migration.

1 0

[elephant-cvs] CVS elephant
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv29083 Modified Files: TODO elephant.asd Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/TODO 2007/03/30 14:34:34 1.76 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/01 14:33:23 1.77 @@ -18,7 +18,9 @@ - Verify db_deadlock for other lisps (launch and kill background program I/F) Bugs: +- Fix awkward serializer API - Support for asdf-install? +- Edi's patches & suggestions for windows Test coverage: - Clean up interface to tests @@ -26,14 +28,13 @@ - Multi-threading stress tests? Ensure that there are conflicts and lots of serialization happening concurrently to make sure that multi-threading is in good shape (Henrik's code) - Unicode tests - - Ensure that variable length UTF-8 reps are automatically stored as UTF-16 + - Ensure that variable length UTF-8 reps are automatically stored as UTF-16? - Class / DB sychronization tests Documentation: ~ License and copyright file headers -- Redo tutorial -- Proper user guide - Update install, build and test procedures +- Proper user guide - Upgrade, migration and other system level issues - Performance and design issues - More notes about transaction performance @@ -42,7 +43,6 @@ - Add notes about optimize-storage - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) -- Add document section about backend interface & developer decisions 0.6.1 - Features COMPLETED to date ---------------------------------- --- /project/elephant/cvsroot/elephant/elephant.asd 2007/03/30 23:36:52 1.40 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/01 14:33:23 1.41 @@ -304,6 +304,7 @@ (:file "serializer2") ;; 0.6.1 db's (:file "unicode2") (:file "migrate") + (:file "query") (:file "backend")) :serial t :depends-on (memutil utils)))))

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 30 Mar '07

30 Mar '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv17159/src/elephant Modified Files: controller.lisp Log Message: Fixed symbol mgmt bug in serializer2 init --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:36:53 1.43 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:42:35 1.44 @@ -290,7 +290,7 @@ (setf (controller-serialize sc) (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER2))) (setf (controller-deserialize sc) - (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER2)))))) + (intern "DESERIALIZE" (find-package :ELEPHANT-SERIALIZER2)))))) ;; ;; Handling package changes in legacy databases

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 30 Mar '07

30 Mar '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv15195/src/elephant Modified Files: backend.lisp classes.lisp collections.lisp controller.lisp metaclasses.lisp serializer2.lisp variables.lisp Log Message: Sanitize class indexing option; more documentation stuff --- /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 17:46:14 1.14 +++ /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 23:36:53 1.15 @@ -76,6 +76,9 @@ #:transaction-store #:transaction-object #:execute-transaction + #:controller-start-transaction + #:controller-abort-transaction + #:controller-commit-transaction ;; Registration #:register-backend-con-init --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/24 12:16:03 1.24 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/30 23:36:53 1.25 @@ -47,30 +47,16 @@ ;; METACLASS INITIALIZATION AND CHANGES ;; ================================================ -(defmethod ensure-class-using-class :around ((class null) name &rest args &key index) - "Support the :index class option" - (let ((result (apply #'call-next-method class name (remove-keywords '(:index) args)))) - (when (and index (subtypep (type-of result) 'persistent-metaclass)) - (update-indexed-record result nil :class-indexed t)) - result)) - -(defmethod ensure-class-using-class ((class persistent-metaclass) name &rest args &key index) - "Support the :index class option on redefinition" - (let ((result (apply #'call-next-method class name (remove-keywords '(:index) args)))) - (when index - (update-indexed-record result nil :class-indexed t)) - result)) - -(defmethod shared-initialize :around ((class persistent-metaclass) slot-names &rest args &key direct-superclasses) +(defmethod shared-initialize :around ((class persistent-metaclass) slot-names &rest args &key direct-superclasses index) "Ensures we inherit from persistent-object." (let* ((persistent-metaclass (find-class 'persistent-metaclass)) (persistent-object (find-class 'persistent-object)) (not-already-persistent (loop for superclass in direct-superclasses never (eq (class-of superclass) persistent-metaclass)))) + (when index + (update-indexed-record class nil :class-indexed t)) (if (and (not (eq class persistent-object)) not-already-persistent) (apply #'call-next-method class slot-names -;; :direct-superclasses (cons persistent-object -;; direct-superclasses) args) :direct-superclasses (append direct-superclasses (list persistent-object)) args) (call-next-method)))) --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/03/25 14:57:49 1.19 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/03/30 23:36:53 1.20 @@ -339,15 +339,18 @@ (defun lisp-compare-equal (a b) (equal a b)) +(defgeneric map-btree (fn btree &rest args &key start end value) + (:documentation "Map btree maps over a btree from the value start to the value of end. + If values are not provided, then it maps over all values. BTrees + do not have duplicates, but map-btree can also be used with indices + in the case where you don't want access to the primary key so we + require a value argument as well for mapping duplicate value sets.")) + ;; NOTE: the use of nil for the last element in a btree only works because the C comparison ;; function orders by type tag and nil is the highest valued type tag so nils are the last ;; possible element in a btree ordered by value. + (defmethod map-btree (fn (btree btree) &rest args &key start end (value nil value-set-p)) - "Map btree maps over a btree from the value start to the value of end. - If values are not provided, then it maps over all values. BTrees - do not have duplicates, but map-btree can also be used with indices - in the case where you don't want access to the primary key so we - require a value argument as well for mapping duplicate value sets." (let ((end (if value-set-p value end))) (ensure-transaction (:store-controller (get-con btree)) (with-btree-cursor (curs btree) @@ -368,8 +371,8 @@ (funcall fn k v) (return nil))))))))) -(defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p)) - "Map-index is like map-btree but for secondary indices, it +(defgeneric map-index (fn btree &rest args &key start end value) + (:documentation "Map-index is like map-btree but for secondary indices, it takes a function of three arguments: key, value and primary key. As with map-btree the keyword arguments start and end determine the starting element and ending element, inclusive. @@ -377,7 +380,9 @@ the last element in the index. If you want to traverse only a set of identical key values, for example all nil values, then use the value keyword which will override any values of start - and end." + and end.")) + +(defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p)) (declare (dynamic-extent args) (ignorable args)) (let ((sc (get-con index)) --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 14:34:35 1.42 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:36:53 1.43 @@ -250,6 +250,12 @@ (when (member ver (rest row) :test #'equal)) t) nil)) +(defgeneric upgrade (sc spec) + (:documentation "Given an open store controller from a prior version, + open a new store specified by spec and migrate the + data from the original store to the new one, upgrading + it to the latest version")) + (defmethod upgrade ((sc store-controller) target-spec) (unless (upgradable-p sc) (error "Cannot upgrade ~A from version ~A to version ~A~%Valid upgrades are:~%~A" @@ -275,12 +281,16 @@ associated with the database version that is opened." (cond ((prior-version-p (database-version sc) '(0 6 0)) (setf (controller-serializer-version sc) 1) - (setf (controller-serialize sc) 'elephant-serializer1::serialize) - (setf (controller-deserialize sc) 'elephant-serializer1::deserialize)) + (setf (controller-serialize sc) + (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER1))) + (setf (controller-deserialize sc) + (intern "DESERIALIZE" (find-package :ELEPHANT-SERIALIZER1)))) (t (setf (controller-serializer-version sc) 2) - (setf (controller-serialize sc) 'elephant-serializer2::serialize) - (setf (controller-deserialize sc) 'elephant-serializer2::deserialize)))) + (setf (controller-serialize sc) + (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER2))) + (setf (controller-deserialize sc) + (intern "SERIALIZE" (find-package :ELEPHANT-SERIALIZER2)))))) ;; ;; Handling package changes in legacy databases --- /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/03/23 16:08:10 1.13 +++ /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/03/30 23:36:53 1.14 @@ -23,8 +23,11 @@ (declaim #-elephant-without-optimize (optimize (speed 3) (safety 1))) (defclass persistent () - ((%oid :accessor oid :initarg :from-oid) - (dbconnection-spec-pst :type (or list string) :accessor dbcn-spc-pst :initarg :dbconnection-spec-pst)) + ((%oid :accessor oid :initarg :from-oid + :documentation "All persistent objects have an oid") + (dbconnection-spec-pst :type (or list string) :accessor dbcn-spc-pst :initarg :dbconnection-spec-pst + :documentation "Persistent objects use a spec pointer to identify which store + they are connected to")) (:documentation "Abstract superclass for all persistent classes (common to user-defined classes and collections.)")) --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/03/30 14:34:35 1.34 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/03/30 23:36:53 1.35 @@ -24,8 +24,6 @@ (:import-from :elephant *circularity-initial-hash-size* get-cached-instance - controller-symbol-cache - controller-symbol-id-cache slot-definition-allocation slot-definition-name compute-slots --- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/03/30 17:45:41 1.13 +++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/03/30 23:36:53 1.14 @@ -64,10 +64,11 @@ ;; properly load in asdf due to some circular dependencies ;; between lisp files -(eval-when (load-toplevel compile-toplevel) +(eval-when (:compile-toplevel :load-toplevel) (mapcar (lambda (symbol) (intern symbol :elephant)) - '(get-cached-instance))) + '("GET-CACHED-INSTANCE" + "SET-DB-SYNCH")))

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 30 Mar '07

30 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv15195/doc Modified Files: Makefile data-store-reference.texinfo make-ref.lisp reference.texinfo Log Message: Sanitize class indexing option; more documentation stuff --- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 13:55:15 1.4 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/30 23:36:52 1.5 @@ -8,3 +8,5 @@ makeinfo -v --html --css-include=style.css --force elephant.texinfo makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo +pdf: includes-stuff + texi2dvi --pdf elphant.texinfo --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 17:28:50 1.3 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 23:36:52 1.4 @@ -38,10 +38,10 @@ * Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries @end menu -@node Registration +@node DSR Registration @comment node-name, next, previous, up @section Registration -@cindex Registration +@cindex Registration and Initialization Elephant looks at the first element of the specification list to determine which backend code base to use. The master table for this @@ -55,7 +55,13 @@ @include includes/fun-elephant-backend-register-backend-con-init.texinfo -@node Store Controllers +If the backend requires any special user-specified configuration, +augment the key types in config.sexp with what you need and use the +following function to access. + +@include includes/fun-elephant-backend-get-user-configuration-parameter.texinfo + +@node DSR Store Controllers @comment node-name, next, previous, up @section Store Controllers @cindex Store Controllers @@ -63,9 +69,10 @@ Subclass store-controller and implement store and close controller which are called by open-store and close-store respectively. -@include includes/class-elephant-backend-store-controller.texinfo +@include includes/class-elephant-store-controller.texinfo @include includes/fun-elephant-backend-open-controller.texinfo @include includes/fun-elephant-backend-close-controller.texinfo +@include includes/fun-elephant-backend-connection-is-indeed-open.texinfo For upgrading and opening legacy databases it is important that a store be able to indicate which version of elephant was used to create @@ -79,21 +86,24 @@ There are some utilities for serializing simple data without a serializer using the memutil package. -@include - -These functions are useful utilities for implementing -store-controllers. +@include includes/fun-elephant-backend-serialize-database-version-key.texinfo +@include includes/fun-elephant-backend-serialize-database-version-value.texinfo +@include includes/fun-elephant-backend-deserialize-database-version-value.texinfo -@include includes/fun-elephant-backend-get-con.texinfo -@include includes/fun-elephant-backend-oid.texinfo -@include includes/fun-elephant-backend-next-oid.texinfo -@include includes/fun-elephant-backend-connection-is-indeed-open.texinfo -@include includes/fun-elephant-get-user-configuration-parameter.texinfo -@node Slot Access +@node DSR Persistent Objects and Slot Access @comment node-name, next, previous, up @section Slot Access -@cindex Slot Access +@cindex Persistent Objects and Slot Access + +@include includes/class-elephant-persistent.texinfo +@include includes/fun-elephant-backend-get-con.texinfo +@c @include includes/fun-elephant-backend-oid.texinfo + +All objects require a unique id. During new object creation the +backend is asked to produce a unique id. + +@include includes/fun-elephant-backend-next-oid.texinfo These functions are called by the metaclass protocol to support operations on persistent class slots. @@ -103,7 +113,7 @@ @include includes/fun-elephant-backend-persistent-slot-boundp.texinfo @include includes/fun-elephant-backend-persistent-slot-makunbound.texinfo -@node Collections +@node DSR Collections @comment node-name, next, previous, up @section Collections @cindex Collections @@ -133,7 +143,12 @@ @include includes/fun-elephant-get-index.texinfo @include includes/fun-elephant-remove-index.texinfo -@node Cursors +Critical to indexing and queries are the map operators for collections + +@include includes/fun-elephant-map-btree.texinfo +@include includes/fun-elephant-map-index.texinfo + +@node DSR Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors @@ -143,22 +158,28 @@ @c #:cursor-oid @c #:cursor-initialized-p -@node Transactions +@node DSR Transactions @comment node-name, next, previous, up @section Transactions @cindex Transactions -@c #:*current-transaction* -@c #:make-transaction-record -@c #:transaction-store -@c #:transaction-object - -@c #:execute-transaction -@c #:controller-start-transaction -@c #:controller-commit-transaction -@c #:controller-abort-transaction +These functions must be implemented or stubbed in any +backend. + +@include includes/fun-elephant-backend-execute-transaction.texinfo +@include includes/fun-elephant-backend-controller-start-transaction.texinfo +@include includes/fun-elephant-backend-controller-commit-transaction.texinfo +@include includes/fun-elephant-backend-controller-abort-transaction.texinfo + +These are supporting functions and variables for implementing +transactions. + +@include includes/var-elephant-backend-star-current-transaction-star.texinfo +@include includes/fun-elephant-backend-make-transaction-record.texinfo +@include includes/fun-elephant-backend-transaction-store.texinfo +@include includes/fun-elephant-backend-transaction-object.texinfo -@node Multithreading Considerations +@node DSR Multithreading Considerations @comment node-name, next, previous, up @section Multithreading Considerations @cindex Multithreading @@ -166,7 +187,7 @@ @c utils locks @c utils thread-vars -@node Handling Serialization +@node DSR Handling Serialization @comment node-name, next, previous, up @section Handling Serialization @cindex Serialization @@ -176,12 +197,12 @@ @c #:deserialize-from-base64-string @c #:serialize-to-base64-string -@node Memory utilities +@node DSR Memory utilities @comment node-name, next, previous, up @section Memory utilities @cindex Memory utilities -@node Foreign libraries +@node DSR Foreign libraries @comment node-name, next, previous, up @section Foreign libraries @cindex Foreign libraries --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/30 14:34:34 1.5 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/30 23:36:52 1.6 @@ -1,29 +1,39 @@ (require 'asdf) -(asdf:operate 'asdf:load-op 'elephant-tests) +(asdf:operate 'asdf:load-op 'elephant) +(load (merge-pathnames + #p"src/elephant/query" + (asdf:component-pathname (asdf:find-system 'elephant)))) + (defparameter include-dir-path (namestring (merge-pathnames #p"doc/includes/" - (asdf:component-pathname (asdf:find-system 'elephant-tests))))) + (asdf:component-pathname (asdf:find-system 'elephant))))) (defparameter docstrings-path (namestring (merge-pathnames #p"doc/docstrings.lisp" - (asdf:component-pathname (asdf:find-system 'elephant-tests))))) + (asdf:component-pathname (asdf:find-system 'elephant))))) (sb-posix:chdir include-dir-path) (load docstrings-path) +(in-package :elephant) + +(defclass simple-store-controller (store-controller) + ()) + (defun make-docs () - (elephant:open-store elephant-tests::*testbdb-spec*) - (make-instance 'elephant::persistent-collection) - (make-instance 'elephant::secondary-cursor) - (make-instance 'elephant::indexed-btree) - (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" - (find-package :elephant) - (find-package :elephant-backend) - (find-package :elephant-memutil) - (find-package :elephant-system))) + (let ((sc (make-instance 'simple-store-controller))) + (setf (controller-spec sc) nil) + (make-instance 'elephant::persistent-collection :sc sc :from-oid 10) + (make-instance 'elephant::secondary-cursor) + (make-instance 'elephant::indexed-btree :sc sc :from-oid 10) + (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" + (find-package :elephant) + (find-package :elephant-backend) + (find-package :elephant-memutil) + (find-package :elephant-system)))) (make-docs) --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 14:34:34 1.7 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 23:36:52 1.8 @@ -110,12 +110,15 @@ @include includes/fun-elephant-get-value.texinfo @include includes/fun-elephant-setf-get-value.texinfo @include includes/fun-elephant-remove-kv.texinfo +@include includes/fun-elephant-map-btree.texinfo +@include includes/fun-elephant-map-index.texinfo @include includes/fun-elephant-add-index.texinfo @include includes/fun-elephant-get-index.texinfo @include includes/fun-elephant-get-primary-key.texinfo @include includes/fun-elephant-remove-index.texinfo + @node Cursors @comment node-name, next, previous, up @section Cursors @@ -125,7 +128,6 @@ @include includes/class-elephant-secondary-cursor.texinfo @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo -@include includes/fun-elephant-map-btree.texinfo @include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-cursor-current.texinfo @@ -163,15 +165,10 @@ @include includes/macro-elephant-with-transaction.texinfo -@include includes/var-elephant-star-auto-commit-star.texinfo -@include includes/var-elephant-star-current-transaction-star.texinfo -@include includes/fun-elephant-start-ele-transaction.texinfo -@include includes/fun-elephant-commit-transaction.texinfo -@include includes/fun-elephant-abort-transaction.texinfo - @node Migration and Upgrading @comment node-name, next, previous, up @section Migration and Upgrading @cindex Migration and Upgrading @include includes/fun-elephant-migrate.texinfo +@include includes/fun-elephant-upgrade.texinfo

1 0