Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv32267
Modified Files: Makefile make-ref.lisp Added Files: data-store-reference.texinfo elephant-design.texinfo lisp-data-store.texinfo scenarios.texinfo style.css user-guide.texinfo Removed Files: notes.texinfo Log Message: First batch of edits for new user manual
--- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 12:16:02 1.3 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 13:55:15 1.4 @@ -1,8 +1,10 @@
+all: docs
+includes-stuff: + cd includes; sbcl < ../make-ref.lisp
docs: includes-stuff - makeinfo -v --html --force elephant.texinfo + makeinfo -v --html --css-include=style.css --force elephant.texinfo + makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo
-includes-stuff: - cd includes; sbcl < ../make-ref.lisp --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 12:16:02 1.3 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 13:55:15 1.4 @@ -1,10 +1,21 @@ (require 'asdf) (asdf:operate 'asdf:load-op 'elephant-tests) -(sb-posix:chdir "/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/") -(load "/Users/eslick/Work/fsrc/elephant-cvs/doc/docstrings.lisp") +(defparameter include-dir-path + (namestring + (merge-pathnames + #p"doc/includes/" + (asdf:component-pathname (asdf:find-system 'elephant-tests))))) + +(defparameter docstrings-path + (namestring + (merge-pathnames + #p"doc/docstrings.lisp" + (asdf:component-pathname (asdf:find-system 'elephant-tests))))) + +(sb-posix:chdir include-dir-path) +(load docstrings-path)
(defun make-docs () -;; (when (check-complete) (when t (elephant:open-store elephant-tests::*testbdb-spec*) (make-instance 'elephant::persistent-collection)
--- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*-
@node Data Store API Reference @comment node-name, next, previous, up @chapter Data Store API Reference @cindex Data Store API Reference @cindex Data Store @cindex API Reference
These are the functions that need to be overridden to implement support for a data store backend. Included are the exported elephant functions that need methods defined on them. Some functions here are utilities from the main elephant package that support store implementations. Migration, class indices and query interfaces are implemented on top of the store API and require no special support by implementors.
@menu * Registration:: Register the backend to parse controller specifications * Store Controllers:: Subclassing the store controller. * Slot access:: Support for metaprotocol slot access. * Collections:: BTrees and indices. * Cursors:: Traversing BTrees. * Transactions:: Transaction implementation. * Multithreading:: Multithreading considerations. * Serialization:: Facilities for serializing objects. * C Utilities:: Writing primitive C types. * Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries @end menu
@node Registration @comment node-name, next, previous, up @section Registration @cindex Registration
@include includes/fun-elephant-register-backend-con-init.texinfo @include includes/fun-elephant-lookup-backend-con-init.texinfo
@node Store Controllers @comment node-name, next, previous, up @section Store Controllers @cindex Store Controllers
Subclass store-controller and implement store and close controller which are called by open-store and close-store respectively.
@include includes/fun-elephant-store-controller.texinfo @include includes/fun-elephant-backend-open-controller.texinfo @include includes/fun-elephant-backend-close-controller.texinfo
The slots for these accessors must be initialized.
@include includes/fun-elephant-backend-database-version.texinfo @include includes/fun-elephant-backend-controller-serialize.texinfo @include includes/fun-elephant-backend-controller-deserialize.texinfo @include includes/fun-elephant-backend-root.texinfo @include includes/fun-elephant-backend-class-root.texinfo
These functions are important utilities for implementing store-controllers.
@include includes/fun-elephant-backend-oid.texinfo @include includes/fun-elephant-backend-get-con.texinfo @include includes/fun-elephant-backend-next-oid.texinfo @include includes/fun-elephant-backend-connection-is-indeed-open.texinfo
@node Slot Access @comment node-name, next, previous, up @section Slot Access @cindex Slot Access
These functions are called by the metaclass protocol to support operations on persistent class slots.
@include includes/fun-elephant-backend-persistent-slot-writer.texinfo @include includes/fun-elephant-backend-persistent-slot-reader.texinfo @include includes/fun-elephant-backend-persistent-slot-boundp.texinfo @include includes/fun-elephant-backend-persistent-slot-makunbound.texinfo
@node Collections @comment node-name, next, previous, up @section Collections @cindex Collections
@c #:btree #:btree-index #:indexed-btree @c #:build-indexed-btree #:build-btree #:existsp @c #:map-indices
@node Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors
@c #:cursor @c #:cursor-btree @c #:cursor-oid @c #:cursor-initialized-p
@node Transactions @comment node-name, next, previous, up @section Transactions @cindex Transactions
@c #:*current-transaction* @c #:make-transaction-record @c #:transaction-store @c #:transaction-object @c #:execute-transaction @c #:controller-start-transaction @c #:controller-commit-transaction @c #:controller-abort-transaction
@node Multithreading @comment node-name, next, previous, up @section Multithreading @cindex Multithreading
@node Serialization @comment node-name, next, previous, up @section Serialization @cindex Serialization
@c #:deserialize #:serialize @c #:serialize-symbol-complete @c #:deserialize-from-base64-string @c #:serialize-to-base64-string
@node Memory utilities @comment node-name, next, previous, up @section Memory utilities @cindex Memory utilities
@node Foreign libraries @comment node-name, next, previous, up @section Foreign libraries @cindex Foreign libraries
--- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 1.1 Debugger entered--Lisp error: (void-variable Design) eval(Design) eval-last-sexp-1(nil) eval-last-sexp(nil) call-interactively(eval-last-sexp) --- /project/elephant/cvsroot/elephant/doc/lisp-data-store.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/lisp-data-store.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*-
@node Lisp Data Store @comment node-name, next, previous, up @chapter Lisp Data Store @cindex Lisp Data Store @cindex Data Store @cindex API Reference
--- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*-
@node Usage Scenarios @comment node-name, next, previous, up @chapter Usage Scenarios @cindex Usage Scenarios
Sorry, haven't written this section yet.
Simple file replacement and indexing - Keep track of ordinary objects, ignore metaprotocol
Persist system objects - Intermingle persistent objects and regular objects - Look up objects using class indices
Full database system - storage, rich data models, references, queries, etc
Multithreaded web applications - DB + multithreading
Object-oriented data storage, large graph traversals
--- /project/elephant/cvsroot/elephant/doc/style.css 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/style.css 2007/03/24 13:55:15 1.1
--- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*-
@node User Guide @comment node-name, next, previous, up @chapter User Guide @cindex User Guide
@menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. * Reachability:: Determining liveness in a store. * Persistent objects:: All the dirt on persistent objects. * Class indices:: In-depth discussion about indexing persistent indices. * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. * Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Repository Migration and Upgrade:: How to move objects from one repository to another. * Garbage collection:: How to recover storage and OIDs in long-lived repositories. * Performance tuning:: How to get the most from Elephant. @end menu
@node Persistent objects @comment node-name, next, previous, up @section Persistent Objects
Finally, if you for some reason make an instance with a specified OID which already exists in the database, @code{initargs} take precedence over values in the database, which take precedences over @code{initforms}.
Also currently there is a bug where @code{initforms} are always evaluated, so beware. (What is the current model here?)
@node The Store Controller @comment node-name, next, previous, up @section The Store Controller
What is @code{open-store} doing? It creates a @code{store-controller} object, and sets the special @code{*store-controller*} to point to it. The store controller holds the handles to the database environment and tables, and some other bookkeeping. If for some reason you need to run recovery on the database (see sleepycat docs) you can specify that with the @code{:recover} and @code{:recover-fatal} keys.
To create one by hand one can do,
@lisp * (setq *store-controller* (make-instance 'store-controller :path "testdb")) => #<STORE-CONTROLLER @{49252F75@}>
* (open-controller *store-controller*) => #<STORE-CONTROLLER @{49252F75@}> @end lisp
but
@lisp * (open-store "testdb")) @end lisp
is the preferred mechanism.
This opens the environment and database. The @code{persistent-*} objects reference the @code{*store-controller*} special. (This is in part because slot accessors can't take additional arguments.) If for some reason you want to operate on 2 store controllers, you'll have to do that by flipping the @code{*store-controller*} special.
@code{close-store} closes the store controller. Alternatively @code{close-controller} can be called on a handle. Don't forget to do this or else you may need to run recovery later. There is a @code{with-open-controller} macro. Opening and closing a controller is very expensive.
@node{Using BTrees} @comment node-name, next, previous, up @section Using BTrees
The btree class are to hash-tables as persistent-objects are to ordinary objects. BTrees have a hash-table-like interface, but store their keys and values directly as rows in a Sleepycat BTree. Btrees may be persisted simply by their OID. Hence they have all the nice properties of persistent objects: identity, fast serialization / deserialization, no merge conflicts.....
(defvar *friends-birthdays* (make-btree)) => *FRIENDS-BIRTHDAYS*
(add-to-root "friends-birthdays" *friends-birthdays*) => #<BTREE {4951CF6D}>
(setf (get-value "Andrew" *friends-birthdays*) (encode-universal-time 0 0 0 22 12 1976)) => 2429071200
(setf (get-value "Ben" *friends-birthdays*) (encode-universal-time 0 0 0 14 4 1976)) => 2407298400
(get-value "Andrew" *friends-birthdays*) => 2429071200 => T
(decode-universal-time *) => 0 0 0 22 12 1976 2 NIL 6
Because of serialization semantics, BTrees hash on a value, not identity. This is probably ok for strings, numbers, and persistent things, but may be strange for other values.
@node Repository Migration @comment node-name, next, previous, up @section Repository Migration
This version of Elephant supports migration between store controllers of any backend type.
The tests @code{migrate1} - @code{migrate5} are demonstrations of this capability.
There is a single generic function @code{migrate} that is used to copy different object types to a target repository. It is assumed that typically migrate will be called on two repositories and all live objects (those reachable in the root or class-root) will be copied to the target repository via recursive calls to migrate for specific objects.
When persistent instances are copied, their internal pointer will be updated to point to the new repository so after migration the lisp image should be merely updated to refer to the target repository in the *store-controller* variable or whatever variable the application is using to store the primary controller instance.
There are some limitations to the current migration implementation: @enumerate @item Migrate currently will not handle circular list objects
@item Migrate does not support arrays with nested persistent objects
@item Indexed classes only have their class index copied if you use the top level migration. Objects will be copied without slot data if you try to migrate an object outside of a store-to-store migration due to the class object belonging to one store or another
@item Migrate assumes that after migration, indexed classes belong to the target store.
@item In general, migration is a one-time activity and afterwards (or after a validation test) the source store should be closed. Any failures in migration should then be easy to catch.
@item Each call to migration will be good about keeping track of already copied objects to avoid duplication. Duplication shouldn't screw up the semantics, just add storage overhead but is to be avoided. However this information is not saved between calls and there's no other way to do comparisons between objects across stores (different oid namespaces) so user beware of the pitfalls of partial migrations...
@item Migrate keeps a memory-resident hash of all objects; this means you cannot currently migrate a store that has more data than your main memory. (This could be fixed by keeping the oid table in the target store and deleting it on completion)
@item Migration does not maintain OID equivalence so any datastructures which index into those will have to have a way to reconstruct themselves (better to keep the object references themselves rather than oids in general) but they can overload the migrate method to accomplish this cleanly @end enumerate
Users can customize migration if they create unusual datastructures that are not automatically supported by the existing @code{migrate} methods. For example, a datastructure that stores only object OIDs instead of serialized object references will need to overload migrate to ensure that all referenced objects are in fact copied (otherwise the OIDs will just be treated as fixnums potentially leaving dangling references.
To customize migration overload a version of migrate to specialize on your specific persistent class type.
@lisp (defmethod migrate ((dst store-controller) (src my-class))) @end lisp
In the body of this method you can call @code{(call-next-method)} to get a destination repository object with all the slots copied over to the target repository which you can then overwrite. To avoid the default persistent slot copying, bind the dynamic variable @code{*inhibit-slot-writes*} in your user method using @code(with-inhibited-slot-copy () ...)} a convenience macro.
@node Threading @comment node-name, next, previous, up @section Threading
Sleepycat plays well with threads and processes. The store controller is thread-safe by default, that is, can be shared amongst threads. This is set by the @code{:thread} key. Transactions may not be shared amongst threads except serially. One thing which is NOT thread and process safe is recovery, which should be run when no one is else is talking to the database environment. Consult the Sleepycat docs for more information.
Elephant uses some specials to hold parameters and buffers. If you're using a natively threaded lisp, you can initialize these specials to thread-local storage by using the @code{run-elephant-thread} function, assuming your lisp creates thread-local storage for let-bound specials. (This functionality is currently broken)
[72 lines skipped]