March 2007 - elephant-cvs@common-lisp.net

[elephant-cvs] CVS elephant/doc
by ieslick 25 Mar '07

25 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv8936 Modified Files: tutorial.texinfo user-guide.texinfo Log Message: Latest documentation edits - still very broken so don't look! --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/24 12:16:02 1.7 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/25 11:04:38 1.8 @@ -6,15 +6,15 @@ @cindex Tutorial @menu -* Overview:: Overview of elphant's features +* Overview:: Overview of elephant's features. * Getting Started:: Opening and accessing a store. * The Store Root:: Accessing persistent data. * Serialization:: Storage semantics for lisp values. * Persistent Classes:: Persistent semantics for objects. * Rules about Persistent Classes:: What you need to know. -* Persistent collections:: Keep track of collections of objects. -* Class Indices:: Simple way to keep track of instances. -* Using Transactions:: Enabling ACID database properties. +* Persistent collections:: Keep track of collections of values. +* Indexing Persistent Classes:: Simple way to keep track of persistent instances. +* Using Transactions:: Providing ACID database properties. @end menu @node Overview @@ -377,82 +377,249 @@ the most recent commits, right? Note that this can be used as a weak form of IPC. But also note that in particular, if your slot value is not an immediate value, reading will cons or allocate the value. Gets -are not an expensive operation; you can perform tens of thousands of -primitive reads per second. However, if you're concerned, cache -large values. +are not an expensive operation; you can perform thousands to tens of +thousands of primitive reads per second. However, if you're +concerned, cache large values in memory. @node Persistent collections @comment node-name, next, previous, up @section Persistent collections +The remaining problem outlined in @ref{Serialization} is that +operations which mutate aggregate objects are not persistent. While +we solved this problem for objects, there is no collection type such +as arrays, hashes or lists which provide this ability. Elephant +provides two primary types of collections, a @code{btree} and a +@code{indexed-btree}. + +We will focus on the core concepts of BTrees in this section, for a +detailed review including the behavior of indexed BTrees, @pxref{Using +BTrees}, @ref{Secondary Indices} and @ref{Using Cursors} in the +@ref{User Guide}. +Elephant provides a rich data structure called a BTree for storing +large sets of key-value pairs. Every key-value pair is stored +independantly in Elephant just like persistent object slots. +Therefore they inherit all the nice properties of persistent objects: +identity, fast serialization / deserialization, no merge conflicts, +etc. + +The primary interface to @code{btree} objects is through +@code{get-value}. You can also @code{setf} @code{get-value} to store +key-value pairs. + +@lisp +(defvar *friends-birthdays* (make-btree)) +=> *FRIENDS-BIRTHDAYS* + +(add-to-root "friends-birthdays" *friends-birthdays*) +=> #<BTREE @{4951CF6D@}> + +(setf (get-value "Ben" *friends-birthdays*) + (encode-universal-time 0 0 0 14 4 1973)) +=> 2312600400 + +(setf (get-value "Andrew" *friends-birthdays*) + (encode-universal-time 0 0 0 22 12 1976)) +=> 2429071200 + +(get-value "Andrew" *friends-birthdays*) +=> 2429071200 +=> T + +(decode-universal-time *) +=> 0 + 0 + 0 + 22 + 12 + 1976 + 2 + NIL + 6 +@end lisp + +In addition to the hash-table like interface, @code{btree} stores +pairs sorted by the lisp value of the key, lowest to highest. This is +works well for numbers, strings, symbols and persistent objects, but +due to serialization semantics may be strange for other values like +arrays, lists, standard-objects, etc. + +Because elements are sorted by value, we should be able to iterate +over all the elements of the BTree in order. We entered the data in +reverse alphabetic order, but will read it out in alphabetical order. + +@lisp +(map-btree (lambda (k v) + (format t "name: ~A utime: ~A~%" k + (subseq (multiple-value-list (decode-universal-time v)) 3 6))) + *friends-birthdays*) +"Andrew" +"Ben" +=> NIL +@end lisp -@node Class Indices +But what if we want to read out our friends from oldest to youngest, +or youngest to oldest? In the @ref{User Guide}, specifically the +section on @ref{Secondary indices} you will discover ways to sort +according to the order defined by a lisp function of the key-value pair. + +@node Indexing Persistent Classes @comment node-name, next, previous, up -@section Class Indices +@section Indexing Persistent Classes + +Class indices simplify the recording and retrieving of persistent +objects. An indexed class stores every instance of the class that is +created, ensuring that every object is automatically persisted between +sessions. + +@lisp +(defpclass friend () + ((name :accessor name :initarg :name) + (birthday :initarg :birthday)) + (:index t)) +=> #<PERSISTENT-METACLASS FRIEND> + +(defmethod print-object ((f friend) stream) + (format stream "#<~A>" (name f))) + +(defun encode-birthday (dmy) + (apply #'encode-universal-time + (append '(0 0 0) dmy))) + +(defmethod (setf birthday) (dmy (f friend)) + (setf (slot-value f 'birthday) + (encode-birthday dmy)) + dmy) + +(defun decode-birthday (utime) + (subseq (multiple-value-list (decode-universal-time utime)) 3 6)) + +(defmethod birthday ((f friend)) + (decode-birthday (slot-value f 'birthday))) +@end lisp + +Notice the class argument ``:index t''. This tells Elephant to store +a reference to this class. Under the covers, there are a set of +btrees that keep track of classes, but we won't need to worry about +that as all the functionality has been nicely packaged for you. + +We also created our own birthday accessor for convenience so it +accepts and returns birthdays in a list consisting of month, day and +year such as @code{(27 3 1972)}. The index key will be the encoded +universal time, however. -Class indices are a very convenient way of gaining the efficiency that -BTrees provide. If a given object is most often sought by the value -of one of its slots, which is of course quite common, it is convenient -to define a class index on that slot, although the same functionality -can be gained in a more complicated way through the use of secondary -indices. - -The file @file{tests/testindexing.lisp} provides many useful examples -of both declaring class indexes and using the API to seek objects using them. - -The following code from that file in the test ``indexing-range'' demonstrates -the convenience of a class indexes and the function ``get-instances-by-range''. -Note in the definition of the ``slot1'' the keyword ``:index'' is used to -specify that this slot should be indexed. - -@lisp - (defclass idx-four () - ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) - (:metaclass persistent-metaclass)) - - - (defun make-idx-four (val) - (make-instance 'idx-four :slot1 val)) - - (with-transaction () - (mapc #'make-idx-four '(1 1 1 2 2 4 5 5 5 6 10))) - - (let ((x1 (get-instances-by-range 'idx-four 'slot1 2 6)) - (x2 (get-instances-by-range 'idx-four 'slot1 0 2)) - (x3 (get-instances-by-range 'idx-four 'slot1 6 15)) - ) - (format t " x1 = ~A~%" (mapcar #'slot1 x1)) - (format t " x2 = ~A~%" (mapcar #'slot1 x2)) - (format t " x3 = ~A~%" (mapcar #'slot1 x3)) -@end lisp - -Additionally, the test -@lisp -(do-test 'INDEXING-TIMING) -@end lisp -Can be used to judge the performance of indexing a medium sized dataset. - -The file @file{src/elephant/classindex.lisp} provides the source code and -some crisp documentation of the class indexing system. - -Note that for retrieving items, the API is provided by three functions: - -@lisp -(defgeneric get-instances-by-class (persistent-metaclass)) -(defgeneric get-instances-by-value (persistent-metaclass slot-name value)) -(defgeneric get-instances-by-range (persistent-metaclass slot-name start end)) -@end lisp - -By using these functions, any class that is a subclass of persistent-metaclass -can also be thought of as a container of all of its instances, which are -persistent in the database between lisp invocations. Moreover an individual -object can be looked up on O(log n) time via a value on which it is indexed. - -At the top of this same file, you will find the a description of the API -which can be used to dynamically add and remove indexes. (Adding and -removing indexes can also be performed by a re-execution of the ``defclass'' -macro with different values.) +Now we can easily manipulate all the instances of a class. + +@lisp +(defun print-friend (friend) + (format t " name: ~A birthdate: ~A~%" (name friend) (birthday friend))) + +(make-instance 'friend :name "Carlos" :birthday (encode-birthday '(1 1 1972))) +(make-instance 'friend :name "Adriana" :birthday (encode-birthday '(24 4 1980))) +(make-instance 'friend :name "Zaid" :birthday (encode-birthday '(14 8 1976))) + +(get-instances-by-class 'friends) +=> (#<Carlos> #<Adriana> #<Zaid>) + +(mapcar #'print-friend *) + name: Carlos birthdate: (1 1 1972) + name: Adriana birthdate: (24 4 1980) + name: Zaid birthdate: (14 8 1976) +=> (#<Carlos> #<Adriana> #<Zaid>) +@end lisp + +But what if we have thousands of friends? Aside from never getting +work done, our get-instances-by-class will be doing a great deal of +consing, eating up lots of memory and wasting our time. Fortunately +there is a more efficient way of dealing with all the instances of a +class. + +@lisp +(map-class #'print-friend 'friend) + name: Carlos birthdate: (1 1 1972) + name: Adriana birthdate: (24 4 1980) + name: Zaid birthdate: (14 8 1976) +=> NIL +@end lisp + +@code{map-class} has the advantage that it does not keep references to +objects after they are processed. The garbage collector can come +along, clear references from the weak instance cache so that your +working set is finite. The list version above conses all objects into +memory before you can do anything with them. The deserialization +costs are very low in both cases. + +Notice that the order in which the records are printed are not sorted +according to either name or birthdate. Elephant makes no guarantee +about the ordering of class elements, so you cannot depend on the +insertion ordering shown here. + +So what if we want ordered elements? How do we access our friends +according to name and birthdate? This is where slot indices come into +play. + +@lisp +(defpclass friend () + ((name :accessor name :initarg :name :index t) + (birthday :initarg :birthday :index t))) +@end lisp + +Notice the :index argument to the slots. Also notice that we dropped +the class :index argument. Specifying that a slot is indexed +automatically registers the class as indexed. While slot indices +increase the cost of writes and disk storage, each entry is only +slightly larger than the size of the slot value. Numbers, small +strings and symbols are good candidate types for indexed slots, but +any value may be used, even different types. + +Once we've indexed a slot, we can use another set of +@code{get-instances} and @code{map} functions to access objects +in-order and by their slot value. + +@lisp +(get-instances-by-value 'friends 'name "Carlos") +=> (#<Carlos>) + +(get-instances-by-range 'friends 'name "Adam" "Devin") +=> (#<Adriana> #<Carlos>) + +(get-instances-by-range 'friend 'birthday (encode-birthday '(1 1 1974)) (encode-birthday '(31 12 1984))) +=> (#<Zaid> #<Adriana>) + +(mapc #'print-friend *) + name: Zaid birthdate: (14 8 1976) + name: Adriana birthdate: (24 4 1980) +=> (#<Zaid> #<Adriana>) + +(map-class-index #'print-friend 'friend 'name "Carlos" "Carlos") + name: Carlos birthdate: (1 1 1972) +=> NIL + +(map-class-index #'print-friend 'friend 'name "Adam" "Devin") + name: Adriana birthdate: (24 4 1980) + name: Carlos birthdate: (1 1 1972) +=> NIL + +(map-class-index #'print-friend 'friend 'birthday + (encode-birthday '(1 1 1974)) + (encode-birthday '(31 12 1984))) + name: Zaid birthdate: (14 8 1976) + name: Adriana birthdate: (24 4 1980) +=> NIL + +(map-class-index #'print-friend 'friend 'birthday nil (encode-birthday '(10 10 1978))) + name: Carlos birthdate: (1 1 1972) + name: Zaid birthdate: (14 8 1976) +=> NIL + +(map-class-index #'print-friend 'friend 'birthday + (encode-birthday '(10 10 1975)) + nil) + name: Zaid birthdate: (14 8 1976) + name: Adriana birthdate: (24 4 1980) +=> NIL +@end lisp You can enable/disable class indexing for an entire class. When you disable indexing all references to instances of that class are lost. If you re-enable @@ -606,16 +773,14 @@ to retry the transaction a fixed number of times by re-executing the whole body. -You can see in the example that a single statement in lisp can include -several primitive Elephant operations as in the decf statement in -withdraw. It takes some careful thinking to properly implement -transactions, for a complete treatment @pxref{Transaction details}. - -The other thing transactions can give us is the ability to put off -synchronizing our data to disk. The expensive part of persistent -writes is flushing data to disk. Since a transaction caches values, -all the read and written values are kept in memory until the -transaction is complete, this can dramatically improve performance. +The other value transactions provide is the capability to delay +flushing dirty data to disk. The most time-intensive part of +persistent operations is flushing newly written data to disk. Using +the default auto-commit behavior requires a flush on every operation +which can become very expensive. Because a transaction caches values, +all the values read or written are cached in memory until the +transaction completes, dramatically decreasing the number of flushes +and the total time taken. @lisp (defpclass test () @@ -625,8 +790,9 @@ (make-instance 'test :slot1 i))) @end lisp -This can take a long time. Here each new objects that is created has -to independantly write its value to disk and accept a disk flush cost. +This can take a long time, well over a minute on the CLSQL data store. +Here each new objects that is created has to independantly write its +value to disk and accept a disk flush cost. @lisp (time (with-transaction () @@ -634,7 +800,8 @@ (make-instance 'test :slot1 i)))) @end lisp -Here, ....... +Wrapping this operation in a transaction dramatically increases the +time from 10's of seconds to a second or less. @lisp (time (with-transaction () @@ -642,14 +809,64 @@ (make-instance 'test :slot1 i)))) @end lisp -These are huge differences in performance! -Of course since we are caching all this data in memory, we cannot have -infinitely sized transactions. Large operations need to get split up -into subtransactions. When dealing with persistent objects a good -rule of thumb is to keep your transactions less than 1000 at a time. +When we increase the number of objects within the transaction, the +time cost does not go up linearly. This is because the total time to +write a hundred simple objects is still dominated by the final +synchronization step. + [56 lines skipped] --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 1.1 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/25 11:04:38 1.2 @@ -76,49 +76,71 @@ @code{with-open-controller} macro. Opening and closing a controller is very expensive. + +@node{Class indices} +@comment node-name, next, previous, up +@section Class indicies + +You can enable/disable class indexing for an entire class. When you disable +indexing all references to instances of that class are lost. If you re-enable +class indexing only newly created classes will be stored in the class index. +You can manually restore them by using @code{find-class-index} to get the +clas index BTree if you have an alternate in-memory index. + +You can add/remove a secondary index for a slot. So long as the class index +remains, this can be done multiple times without losing any data. + +There is also a facility for defining 'derived slots'. These can be non-slot +parameters which are a function of the class's persistent slot values. For +example you can use an index to keep an alternate representation available +for fast indexing. If an object has an x,y coordinate, you could define a +derived index for r,theta which stored references in polar coordinates. +These would be ordered so you could iterate over a class-index to get objects +in order of increasing radius from the origin or over a range of theta. + +Beware, however, that derived indices have to compute their result every +time you update any persistent instance's slot. This is because there is +no way to know which persistent slots the derived index value(s) depends +on. Thus there is a fairly significant computational cost to objects +with frequent updates having derived indices. The storage cost, however, +may be less as all that is added is the index value and an OID reference +into the class index. To add a slot value you add a serialized +OID+class-ref+slotname to index value which can be much larger if you +use long slotnames and package names and unicode. + +Thus, the question of if and how a given class should be indexed is +very flexible and dynamic, and does not need to be determined at the +beginning of your development. This represents the ability to ``late bind'' +the decision of what to index. + +In general, there is always a tradeoff: an indexed slot increases storage +associated with that slot and slows down write operations. Reads however remain +as fast as for unindexed persistent slots. The Elephant system +makes it simple to choose where and when one wants to utilize this tradeoff. + +Finally, that file @file{src/elephant/classindex-utils.lisp} documents +tools for handling class redefinitions and the policy that should be +used for synchronizing the classes with the database. This process is +somewhat user customizable; documentation for this exists in the source +file referenced above. + @node{Using BTrees} @comment node-name, next, previous, up @section Using BTrees -The btree class are to hash-tables as persistent-objects are to -ordinary objects. BTrees have a hash-table-like interface, but store -their keys and values directly as rows in a Sleepycat BTree. Btrees -may be persisted simply by their OID. Hence they have all the nice -properties of persistent objects: identity, fast serialization / -deserialization, no merge conflicts..... - - (defvar *friends-birthdays* (make-btree)) - => *FRIENDS-BIRTHDAYS* - - (add-to-root "friends-birthdays" *friends-birthdays*) - => #<BTREE {4951CF6D}> - - (setf (get-value "Andrew" *friends-birthdays*) - (encode-universal-time 0 0 0 22 12 1976)) - => 2429071200 - - (setf (get-value "Ben" *friends-birthdays*) - (encode-universal-time 0 0 0 14 4 1976)) - => 2407298400 - - (get-value "Andrew" *friends-birthdays*) - => 2429071200 - => T - - (decode-universal-time *) - => 0 - 0 - 0 - 22 - 12 - 1976 - 2 - NIL - 6 - -Because of serialization semantics, BTrees hash on a value, not -identity. This is probably ok for strings, numbers, and persistent -things, but may be strange for other values. +A BTree is a data structure designed for on-disk databases. +Traditional binary trees are a tree structure that stores a key and +value and two links in each node. To get to a value, you compare your +query key to the node key. If it is equal, return the value in this +node. If it is less, follow the first link and if it is greater, +follow the second link. The problem here is that every link requires +a disk seek. + +The BTree exploits the properties of memory/disk data heirarchies. +The key properties are that disk seeks are expensive, loading large +blocks of data is relatively inexpensive after a seek and comparisons +on objects that are stored in memory is cheap. + @node Repository Migration @comment node-name, next, previous, up

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv32267 Modified Files: Makefile make-ref.lisp Added Files: data-store-reference.texinfo elephant-design.texinfo lisp-data-store.texinfo scenarios.texinfo style.css user-guide.texinfo Removed Files: notes.texinfo Log Message: First batch of edits for new user manual --- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 12:16:02 1.3 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 13:55:15 1.4 @@ -1,8 +1,10 @@ +all: docs +includes-stuff: + cd includes; sbcl < ../make-ref.lisp docs: includes-stuff - makeinfo -v --html --force elephant.texinfo + makeinfo -v --html --css-include=style.css --force elephant.texinfo + makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo -includes-stuff: - cd includes; sbcl < ../make-ref.lisp --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 12:16:02 1.3 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 13:55:15 1.4 @@ -1,10 +1,21 @@ (require 'asdf) (asdf:operate 'asdf:load-op 'elephant-tests) -(sb-posix:chdir "/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/") -(load "/Users/eslick/Work/fsrc/elephant-cvs/doc/docstrings.lisp") +(defparameter include-dir-path + (namestring + (merge-pathnames + #p"doc/includes/" + (asdf:component-pathname (asdf:find-system 'elephant-tests))))) + +(defparameter docstrings-path + (namestring + (merge-pathnames + #p"doc/docstrings.lisp" + (asdf:component-pathname (asdf:find-system 'elephant-tests))))) + +(sb-posix:chdir include-dir-path) +(load docstrings-path) (defun make-docs () -;; (when (check-complete) (when t (elephant:open-store elephant-tests::*testbdb-spec*) (make-instance 'elephant::persistent-collection) --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*- @node Data Store API Reference @comment node-name, next, previous, up @chapter Data Store API Reference @cindex Data Store API Reference @cindex Data Store @cindex API Reference These are the functions that need to be overridden to implement support for a data store backend. Included are the exported elephant functions that need methods defined on them. Some functions here are utilities from the main elephant package that support store implementations. Migration, class indices and query interfaces are implemented on top of the store API and require no special support by implementors. @menu * Registration:: Register the backend to parse controller specifications * Store Controllers:: Subclassing the store controller. * Slot access:: Support for metaprotocol slot access. * Collections:: BTrees and indices. * Cursors:: Traversing BTrees. * Transactions:: Transaction implementation. * Multithreading:: Multithreading considerations. * Serialization:: Facilities for serializing objects. * C Utilities:: Writing primitive C types. * Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries @end menu @node Registration @comment node-name, next, previous, up @section Registration @cindex Registration @include includes/fun-elephant-register-backend-con-init.texinfo @include includes/fun-elephant-lookup-backend-con-init.texinfo @node Store Controllers @comment node-name, next, previous, up @section Store Controllers @cindex Store Controllers Subclass store-controller and implement store and close controller which are called by open-store and close-store respectively. @include includes/fun-elephant-store-controller.texinfo @include includes/fun-elephant-backend-open-controller.texinfo @include includes/fun-elephant-backend-close-controller.texinfo The slots for these accessors must be initialized. @include includes/fun-elephant-backend-database-version.texinfo @include includes/fun-elephant-backend-controller-serialize.texinfo @include includes/fun-elephant-backend-controller-deserialize.texinfo @include includes/fun-elephant-backend-root.texinfo @include includes/fun-elephant-backend-class-root.texinfo These functions are important utilities for implementing store-controllers. @include includes/fun-elephant-backend-oid.texinfo @include includes/fun-elephant-backend-get-con.texinfo @include includes/fun-elephant-backend-next-oid.texinfo @include includes/fun-elephant-backend-connection-is-indeed-open.texinfo @node Slot Access @comment node-name, next, previous, up @section Slot Access @cindex Slot Access These functions are called by the metaclass protocol to support operations on persistent class slots. @include includes/fun-elephant-backend-persistent-slot-writer.texinfo @include includes/fun-elephant-backend-persistent-slot-reader.texinfo @include includes/fun-elephant-backend-persistent-slot-boundp.texinfo @include includes/fun-elephant-backend-persistent-slot-makunbound.texinfo @node Collections @comment node-name, next, previous, up @section Collections @cindex Collections @c #:btree #:btree-index #:indexed-btree @c #:build-indexed-btree #:build-btree #:existsp @c #:map-indices @node Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors @c #:cursor @c #:cursor-btree @c #:cursor-oid @c #:cursor-initialized-p @node Transactions @comment node-name, next, previous, up @section Transactions @cindex Transactions @c #:*current-transaction* @c #:make-transaction-record @c #:transaction-store @c #:transaction-object @c #:execute-transaction @c #:controller-start-transaction @c #:controller-commit-transaction @c #:controller-abort-transaction @node Multithreading @comment node-name, next, previous, up @section Multithreading @cindex Multithreading @node Serialization @comment node-name, next, previous, up @section Serialization @cindex Serialization @c #:deserialize #:serialize @c #:serialize-symbol-complete @c #:deserialize-from-base64-string @c #:serialize-to-base64-string @node Memory utilities @comment node-name, next, previous, up @section Memory utilities @cindex Memory utilities @node Foreign libraries @comment node-name, next, previous, up @section Foreign libraries @cindex Foreign libraries --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 1.1 Debugger entered--Lisp error: (void-variable Design) eval(Design) eval-last-sexp-1(nil) eval-last-sexp(nil) call-interactively(eval-last-sexp) --- /project/elephant/cvsroot/elephant/doc/lisp-data-store.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/lisp-data-store.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*- @node Lisp Data Store @comment node-name, next, previous, up @chapter Lisp Data Store @cindex Lisp Data Store @cindex Data Store @cindex API Reference --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*- @node Usage Scenarios @comment node-name, next, previous, up @chapter Usage Scenarios @cindex Usage Scenarios Sorry, haven't written this section yet. Simple file replacement and indexing - Keep track of ordinary objects, ignore metaprotocol Persist system objects - Intermingle persistent objects and regular objects - Look up objects using class indices Full database system - storage, rich data models, references, queries, etc Multithreaded web applications - DB + multithreading Object-oriented data storage, large graph traversals --- /project/elephant/cvsroot/elephant/doc/style.css 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/style.css 2007/03/24 13:55:15 1.1 --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 NONE +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 1.1 @c -*-texinfo-*- @node User Guide @comment node-name, next, previous, up @chapter User Guide @cindex User Guide @menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. * Reachability:: Determining liveness in a store. * Persistent objects:: All the dirt on persistent objects. * Class indices:: In-depth discussion about indexing persistent indices. * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. * Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Repository Migration and Upgrade:: How to move objects from one repository to another. * Garbage collection:: How to recover storage and OIDs in long-lived repositories. * Performance tuning:: How to get the most from Elephant. @end menu @node Persistent objects @comment node-name, next, previous, up @section Persistent Objects Finally, if you for some reason make an instance with a specified OID which already exists in the database, @code{initargs} take precedence over values in the database, which take precedences over @code{initforms}. Also currently there is a bug where @code{initforms} are always evaluated, so beware. (What is the current model here?) @node The Store Controller @comment node-name, next, previous, up @section The Store Controller What is @code{open-store} doing? It creates a @code{store-controller} object, and sets the special @code{*store-controller*} to point to it. The store controller holds the handles to the database environment and tables, and some other bookkeeping. If for some reason you need to run recovery on the database (see sleepycat docs) you can specify that with the @code{:recover} and @code{:recover-fatal} keys. To create one by hand one can do, @lisp * (setq *store-controller* (make-instance 'store-controller :path "testdb")) => #<STORE-CONTROLLER @{49252F75@}> * (open-controller *store-controller*) => #<STORE-CONTROLLER @{49252F75@}> @end lisp but @lisp * (open-store "testdb")) @end lisp is the preferred mechanism. This opens the environment and database. The @code{persistent-*} objects reference the @code{*store-controller*} special. (This is in part because slot accessors can't take additional arguments.) If for some reason you want to operate on 2 store controllers, you'll have to do that by flipping the @code{*store-controller*} special. @code{close-store} closes the store controller. Alternatively @code{close-controller} can be called on a handle. Don't forget to do this or else you may need to run recovery later. There is a @code{with-open-controller} macro. Opening and closing a controller is very expensive. @node{Using BTrees} @comment node-name, next, previous, up @section Using BTrees The btree class are to hash-tables as persistent-objects are to ordinary objects. BTrees have a hash-table-like interface, but store their keys and values directly as rows in a Sleepycat BTree. Btrees may be persisted simply by their OID. Hence they have all the nice properties of persistent objects: identity, fast serialization / deserialization, no merge conflicts..... (defvar *friends-birthdays* (make-btree)) => *FRIENDS-BIRTHDAYS* (add-to-root "friends-birthdays" *friends-birthdays*) => #<BTREE {4951CF6D}> (setf (get-value "Andrew" *friends-birthdays*) (encode-universal-time 0 0 0 22 12 1976)) => 2429071200 (setf (get-value "Ben" *friends-birthdays*) (encode-universal-time 0 0 0 14 4 1976)) => 2407298400 (get-value "Andrew" *friends-birthdays*) => 2429071200 => T (decode-universal-time *) => 0 0 0 22 12 1976 2 NIL 6 Because of serialization semantics, BTrees hash on a value, not identity. This is probably ok for strings, numbers, and persistent things, but may be strange for other values. @node Repository Migration @comment node-name, next, previous, up @section Repository Migration This version of Elephant supports migration between store controllers of any backend type. The tests @code{migrate1} - @code{migrate5} are demonstrations of this capability. There is a single generic function @code{migrate} that is used to copy different object types to a target repository. It is assumed that typically migrate will be called on two repositories and all live objects (those reachable in the root or class-root) will be copied to the target repository via recursive calls to migrate for specific objects. When persistent instances are copied, their internal pointer will be updated to point to the new repository so after migration the lisp image should be merely updated to refer to the target repository in the *store-controller* variable or whatever variable the application is using to store the primary controller instance. There are some limitations to the current migration implementation: @enumerate @item Migrate currently will not handle circular list objects @item Migrate does not support arrays with nested persistent objects @item Indexed classes only have their class index copied if you use the top level migration. Objects will be copied without slot data if you try to migrate an object outside of a store-to-store migration due to the class object belonging to one store or another @item Migrate assumes that after migration, indexed classes belong to the target store. @item In general, migration is a one-time activity and afterwards (or after a validation test) the source store should be closed. Any failures in migration should then be easy to catch. @item Each call to migration will be good about keeping track of already copied objects to avoid duplication. Duplication shouldn't screw up the semantics, just add storage overhead but is to be avoided. However this information is not saved between calls and there's no other way to do comparisons between objects across stores (different oid namespaces) so user beware of the pitfalls of partial migrations... @item Migrate keeps a memory-resident hash of all objects; this means you cannot currently migrate a store that has more data than your main memory. (This could be fixed by keeping the oid table in the target store and deleting it on completion) @item Migration does not maintain OID equivalence so any datastructures which index into those will have to have a way to reconstruct themselves (better to keep the object references themselves rather than oids in general) but they can overload the migrate method to accomplish this cleanly @end enumerate Users can customize migration if they create unusual datastructures that are not automatically supported by the existing @code{migrate} methods. For example, a datastructure that stores only object OIDs instead of serialized object references will need to overload migrate to ensure that all referenced objects are in fact copied (otherwise the OIDs will just be treated as fixnums potentially leaving dangling references. To customize migration overload a version of migrate to specialize on your specific persistent class type. @lisp (defmethod migrate ((dst store-controller) (src my-class))) @end lisp In the body of this method you can call @code{(call-next-method)} to get a destination repository object with all the slots copied over to the target repository which you can then overwrite. To avoid the default persistent slot copying, bind the dynamic variable @code{*inhibit-slot-writes*} in your user method using @code(with-inhibited-slot-copy () ...)} a convenience macro. @node Threading @comment node-name, next, previous, up @section Threading Sleepycat plays well with threads and processes. The store controller is thread-safe by default, that is, can be shared amongst threads. This is set by the @code{:thread} key. Transactions may not be shared amongst threads except serially. One thing which is NOT thread and process safe is recovery, which should be run when no one is else is talking to the database environment. Consult the Sleepycat docs for more information. Elephant uses some specials to hold parameters and buffers. If you're using a natively threaded lisp, you can initialize these specials to thread-local storage by using the @code{run-elephant-thread} function, assuming your lisp creates thread-local storage for let-bound specials. (This functionality is currently broken) [72 lines skipped]

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv6422/doc Modified Files: Makefile copying.texinfo elephant.texinfo installation.texinfo intro.texinfo make-ref.lisp package-elephant.texinfo reference.texinfo tutorial.texinfo Log Message: Cleanup indexing tests so we always have a clean slate --- /project/elephant/cvsroot/elephant/doc/Makefile 2005/11/23 17:51:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 12:16:02 1.3 @@ -5,4 +5,4 @@ makeinfo -v --html --force elephant.texinfo includes-stuff: - cd includes; lisp < ../make-ref.lisp + cd includes; sbcl < ../make-ref.lisp --- /project/elephant/cvsroot/elephant/doc/copying.texinfo 2006/03/02 14:44:49 1.2 +++ /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/03/24 12:16:02 1.3 @@ -15,15 +15,13 @@ as governed by the terms of the Lisp Lesser GNU Public License @uref{http://opensource.franz.com/preamble.html}, also known as the LLGPL. - - Copyrights include: Copyright (c) 2004 by Andrew Blumberg and Ben Lee -Copyright (c) 2006 by Ian Eslick +Copyright (c) 2006-2007 by Ian Eslick -Copyright (c) 2005,2006 by Robert L. Read +Copyright (c) 2005-2007 by Robert L. Read Portions of this program (namely the C unicode string --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2006/03/02 14:44:49 1.4 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/24 12:16:02 1.5 @@ -6,7 +6,7 @@ @copying Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. -Copyright @copyright{} 2006 Robert L. Read. +Copyright @copyright{} 2006-2007 Robert L. Read and Ian Eslick @quotation Permission is granted to copy, distribute and/or modify this document @@ -19,8 +19,8 @@ @titlepage @title Elephant User Manual -@subtitle Elephant version 0.6 -@author Ben Lee +@subtitle Elephant version 0.6.1 +@author Ben Lee and Ian Eslick @c The following two commands @c start the copyright page. @@ -29,24 +29,35 @@ @insertcopying @end titlepage -@c So the toc is printed at the start. -@contents - @ifnottex @node Top @comment node-name, next, previous, up -@top Elephant +@top Copyright @insertcopying @end ifnottex @menu -* Introduction:: Introducing Elephant! -* Tutorial:: A leisurely walk-through. -* Reference:: API documentation. -* Installation:: Installation and test-suite procedures and issues -* Design Notes:: Internals. -* Copying:: Your rights and freedoms. +* Table of Contents:: +@end menu + +@chapheading Chapters + +@menu +* Introduction:: Introduction to the Elephant Persistent Object System. +* Tutorial:: A basic ``getting started'' tutorial. +* Installation:: Installation and test-suite procedures. +* User Guide:: In depth discussion of all Elephant facilities and features. +* Usage scenarios:: Design scenarios for Elephant applications. +* User API Reference:: Function and class documentation of the user API. +* Elephant Design:: An overview of elephant's internal architecture. +* Data Store API Reference:: Function level documentation for data store implementors. +* Copying:: Your rights and freedoms. +@end menu + +@chapheading Appendices + +@menu * Concept Index:: * Object Index:: * Function / Macro Index:: @@ -54,11 +65,18 @@ * Colophon:: @end menu +@node Table of Contents +@comment node-name, next, previous, up +@contents + @include intro.texinfo @include tutorial.texinfo -@include reference.texinfo -@include notes.texinfo @include installation.texinfo +@include user-guide.texinfo +@include scenarios.texinfo +@include reference.texinfo +@include elephant-design.texinfo +@include data-store-reference.texinfo @include copying.texinfo @node Concept Index --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2006/05/15 13:02:26 1.3 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/24 12:16:02 1.4 @@ -8,10 +8,11 @@ @menu * Installation Basics:: Basic installation * Test-Suites:: Running the test suites -* SQL-Introduction:: The design and status of the SQL back-end extention. -* Extension Status:: The current status of the SQL back-end extention. -* Multi-repository Operation:: Specifying repositories -* Setting up PostGres:: An example +* Berkeley DB Introduction:: The Berkeley DB backend +* SQL Data Store:: The design and status of the SQL back-end extension. +* Lisp Data Store:: A native lisp-based repository. +* Multi-repository Operation:: Specifying repositories. +* Setting up PostGres:: An example. @end menu @node Installation Basics @@ -148,11 +149,14 @@ If you get errors, you may wish to report it the @code{ elephant-devel at common-lisp.net} email list. +@node Berkeley DB Repository +@comment node-name, next, previous, up +@section Berkeley DB Repository -@node SQL-Introduction +@node SQL Repository @comment node-name, next, previous, up -@section SQL-Introduction +@section SQL Repository Although originally designed as an interface to the BerkeleyDB system, the original Elephant system has been experimenetally extended to --- /project/elephant/cvsroot/elephant/doc/intro.texinfo 2006/04/26 17:53:43 1.4 +++ /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/03/24 12:16:02 1.5 @@ -5,66 +5,96 @@ @chapter Introduction @cindex Introduction -Elephant is a persistent object database for Common Lisp that -supports storing CLOS objects and most lisp primitives. -It supports persistent collections via a BTree interface. - -Elephant was originally developed as an interface layer on top -of the Sleepycat / Berkeley DB library, a widely-distributed -embedded database. Many unix systems have it installed by default. -Berkeley DB is ACID compliant, transactional, process and -thread safe, and fast relative to relational databases. Recently, -Elephant was extended to provide support for relational database backends. -It has been tested with Postgres and SQLite 3. It supports, with some -care, simultaneous multi-repository operation and enables convenient -migration of data between repositories. +Elephant is a persistent object protocol and database for Common +Lisp. The persistent protocol component of elephant overrides class +creation and standard slot accesses using the Meta-Object Protocol +(MOP) to render slot values persistent. Database functionality +includes the ability to persistently index and retrieve ordered sets +of class instances and ordinary lisp values. Elephant has an +extensive test suite and the core functionality is becoming quite +mature. + +The Elephant code base is available under the LLGPL license. Data +stores each come with their own, separate license and you will have to +evaluate the implications of using them yourself. + +@section History + +Elephant was originally envisioned as a lightweight interface layer on +top of the Berkeley DB library, a widely-distributed embedded database +that many unix systems have installed by default. Berkeley DB is ACID +compliant, transactional, process and thread safe, and fast relative +to relational databases. + +Elephant has been extended to provide support for multiple backends, +specifically a relational database backend based on CL-SQL which has +been tested with Postgres and SQLite 3. It supports, with some care, +multi-repository operation and enables convenient migration of data +between repositories. The support for relational backends and migration to the LLGPL was to allow for broader use of Elephant in both not-for-profit and commercial -settings. +settings. Several additional backends are planned for future releases +including a native Lisp implementation released under the LLGPL. -Elephant goals: +Elephant's current development focus is to enhance the feature set +including a native lisp backend, a simple query language, and flexible +persistence models that selectively break one or more of the ACID +constraints to improve performance. + +@section Elephant Goals @itemize -@item Transparency: most Lisp values are easy to persist without -signifcant effort or special syntax. Talk to the DB entirely from Lisp; -not requirement for domain-specific languages (such as SQL) to access persistent -resources. Enable interactive control of the database with no external -server dependencies. - -@item Safety: ACID, transactions. Concurrent with good multi-user and -multi-threaded semantics, isolation, locking and deadlock detection. -(Deadlock detection does require an external process to be launched) +@item @strong{Transparency:} most Lisp values are easy to persist without +significant effort or special syntax. You can interact with the DB +entirely from Lisp. There is no requirement to use domain-specific +languages, such as SQL, to access persistent resources. Elephant +loads via ASDF and requires no external server (except for some SQL +backends like Postgres). -@item Simplicity: a small library with few surprises for the +@item @strong{Simplicity:} a small library with few surprises for the programmer. Lisp and Berkeley DB together are an excellent substrate; -Elephant tries to leverage their features as much as possible. -Support for multiple backends should be load-time options and mostly +Elephant tries to leverage their features as much as possible. +Support for additional backends are load-time options and more or less transparent to the user. -@item Performance: leverage Sleepycat performance and +@item @strong{Safety:} ACID, transactions. Concurrent with good multi-user (BDB) and +multi-threaded semantics (BDB/SQL), isolation, locking and deadlock +detection. (Deadlock detection does require an external process to be +launched for Berkeley DB) + +@item @strong{Performance:} leverage Berkeley DB performance and/or Relational database reliability. In addition to fast concurrent / transactional modes, -elephant will (eventually) offer an accellerated single-user as -well as in-memory modes that should be comparable to prevalence -style solutions, but leverage a common interface. +elephant will (eventually) offer an accelerated single-user as well as +pure in-memory mode that should be comparable to prevalence style +solutions, but employ a common programmer interface. -@item Historical continuity: Elephant does not try to innovate +@item @strong{Historical continuity:} Elephant does not try to innovate significantly over prior Lisp persistent object stores such as AllegroStore (also based on Berkeley DB), the new AllegroCache, the Symbolics system Statice and PLOB. Anyone familiar with those systems will recognize the Elephant interface. -@item License Flexibility: Elephant is released under the LLGPL. +@item @strong{License Flexibility:} Elephant is released under the LLGPL. Because it supports multiple implementation of the backend, one can choose a backend with licensing and other features appropriate to your needs. @end itemize -Join the Elephant mailing lists to ask your questions and -receive updates. Pointers can be found on the Elephant website at +@section More Information + +Join the Elephant mailing lists to ask your questions and receive +updates. You can also review archives for past discussions and +questions. Pointers can be found on the Elephant website at @uref{http://www.common-lisp.net/project/elephant}. -Installation documents can be found in the file @file{INSTALL}. -Opportunities to contribute can be found in the file @file{TODO}. +Installation instructions can be found in the @ref{Installation} +section. Bugs can be reported via the Elephant Trac system at + +@uref{http://trac.common-lisp.net/elephant/}. + +This also serves as a good starting point for finding out what new +features or capabilities you can contribute to Elephant. The Trac +system also contains a wiki with design discussions and a FAQ. --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2005/11/23 17:51:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 12:16:02 1.3 @@ -1,10 +1,15 @@ (require 'asdf) -(require 'elephant) -(load "../docstrings.lisp") +(asdf:operate 'asdf:load-op 'elephant-tests) +(sb-posix:chdir "/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/") +(load "/Users/eslick/Work/fsrc/elephant-cvs/doc/docstrings.lisp") (defun make-docs () ;; (when (check-complete) (when t - (sb-texinfo:generate-includes #p"includes" (find-package :ele)))) + (elephant:open-store elephant-tests::*testbdb-spec*) + (make-instance 'elephant::persistent-collection) + (make-instance 'elephant::secondary-cursor) + (make-instance 'elephant::indexed-btree) + (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" (find-package :elephant) (find-package :elephant-backend) (find-package 'elephant-memutil) (find-package 'elephant-system)))) (make-docs) --- /project/elephant/cvsroot/elephant/doc/package-elephant.texinfo 2004/09/19 17:44:42 1.1 +++ /project/elephant/cvsroot/elephant/doc/package-elephant.texinfo 2007/03/24 12:16:02 1.2 @@ -1,6 +1,5 @@ @anchor{Package elephant} @defvr {Package} elephant -Elephant: an object-oriented database for Common Lisp. -Uses the @code{sleepycat} package to talk to Berkeley @code{db} / -Sleepycat. +Elephant: an object-oriented database for Common Lisp with + multiple backends for Berkeley @code{db}, @code{sql} and others. @end defvr --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2006/05/15 13:02:26 1.5 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/24 12:16:02 1.6 @@ -1,52 +1,42 @@ @c -*-texinfo-*- -@node Reference +@node User API Reference @comment node-name, next, previous, up -@chapter Reference -@cindex Reference +@chapter User API Reference +@cindex User API Reference @cindex API Reference @menu -* Controller:: The connection to Sleepycat. -* Transactions:: Transactions. -* Persistent Objects:: CLOS persistence. -* Persistent Slot Indexing:: Convenient indexing. +* Store Controllers:: Connecting to a data store. +* Persistent Objects:: Creating and using persistent objects. +* Persistent Object Indexing:: Convenient indexing. +* Query Interfaces:: Finding instances. * Collections:: BTrees and indices. * Cursors:: Traversing BTrees. -* Sleepycat:: Some functions from the low-level Sleepycat interface. +* Transactions:: Transactions. +* Multithreading:: Multithreading. +* Migration and Upgrading:: Migration and upgrading. @end menu -@node Controller +@node Store Controllers @comment node-name, next, previous, up -@section Controller -@cindex Controller +@section Store Controllers +@cindex Store Controllers + +Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller. + +@include includes/class-elephant-store-controller.texinfo +@include includes/var-elephant-star-store-controller-star.texinfo @include includes/fun-elephant-open-store.texinfo @include includes/fun-elephant-close-store.texinfo @include includes/macro-elephant-with-open-store.texinfo -@include includes/fun-elephant-add-to-root.texinfo @include includes/fun-elephant-get-from-root.texinfo -@include includes/fun-elephant-run-elephant-thread.texinfo - -@include includes/var-elephant-star-store-controller-star.texinfo -@include includes/class-elephant-store-controller.texinfo -@include includes/fun-elephant-open-controller.texinfo -@include includes/fun-elephant-close-controller.texinfo -@include includes/macro-elephant-with-open-controller.texinfo - -@node Transactions -@comment node-name, next, previous, up -@section Transactions -@cindex Transactions - -@include includes/macro-elephant-with-transaction.texinfo - -@include includes/var-elephant-star-auto-commit-star.texinfo -@include includes/var-elephant-star-current-transaction-star.texinfo -@include includes/fun-elephant-start-ele-transaction.texinfo -@include includes/fun-elephant-commit-transaction.texinfo -@include includes/fun-elephant-abort-transaction.texinfo +@include includes/fun-elephant-add-to-root.texinfo +@include includes/fun-elephant-remove-from-root.texinfo +@include includes/fun-elephant-root-existsp.texinfo +@include includes/fun-elephant-map-root.texinfo @node Persistent Objects @comment node-name, next, previous, up @@ -57,10 +47,14 @@ @include includes/class-elephant-persistent.texinfo @include includes/class-elephant-persistent-object.texinfo -@node Persistent Slot Indexing +@include includes/macro-elephant-defpclass.texinfo + +@include includes/fun-elephant-drop-pobject.texinfo + +@node Persistent Object Indexing @comment node-name, next, previous, up -@section Persistent Slot Indexing -@cindex Persistent Slot Indexing +@section Persistent Object Indexing +@cindex Persistent Object Indexing @include includes/fun-get-instances-by-class.texinfo @include includes/fun-get-instance-by-value.texinfo @@ -74,6 +68,11 @@ @include includes/fun-add-class-derived-index.texinfo @include includes/fun-remove-class-derived-index.texinfo +@node Query Interfaces +@comment node-name, next, previous, up +@section Query Interfaces +@cindex Query Interfaces + @node Collections @comment node-name, next, previous, up @section Collections @@ -133,24 +132,25 @@ @include includes/fun-elephant-cursor-set-range.texinfo @include includes/fun-elephant-cursor-set.texinfo -@node Sleepycat +@node Transactions @comment node-name, next, previous, up -@section Sleepycat -@cindex Sleepycat +@section Transactions +@cindex Transactions -@include includes/macro-elephant-with-lock.texinfo -@include includes/fun-elephant-db-env-get-flags.texinfo -@include includes/fun-elephant-db-env-get-lock-detect.texinfo -@include includes/fun-elephant-db-env-get-timeout.texinfo -@include includes/fun-elephant-db-env-lock-get.texinfo -@include includes/fun-elephant-db-env-lock-id-free.texinfo -@include includes/fun-elephant-db-env-lock-id.texinfo -@include includes/fun-elephant-db-env-lock-put.texinfo -@include includes/fun-elephant-db-env-set-flags.texinfo -@include includes/fun-elephant-db-env-set-lock-detect.texinfo -@include includes/fun-elephant-db-env-set-timeout.texinfo -@include includes/fun-elephant-db-transaction-abort.texinfo -@include includes/fun-elephant-db-transaction-begin.texinfo -@include includes/fun-elephant-db-transaction-commit.texinfo -@include includes/fun-elephant-db-transaction-id.texinfo +@include includes/macro-elephant-with-transaction.texinfo + +@include includes/var-elephant-star-auto-commit-star.texinfo +@include includes/var-elephant-star-current-transaction-star.texinfo +@include includes/fun-elephant-start-ele-transaction.texinfo +@include includes/fun-elephant-commit-transaction.texinfo +@include includes/fun-elephant-abort-transaction.texinfo +@node Multithreading +@comment node-name, next, previous, up +@section Multithreading +@cindex Multithreading + +@node Migration and Upgrading +@comment node-name, next, previous, up +@section Migration and Upgrading +@cindex Migration and Upgrading --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2006/04/26 17:53:43 1.6 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/24 12:16:02 1.7 @@ -6,306 +6,348 @@ @cindex Tutorial @menu -* Preliminaries:: Some general remarks. -* Getting Started:: Accessing a store. -* Running the Tests:: Gaining confidence. -* The Root:: Staying alive. -* Serialization:: Lisp -> (char *). -* Persistent Classes:: CLOS the Elephant way. +* Overview:: Overview of elphant's features +* Getting Started:: Opening and accessing a store. +* The Store Root:: Accessing persistent data. +* Serialization:: Storage semantics for lisp values. +* Persistent Classes:: Persistent semantics for objects. * Rules about Persistent Classes:: What you need to know. -* Using Transactions:: Using ACID. -* Using BTrees:: Storing lots of things. -* Using Cursors:: Tranversing BTrees. -* Secondary Indices:: By any other name... -* Class Indices:: Speed and Convenience. -* The Store Controller:: Behind the curtain. -* Repository Migration:: How to move objects from one repository to another -* Threading:: Playing nice with others. -* Performance Tips:: Bogoflops for your buck. +* Persistent collections:: Keep track of collections of objects. +* Class Indices:: Simple way to keep track of instances. +* Using Transactions:: Enabling ACID database properties. @end menu -@node Preliminaries +@node Overview @comment node-name, next, previous, up -@section Preliminaries +@section Overview -Elephant is a Common Lisp OODB. It provides a partial solution to the -problem of making Lisp data persistent. It does this through two mechanisms: -a simple API for storing and retrieving lisp values from a persistent store, -and the ability to make CLOS class slot values be persistent. - -When someone says "database," most people think of SQL Relation Data Base -Management Systems (e.g. Oracle, Postgresql, MySql). Elephant can use either -RDBMSs or Berkeley DB (Sleepycat) as a backend repository, but relies on -LISP as its data manipulation system. Unlike systems such as Hibernate -for Java, the user does not need to construct or worry about a mapping -from the object space into the database. Elephant is designed to be a -simple and convenient tool for the programmer. - -Elephant supports easy migration of data between different repositories and -different backends, allowing the user to choose which repository backend they -will use at a particular point in time. - -Berkeley DB/Sleepycat is a database library that was the initial inspiration for Elephant's -design and is well-matched to Elephant's data model. BDB is implemented as a C library, -not a client/server model, so access can be very fast. Berkeley DB is also quite mature, -robust and has many features, such as transactions and replication. While we hope -that you won't need to understand a specific backend to use Elephant, reading the -docs will certainly help you when things go wrong. For the Sleepycat backend, -they can be found at @uref{http://www.sleepycat.com}. +Elephant is a Persistence Metaprotocol and Database for Common Lisp. +It provides the ability for users to define and interact with +persistent objects and to transparently store ordinary lisp values. +Persistent objects are CLOS instances that overload the ordinary slot +access semantics so that every write to a slot is passed through and +written to disk. Non-persistent lisp objects and values can be +written to slots and will be automatically persisted. In addition, +Elephant provides a persistent index which maintains an ordered +collection of lisp values or persistent object references. + +When someone says "database," most people think of SQL Relational Data +Base Management Systems (e.g. Oracle, Postgresql, MySql). Those +systems store data in statically typed tables with unique shared +values to connect rows in separate tables. Objects can be mapped into +these tables in an object-relational mapping that assigns objects to +rows and slot values to columns in a row's table. If a slot +references another type of object, a unique ID can be used to +reference that object's table. CL-SQL, for example, provides +facilities for this kind of object-relational mapping and there are +many systems for other languages that do the same (i.e. Hibernate for +Java). + +While Elephant can use either RDBMSs or Berkeley DB as a data store, +the model it supports is that of objects stored in persistent indices. +Unlike systems such as Hibernate for Java, the user does not need to +construct or worry about a mapping from the object space into the +database. Elephant relies on LISP rather than SQL for its data +manipulation language. Elephant is designed to be a simple and +convenient tool for the programmer. + +Elephant consists of a small universe of basic concepts: + +@itemize +@item @strong{Store controller:} the interface between lisp and a data store. +Most operations require or accept a store controller, or a default +store controller stored in @code{*store-controller*} to function. +@item @strong{BTrees:} Elephant provides a persistent key-value +abstraction based on the BTree data structure. Values can be written +to or read from a BTree and are stored in a sorted order. +@item @strong{Values:} most lisp values, including standard objects, arrays, etc +can be used as either key or value in a persistent BTree. +@item @strong{Persistent objects:} An object where most slot values are stored in +the data store and are written to or retrieved from disk on slot +accesses. Any value that can be written to an index can be written to +a persistent slot. +@item @strong{Transactions:} a dynamic context for executing operations on persistent +objects or BTrees that ensures that a set of changes is made atomically. +@item @strong{BTree indices:} A BTree index is a BTree that stores +an alternative ordering of the elements in a reference BTree. +@end itemize + +There are a set of more advanced concepts you will learn about later, +but these basic concepts will serve to acquaint you with Elephant. + +If you do not already have Elephant installed and building correctly, +read the @ref{Installation} section of this manual and then move on to +@ref{Getting Started}. -Elephant can also use RDBMS backends via the excellent CL-SQL package. -It has been tested with Postgres and SQLite3, and can probably easily work with others. - -@node Running the Tests +@node Getting Started @comment node-name, next, previous, up -@section Running the Tests - -There are three files in the directory @code{tests} that make running -the automated tests particularly easy. @code{BerkeleyDB-tests.lisp} is -for running against the BerkeleyDB backend, and @code{SQLDB-tests.lisp} is -for running agains the CL-SQL backend. @code{MigrationTests.lisp} is -for testing data migration functions, and can be used with either or both backends. - -The normal way to execute the tests, following the instruction in the file -@code{INSTALL}, is to open a listener and execute the lines found in -one of these files, such as: -@lisp -(asdf:operate 'asdf:load-op :elephant-tests) +@section Getting Started -(in-package "ELEPHANT-TESTS") +The first step in using elephant is to open a store controller. A +store controller is an object that coordinates lisp program access +to the chosen data store. -(setf *default-spec* *testbdb-spec*) +To obtain a store controller, you call @code{open-store} with a store +specification. A store specification is a list containing a backend +specifier (@code{:BDB} or @code{:CLSQL}) and a backend-specific +reference. -(do-backend-tests) -@end lisp +For :BDB, the second element is a string or pathname that references a +local directory for the database files. This directory must be +created prior to calling open-store. -The SQL test file differs only in using a different ``controller spec'': -@lisp(setf *default-spec* *testpg-spec*) -@end lisp -These default parameters are set in @file{tests/elephant-tests.lisp}, -they will looks something like this in a default distribution: -@lisp -(:BDB "/home/read/projects/sql-back-end/elephant/tests/testdb/") -@end lisp -and for postgres: @lisp -(:CLSQL (:POSTGRESQL "localhost.localdomain" "test" "postgres" "")) +(open-store '(:BDB ``/users/me/db/my-db/'')) @end lisp +For :CLSQL the second argument is another list consisting of a +specific SQL database and the name of a database file or connection +record to the SQL server. Examples are: -@node Getting Started -@comment node-name, next, previous, up -@section Getting Started - -In order to use Elephant, you have to have an open store controller. -To obtain an open store controller you call @code{open-store} - -The chapter ``SQL back-end'' has information about setting up a -SQL based backend; this tutorial will assume that you are using -Berkeley-DB as a backend. - -Make a directory to put your database store in. (This is called the -environment in Sleepycat terminology.) That's all you need to set up -your store! We'll assume in this tutorial you created a folder -@code{testdb} in the current directory. - -It is strongly recommended that you run the automated tests @xref{Running the Tests} that -come with Elephant before you begin this tutorial; this takes less -than five minutes and if will give you both confidence and clarity -and your continued work. Since the default distribution comes -with a directory structure set up, this is actually the easiest -way to get started with Elephant before beginning this tutorial. -If the tests fail for you, the Elephant developers will help you -solve the problem, but will want to know the outcome of the tests -as a starting point. - -If you have run the tests successfully, you can just do: @lisp -(open-store *default-spec*) +(open-store '(:CLSQL (:SQLITE "/users/me/db/sqlite.db"))) +(open-store '(:CLSQL (:POSTGRESQL "localhost.localdomain" "mydb" "myuser" "")))) @end lisp -But if not you might have to set up your own controller specifier like this: + +We use Berkeley DB as our example backend. To open a BDB +store-controller we can do the following: + @lisp (asdf:operate 'asdf:load-op :elephant) -(use-package "ELE") -(setf *testbdb-spec* -'(:BDB "/home/read/projects/sql-back-end/elephant/tests/testdb/")) -(open-store *testbdb-spec*) -@end lisp +(use-package :elephant) +(setf *test-db-spec* + '(:BDB "/home/me/db/testdb/")) +(open-store *test-db-spec*) +@end lisp + +We do not need to store the reference to the store just now as it is +automatically assigned to the variable, @code{*store-controller*}. +For a deeper discussion of store controller management see the +@ref{User Guide}. + +When you're done with your session, release the store-controller's +resources by calling @code{close-store}. + +Also there is a convenience macro @code{with-open-store} that will +open and close the store, but opening the store is an expensive +operation so it is generally better to leave the store open until your +application no longer needs it. + +@node The Store Root +@comment node-name, next, previous, up +@section The Store Root + +What values live between lisp sessions is called @emph{liveness}. +Liveness in a store is determined by whether the value can be reached +from the root of the store. The root is a special BTree in which +other BTrees and lisp values can be stored. This BTree has a special +interface through the store controller. (There is a second root BTree +called the class root which will be discussed later.) -When you're done with your session, don't forget to +You can put something into the root object by @lisp -* (close-store) -=> NIL +(add-to-root "my key" "my value") +=> "my value" @end lisp -Also there is a convenience macro @code{with-open-store}. +and get things out via -@node The Root -@comment node-name, next, previous, up -@section The Root +@lisp +(get-from-root "my key") +=> "my value" +=> T +@end lisp -Liveness in a store is determined by reachability from the root -object. Technically, liveness also applies to indexed -classes, as described in @xref{Class Indices}, which live in a -separate class-root namespace. When garbage collection is -implemented, dead objects will be collected on gc's.) The root and -class-root objects are BTrees, effectively a table with sorted keys -and log(N) access time. @xref{Using BTrees}. +The second value indicates whether the key was found. This is +important if your key-value pair can have nil as a value. -You can put something into the root object by +You can perform other basic operations as well. @lisp -* (add-to-root "my key" "my value") +(root-existsp "my key") +=> T +(remove-from-root "my key") +=> T +(get-from-root "my key") +=> NIL => NIL @end lisp -and get things out via +To access all the objects in the root, the simplest way is to +simply call @code{map-root} with a function to apply to each +key-value pair. @lisp -* (get-from-root "my key") -=> "my value" -=> T -@end lisp +(map-root + (lambda (k v) + (format t "key: ~A value:~A~%" k v))) +@end lisp -The root object is available as +You can also access the root object directly. @lisp -* (controller-root *store-controller*) -=> #<BTREE @{492AE37D@}> +(controller-root *store-controller*) +=> #<DB-BDB::BDB-BTREE @ #x10e86042> @end lisp -It is an instance of a class "btree"; @xref{Using BTrees}. +It is an instance of a class "btree"; @pxref{Using BTrees}. @node Serialization @comment node-name, next, previous, up @section Serialization -What can you put into the store? An ever-growing list of things: -numbers (except for complexes, which will be easy to support), +What can you put into the store besides strings? Almost all lisp +values and objects can be stored: numbers (except for complexes), symbols, strings, nil, characters, pathnames, conses, hash-tables, -arrays, CLOS objects. Nested and circular things are allowed. You -can store basically anything except lambdas, closures, structures, -packages and streams. (These may eventually get supported too.) - -Unfortunately Berekely DB doesn't understand Lisp, so Lisp data needs -to be serialized to enter the database (converted to byte arrays), and -deserialized to be read. This introduces some caveats (not unique to -Elephant!): +arrays, CLOS objects and structs. Nested and circular things are +allowed. You can store basically anything except lambdas, closures, +class objects, packages and streams. (These may eventually get +supported too.) + +Elephant needs to use a representation of data that is independant of +a specific lisp or data store. Therefore all lisp values that are +stored must be @emph{serialized} into a canonical format. Because +Berkeley DB supports variable length binary buffers, Elephant uses a +binary serialization system. This process has some important +consequences that it is very important to understand: @enumerate -@item Lisp identity can't be preserved. Since this is a store which +@item @strong{Lisp identity can't be preserved}. Since this is a store which persists across invocations of Lisp, this probably doesn't even make -sense. +sense. However if you get an object from the index, store it to a +lisp variable, then get it again - they will not be eq: @lisp -* (setq foo (cons nil nil)) +(setq foo (cons nil nil)) => (NIL) -* (add-to-root "my key" foo) -=> NIL -* (add-to-root "my other key" foo) -=> NIL -* (eq (get-from-root "my key") +(add-to-root "my key" foo) +=> (NIL) +(add-to-root "my other key" foo) +=> (NIL) +(eq (get-from-root "my key") (get-from-root "my other key")) => NIL @end lisp -@item Mutated substructure does not persist +@item @strong{Nested aggregates are stored in one buffer}. +If you store an set of objects in a hash table you try to store a hash +table, all of those objects will get stored in one large binary buffer +with the hash keys. This is true for all other aggregates that can +store type T (cons, array, standard object, etc). + +@item @strong{Mutated substructure does not persist}. @lisp -* (setf (car foo) T) +(setf (car foo) T) => T -* (get-from-root "my key") +(get-from-root "my key") => (NIL) @end lisp This will affect all aggregate types: objects, conses, hash-tables, et cetera. (You can of course manually re-store the cons.) In this sense elephant does not automatically provide persistent collections. If you [871 lines skipped]

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv6422/src/elephant Modified Files: classes.lisp classindex.lisp Log Message: Cleanup indexing tests so we always have a clean slate --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/21 14:29:30 1.23 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/24 12:16:03 1.24 @@ -69,8 +69,9 @@ never (eq (class-of superclass) persistent-metaclass)))) (if (and (not (eq class persistent-object)) not-already-persistent) (apply #'call-next-method class slot-names - :direct-superclasses (cons persistent-object - direct-superclasses) args) +;; :direct-superclasses (cons persistent-object +;; direct-superclasses) args) + :direct-superclasses (append direct-superclasses (list persistent-object)) args) (call-next-method)))) (defmethod finalize-inheritance :around ((instance persistent-metaclass)) --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/03/23 16:08:10 1.31 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/03/24 12:16:03 1.32 @@ -232,34 +232,41 @@ (let ((class-idx (find-class-index class :sc sc :errorp errorp))) (if class-idx (progn - (wipe-class-indexing class class-idx :sc sc) + (wipe-class-indexing class :sc sc) (update-indexed-record class nil)) (when errorp (error "No class index exists in persistent store ~A" sc) (return-from disable-class-indexing nil))))) -(defmethod wipe-class-indexing ((class persistent-metaclass) class-idx &key (sc *store-controller*)) - ;; Clear out the current class record - (with-transaction (:store-controller sc) - (with-btree-cursor (cur class-idx) - (when (cursor-first cur) - (loop while (cursor-delete cur))))) - ;; Get the names of all indices & remove them - (let ((names nil)) - (map-indices (lambda (name secondary-index) - (declare (ignore secondary-index)) - (push name names)) - class-idx) - (dolist (name names) - (if (member name (class-slots class)) - (remove-class-slot-index class name) - (with-transaction (:store-controller sc) - (remove-index class-idx name))))) - ;; Drop the class instance index from the class root - (with-transaction (:store-controller sc) - (remove-kv (class-name class) (controller-class-root sc))) - (setf (%index-cache class) nil) - ) +(defmethod wipe-class-indexing ((class persistent-metaclass) &key (sc *store-controller*)) + (wipe-class-indexing (class-name class) :sc sc)) + +(defmethod wipe-class-indexing ((class-name symbol) &key (sc *store-controller*)) + (let ((cindex (get-value class-name (controller-class-root sc))) + (class (find-class class-name nil))) + (when cindex + ;; Delete all the values + (with-transaction (:store-controller sc) + (with-btree-cursor (cur cindex) + (loop while (cursor-next cur) do + (cursor-delete cur)))) + ;; Get the names of all indices & remove them + (let ((names nil)) + (map-indices (lambda (name secondary-index) + (declare (ignore secondary-index)) + (push name names)) + cindex) + (dolist (name names) + (when (member name (class-slots class)) + (if class + (remove-class-slot-index class name) + (with-transaction (:store-controller sc) + (remove-index cindex name)))))) + ;; Drop the class instance index from the class root + (with-transaction (:store-controller sc) + (remove-kv class-name (controller-class-root sc))) + (when class + (setf (%index-cache class) nil))))) (defmethod add-class-slot-index ((class symbol) slot-name &key (sc *store-controller*)) (add-class-slot-index (find-class class) slot-name :sc sc))

1 0

[elephant-cvs] CVS elephant/tests
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv6422/tests Modified Files: testindexing.lisp Log Message: Cleanup indexing tests so we always have a clean slate --- /project/elephant/cvsroot/elephant/tests/testindexing.lisp 2007/03/08 21:29:53 1.35 +++ /project/elephant/cvsroot/elephant/tests/testindexing.lisp 2007/03/24 12:16:03 1.36 @@ -42,18 +42,22 @@ (deftest indexing-basic-trivial (progn - (when (class-indexedp-by-name 'idx-one) - (disable-class-indexing 'idx-one :errorp nil) - (setf (find-class 'idx-one) nil)) + (defclass idx-one () + ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) + (:metaclass persistent-metaclass)) + (disable-class-indexing 'idx-one :errorp nil) + (setf (find-class 'idx-one) nil) (defclass idx-one () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) (:metaclass persistent-metaclass)) + (defmethod print-object ((obj idx-one) stream) (if (slot-boundp obj 'slot1) (format stream "slot1 = ~A~%" (slot1 obj)) (format stream "slot1 unbound~&") )) + (with-transaction (:store-controller *store-controller*) (setq inst1 (make-instance 'idx-one :slot1 101 :sc *store-controller*)) (setq inst1 (make-instance 'idx-one :slot1 101 :sc *store-controller*)) @@ -73,9 +77,12 @@ ;;(format t "Global vars:~%") ;;(format t "~%basic store: ~A ~A~%" *store-controller* (elephant::controller-spec *store-controller*)) - (when (class-indexedp-by-name 'idx-one) - (disable-class-indexing 'idx-one :errorp nil) - (setf (find-class 'idx-one nil) nil)) + (defclass idx-one () + ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) + (:metaclass persistent-metaclass)) + + (disable-class-indexing 'idx-one :errorp nil) + (setf (find-class 'idx-one nil) nil) (defclass idx-one () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) @@ -104,8 +111,12 @@ (deftest indexing-class-opt (progn - (when (class-indexedp-by-name 'idx-cslot) - (disable-class-indexing 'idx-cslot :errorp nil)) + (defclass idx-cslot () + ((slot1 :initarg :slot1 :initform 0 :accessor slot1)) + (:metaclass persistent-metaclass) + (:index t)) + + (disable-class-indexing 'idx-cslot :errorp nil) (setf (find-class 'idx-cslot) nil) (defclass idx-cslot () @@ -124,14 +135,6 @@ (progn ;; (format t "inherit store: ~A ~A~%" *store-controller* (controller-path *store-controller*)) - (when (class-indexedp-by-name 'idx-two ) - (disable-class-indexing 'idx-two :sc *store-controller* :errorp nil) - (setf (find-class 'idx-two) nil)) - - (when (class-indexedp-by-name 'idx-three ) - (disable-class-indexing 'idx-three :sc *store-controller* :errorp nil) - (setf (find-class 'idx-three) nil)) - (defclass idx-two () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) (slot2 :initarg :slot2 :initform 2 :accessor slot2 :index t) @@ -145,6 +148,24 @@ (slot4 :initarg :slot4 :initform 40 :accessor slot4 :index t)) (:metaclass persistent-metaclass)) + (disable-class-indexing 'idx-two :sc *store-controller* :errorp nil) + (setf (find-class 'idx-two) nil) + + (disable-class-indexing 'idx-three :sc *store-controller* :errorp nil) + (setf (find-class 'idx-three) nil) + + (defclass idx-two () + ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) + (slot2 :initarg :slot2 :initform 2 :accessor slot2 :index t) + (slot3 :initarg :slot3 :initform 3 :accessor slot3) + (slot4 :initarg :slot4 :initform 4 :accessor slot4 :transient t)) + (:metaclass persistent-metaclass)) + + (defclass idx-three (idx-two) + ((slot2 :initarg :slot2 :initform 20 :accessor slot2) + (slot3 :initarg :slot3 :initform 30 :accessor slot3 :index t) + (slot4 :initarg :slot4 :initform 40 :accessor slot4 :index t)) + (:metaclass persistent-metaclass)) (progn (with-transaction () @@ -168,12 +189,13 @@ (deftest indexing-range (progn ;; (format t "range store: ~A ~A~%" *store-controller* (controller-path *store-controller*)) - (when (class-indexedp-by-name 'idx-four ) - (defclass idx-four () - ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) - (:metaclass persistent-metaclass)) - (disable-class-indexing 'idx-four :errorp nil) - (setf (find-class 'idx-four nil) nil)) + + (defclass idx-four () + ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) + (:metaclass persistent-metaclass)) + + (disable-class-indexing 'idx-four :errorp nil) + (setf (find-class 'idx-four nil) nil) (defclass idx-four () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) @@ -205,9 +227,13 @@ (deftest indexing-slot-makunbound (progn - (when (class-indexedp-by-name 'idx-unbound-del) - (disable-class-indexing 'idx-unbound-del :errorp nil) - (setf (find-class 'idx-five-del) nil)) + + (defclass idx-unbound-del () + ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) + (:metaclass persistent-metaclass)) + + (disable-class-indexing 'idx-unbound-del :errorp nil) + (setf (find-class 'idx-five-del) nil) (defclass idx-unbound-del () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) @@ -227,8 +253,12 @@ (deftest indexing-wipe-index (progn - (when (class-indexedp-by-name 'idx-five-del) - (disable-class-indexing 'idx-five-del :errorp nil)) + + (defclass idx-five-del () + ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t)) + (:metaclass persistent-metaclass)) + + (disable-class-indexing 'idx-five-del :errorp nil) (setf (find-class 'idx-five-del) nil) (defclass idx-five-del () @@ -253,14 +283,15 @@ (deftest indexing-reconnect-db (progn - (when (class-indexedp-by-name 'idx-five) - (defclass idx-five () - ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) - (slot2 :initarg :slot2 :initform 2 :accessor slot2) - (slot3 :initarg :slot3 :initform 3 :accessor slot3 :index t)) - (:metaclass persistent-metaclass)) - (disable-class-indexing 'idx-five :errorp nil) - (setf (find-class 'idx-five) nil)) + + (defclass idx-five () + ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) + (slot2 :initarg :slot2 :initform 2 :accessor slot2) + (slot3 :initarg :slot3 :initform 3 :accessor slot3 :index t)) + (:metaclass persistent-metaclass)) + + (disable-class-indexing 'idx-five :errorp nil) + (setf (find-class 'idx-five) nil) (defclass idx-five () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) @@ -295,21 +326,22 @@ (deftest indexing-change-class (progn - (when (class-indexedp-by-name 'idx-six) - (defclass idx-six () - ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) - (slot2 :initarg :slot2 :initform 2 :accessor slot2 :index t)) - (:metaclass persistent-metaclass)) - (disable-class-indexing 'idx-six :errorp nil) - (setf (find-class 'idx-six) nil)) - (when (class-indexedp-by-name 'idx-seven) - (defclass idx-seven () - ((slot1 :initarg :slot1 :initform 10 :accessor slot1 :index nil) - (slot3 :initarg :slot3 :initform 30 :accessor slot3 :index t) - (slot4 :initarg :slot4 :initform 40 :accessor slot4 :index t)) - (:metaclass persistent-metaclass)) - (disable-class-indexing 'idx-seven :errorp nil) - (setf (find-class 'idx-seven) nil)) + (defclass idx-six () + ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) + (slot2 :initarg :slot2 :initform 2 :accessor slot2 :index t)) + (:metaclass persistent-metaclass)) + + (disable-class-indexing 'idx-six :errorp nil) + (setf (find-class 'idx-six) nil) + + (defclass idx-seven () + ((slot1 :initarg :slot1 :initform 10 :accessor slot1 :index nil) + (slot3 :initarg :slot3 :initform 30 :accessor slot3 :index t) + (slot4 :initarg :slot4 :initform 40 :accessor slot4 :index t)) + (:metaclass persistent-metaclass)) + + (disable-class-indexing 'idx-seven :errorp nil) + (setf (find-class 'idx-seven) nil) (defclass idx-six () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) @@ -350,16 +382,17 @@ (deftest indexing-redef-class (progn - (when (class-indexedp-by-name 'idx-eight) - (defclass idx-eight () - ((slot1 :accessor slot1 :initarg :slot1 :index t) - (slot2 :accessor slot2 :initarg :slot2) - (slot3 :accessor slot3 :initarg :slot3 :transient t) - (slot4 :accessor slot4 :initarg :slot4 :index t) - (slot5 :accessor slot5 :initarg :slot5)) - (:metaclass persistent-metaclass)) - (disable-class-indexing 'idx-eight :errorp nil) - (setf (find-class 'idx-eight nil) nil)) + + (defclass idx-eight () + ((slot1 :accessor slot1 :initarg :slot1 :index t) + (slot2 :accessor slot2 :initarg :slot2) + (slot3 :accessor slot3 :initarg :slot3 :transient t) + (slot4 :accessor slot4 :initarg :slot4 :index t) + (slot5 :accessor slot5 :initarg :slot5)) + (:metaclass persistent-metaclass)) + + (disable-class-indexing 'idx-eight :errorp nil) + (setf (find-class 'idx-eight nil) nil) ;; (format t "sc: ~A ct: ~A~%" *store-controller* *current-transaction*) (defclass idx-eight () ((slot1 :accessor slot1 :initarg :slot1 :index t) @@ -368,11 +401,13 @@ (slot4 :accessor slot4 :initarg :slot4 :index t) (slot5 :accessor slot5 :initarg :slot5)) (:metaclass persistent-metaclass)) + (let ((o1 nil) (o2 nil)) (with-transaction () (setf o1 (make-instance 'idx-eight :slot1 1 :slot2 2 :slot3 3 :slot4 4 :slot5 5)) (setf o2 (make-instance 'idx-eight :slot1 10 :slot2 20 :slot3 30 :slot4 40 :slot5 50))) + (defclass idx-eight () ((slot1 :accessor slot1 :initarg :slot1 :initform 11) (slot2 :accessor slot2 :initarg :slot2 :initform 12 :index t)

1 0

[elephant-cvs] CVS elephant
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv6422 Modified Files: TODO config.sexp ele-bdb.asd Log Message: Cleanup indexing tests so we always have a clean slate --- /project/elephant/cvsroot/elephant/TODO 2007/03/21 15:04:59 1.74 +++ /project/elephant/cvsroot/elephant/TODO 2007/03/24 12:16:02 1.75 @@ -30,8 +30,11 @@ - Class / DB sychronization tests Documentation: -- License and copyright file headers -- Add document section about backend interface & developer decisions +~ License and copyright file headers +- Redo tutorial +- Proper user guide +- Update install, build and test procedures +- Upgrade, migration and other system level issues - Performance and design issues - More notes about transaction performance - Serious discussion of threading implications @@ -39,7 +42,7 @@ - Add notes about optimize-storage - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) -- Upgrade, migration and other system level issues +- Add document section about backend interface & developer decisions 0.6.1 - Features COMPLETED to date ---------------------------------- --- /project/elephant/cvsroot/elephant/config.sexp 2007/03/19 20:35:30 1.8 +++ /project/elephant/cvsroot/elephant/config.sexp 2007/03/24 12:16:02 1.9 @@ -25,10 +25,10 @@ (:compiler . :gcc)) #+(or mswindows windows) -((:berkeley-db-include-dir . "C:/Programme/Oracle/Berkeley DB 4.5.20/include/") - (:berkeley-db-lib-dir . "C:/Programme/Oracle/Berkeley DB 4.5.20/bin/") - (:berkeley-db-lib . "C:/Programme/Oracle/Berkeley DB 4.5.20/bin/libdb45.dll") - (:berkeley-db-deadlock . "C:/Programme/Oracle/Berkeley DB 4.5.20/bin/db_deadlock.exe") +((:berkeley-db-include-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/include/") + (:berkeley-db-lib-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/") + (:berkeley-db-lib . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/libdb45.dll") + (:berkeley-db-deadlock . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/db_deadlock.exe") (:pthread-lib . nil) (:clsql-lib . nil) (:compiler . :cygwin)) --- /project/elephant/cvsroot/elephant/ele-bdb.asd 2007/03/20 02:00:07 1.24 +++ /project/elephant/cvsroot/elephant/ele-bdb.asd 2007/03/24 12:16:02 1.25 @@ -37,6 +37,7 @@ (drive-letter (char result (1- colon-pos)))) (setf (char result (1- colon-pos)) #\/) (setf (char result colon-pos) drive-letter) + (setf result (concatenate 'string "/cygdrive" result)) (substitute #\/ #\\ result))) (defclass bdb-c-source (elephant-c-source) ())

1 0

[elephant-cvs] CVS elephant/tests
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv20292 Modified Files: testmigration.lisp Log Message: Fix for add-to-root, etc api change --- /project/elephant/cvsroot/elephant/tests/testmigration.lisp 2007/03/18 20:47:28 1.19 +++ /project/elephant/cvsroot/elephant/tests/testmigration.lisp 2007/03/24 10:51:45 1.20 @@ -43,10 +43,10 @@ (disable-class-indexing x :sc sc1)) '(idx-two idx-three idx-four idx-five idx-six idx-seven idx-eight idx-five-del stress-index idx-unbound-del)) - (add-to-root "x" "y" :store-controller sc1) + (add-to-root "x" "y" :sc sc1) (migrate sc2 sc1) - (equal (get-from-root "x" :store-controller sc1) - (get-from-root "x" :store-controller sc2))) + (equal (get-from-root "x" :sc sc1) + (get-from-root "x" :sc sc2))) (close-store sc1) (close-store sc2)))) t)

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv19929 Modified Files: migrate.lisp Log Message: Fix for add-to-root, etc api change --- /project/elephant/cvsroot/elephant/src/elephant/migrate.lisp 2007/03/11 05:45:14 1.12 +++ /project/elephant/cvsroot/elephant/src/elephant/migrate.lisp 2007/03/24 10:49:59 1.13 @@ -125,7 +125,7 @@ (let ((newval (migrate dst value))) (unless (eq key *elephant-properties-label*) (ensure-transaction (:store-controller dst :txn-nosync t) - (add-to-root key newval :store-controller dst))))) + (add-to-root key newval :sc dst))))) (controller-root src)) dst)

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 24 Mar '07

24 Mar '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv24154 Modified Files: controller.lisp Log Message: Reduce typing for store controller keyword --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/23 16:08:10 1.40 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/24 03:03:00 1.41 @@ -420,34 +420,34 @@ ; Operations on the root index ;; -(defun add-to-root (key value &key (store-controller *store-controller*)) +(defun add-to-root (key value &key (sc *store-controller*)) "Add an arbitrary persistent thing to the root, so you can retrieve it in a later session. N.B. this means it (and everything it points to) won't get gc'd." (declare (type store-controller store-controller)) (assert (not (eq key *elephant-properties-label*))) - (setf (get-value key (controller-root store-controller)) value)) + (setf (get-value key (controller-root sc)) value)) -(defun get-from-root (key &key (store-controller *store-controller*)) +(defun get-from-root (key &key (sc *store-controller*)) "Get a something from the root." - (declare (type store-controller store-controller)) - (get-value key (controller-root store-controller))) + (declare (type store-controller sc)) + (get-value key (controller-root sc))) -(defun root-existsp (key &key (store-controller *store-controller*)) +(defun root-existsp (key &key (sc *store-controller*)) "Test whether a key exists in the root" - (declare (type store-controller store-controller)) - (if (existsp key (controller-root store-controller)) + (declare (type store-controller sc)) + (if (existsp key (controller-root sc)) t nil)) -(defun remove-from-root (key &key (store-controller *store-controller*)) +(defun remove-from-root (key &key (sc *store-controller*)) "Remove something from the root." - (declare (type store-controller store-controller)) - (remove-kv key (controller-root store-controller))) + (declare (type store-controller sc)) + (remove-kv key (controller-root sc))) -(defun map-root (fn &key (store-controller *store-controller*)) +(defun map-root (fn &key (sc *store-controller*)) "Map over all key-value pairs in the root" - (map-btree fn (controller-root store-controller))) + (map-btree fn (controller-root sc))) ;; ;; Explicit storage reclamation

1 0

[elephant-cvs] CVS elephant
by ieslick 23 Mar '07

23 Mar '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv22309 Modified Files: ele-clsql.asd Log Message: Removed legacy path computations for clsql --- /project/elephant/cvsroot/elephant/ele-clsql.asd 2007/02/07 22:54:12 1.9 +++ /project/elephant/cvsroot/elephant/ele-clsql.asd 2007/03/23 16:31:52 1.10 @@ -27,18 +27,18 @@ ;; (clsql:push-library-path *elephant-lib-path*) ;; ) -(defparameter *clsql-foreign-lib-path* #p"/usr/lib") -(defparameter *elephant-lib-path* #p"/usr/local/share/common-lisp/elephant/") +;;(defparameter *clsql-foreign-lib-path* #p"/usr/lib") +;;(defparameter *elephant-lib-path* #p"/usr/local/share/common-lisp/elephant/") -(defmethod asdf:perform :after ((o asdf:load-op) - (c (eql (asdf:find-system 'clsql)))) - (let ((plp (find-symbol (symbol-name '#:push-library-path) - (find-package 'clsql)))) - (funcall plp - *clsql-foreign-lib-path*) - (funcall plp - *elephant-lib-path*) -)) +;;(defmethod asdf:perform :after ((o asdf:load-op) +; (c (eql (asdf:find-system 'clsql)))) +;; (let ((plp (find-symbol (symbol-name '#:push-library-path) +;; (find-package 'clsql)))) +;; (funcall plp +;; *clsql-foreign-lib-path*) +;; (funcall plp +;; *elephant-lib-path*) +;;)) (defsystem ele-clsql :name "elephant"

1 0