- elephant-cvs@lists.common-lisp.net - mailman3.common-lisp.net

[elephant-cvs] CVS elephant
by ieslick 06 Apr '07

06 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv21893 Modified Files: TODO elephant.asd Log Message: Trial pset abstraction; fix for debug serialize of complex and more documentation edits --- /project/elephant/cvsroot/elephant/TODO 2007/04/01 14:33:23 1.77 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/06 02:51:46 1.78 @@ -18,7 +18,7 @@ - Verify db_deadlock for other lisps (launch and kill background program I/F) Bugs: -- Fix awkward serializer API +- Fix awkward serializer API - Support for asdf-install? - Edi's patches & suggestions for windows @@ -44,6 +44,11 @@ - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) +Additional features to document in Trac: +- Add error/restart conditions when lose data on: class redef, index drop, etc. + +:association slot option + 0.6.1 - Features COMPLETED to date ---------------------------------- --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/01 14:33:23 1.41 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/06 02:51:46 1.42 @@ -298,6 +298,7 @@ (:file "serializer") (:file "controller") (:file "collections") + (:file "pset") (:file "classindex-utils") (:file "classindex") (:file "serializer1") ;; 0.6.0 db's

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 06 Apr '07

06 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv21893/doc Modified Files: user-guide.texinfo Log Message: Trial pset abstraction; fix for debug serialize of complex and more documentation edits --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/04 15:28:28 1.9 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/06 02:51:47 1.10 @@ -89,41 +89,33 @@ When you finish your application, @code{close-store} will close the store controller. Failing to do this properly may lead to a need to run recovery on the data store during the next session. Again, see -the relevant data store sections for more details. +the relevant data store sections for more detail. + @node Serialization details @comment node-name, next, previous, up @section Serialization details -This section captures the details of how various types of objects are -serialized and some considerations to keep in mind when storing lisp -objects. - -The high level factors that you need to keep in mind are: - -@itemize -@item Circular References: -The serializer properly handles circular references to/from objects -such as cons cells, standard objects, arrays, etc. It accomplishes -this by assigning an ID to any non-atomic object and keeping a mapping -between previously serialized objects and these ids. -@end itemize +There are consequences to trying to move values from lisp memory onto +disk in order to persist them. The first consequence is that that +pointers cannot be guaranteed to be valid and so references to lisp +objects cannot be maintained. This is very similar to the problems +with passing references in foreign function interfaces. The second, +and more frustrating limitation is that lisp operations that commit +side effects on aggregate objects, such as objects, arrays, etc, +cannot be trapped and replicated on the disk representation. This +leads up to a very important consequence: all lisp objects are stored +by @emph{value}. This policy has a number of consequences which are +detailed below. -Here is an introduction to - -@itemize -@item -@end itemize - -We will also review and add to the considerations outlined in the tutorial: +@subsection{Restrictions of Store-by-Value} @enumerate - - -@item @strong{Lisp identity can't be preserved}. Since this is a store which -persists across invocations of Lisp, this probably doesn't even make -sense. However if you get an object from the index, store it to a -lisp variable, then get it again - they will not be eq: +@item @strong{Lisp identity can't be preserved}. + Since this is a store which persists across invocations of Lisp, +this probably doesn't even make sense. However if you get an object +from the index, store it to a lisp variable, then get it again - they +will not be eq: @lisp (setq foo (cons nil nil)) @@ -137,17 +129,35 @@ => NIL @end lisp -@item @strong{Nested aggregates are stored in one buffer}. +@item @strong{Nested aggregates are serialized recursively into a single buffer}. If you store an set of objects in a hash table you try to store a hash table, all of those objects will get stored in one large binary buffer -with the hash keys. This is true for all other aggregates that can -store type T (cons, array, standard object, etc). +with the hash keys. This is true for all aggregates that can store +type T (cons, array, standard object, etc). @item @strong{Circular References}. -The serializer properly handles circular references to/from objects -such as cons cells, standard objects, arrays, etc. It accomplishes +One benefit provided by the serializer is that the recursive +serialization process does not lead to infinite loops when they +encounter circular references among aggregate types. It accomplishes this by assigning an ID to any non-atomic object and keeping a mapping -between previously serialized objects and these ids. +between previously serialized objects and these ids. This same +mapping is used to reconstruct references in lisp memory on +deserialization such that the original structure is properly +reproduced. + +@item @strong{Storage limitations}. +The serializer writes sequentially into a contiguous foreign byte +array before passing that array to a given data store's API. There +are practical limits to the size of the foreign buffer that lisp can +allocate (usually somewhere on the order of 10-100MB due to address +space fragmentation). Moreoever, most data stores will have a +practical limit to the size of a transaction or the size of key or +value they will store. Either of these considerations should +encourage you to plan to limit the size of objects that you serialize +to disk. A good rule of thumb is to stay under a handful of +megabytes. We have successfully serialized arrays over 100MB in the +past, but have not tested the robustness of these large values over +time. @item @strong{Mutated substructure does not persist}. @@ -163,15 +173,6 @@ elephant does not automatically provide persistent collections. If you want to persist every access, you have to use BTrees (@pxref{Using BTrees}). -@item @strong{Storage limitations}. -The serializer writes sequentially into a foreign memory byte array -before passing that array to a given data store's API. There are -practical limits to the size of this buffer. Moreoever, in most data -stores there is a practical limit to the size of a transaction. -Either of these considerations should encourage you to plan to limit -the size of objects that you serialize to disk. A good rule of thumb -is to stay under a megabyte. - @item @strong{Serialization and deserialization can be costly}. While serialization is pretty fast, but it is still expensive to store large objects wholesale. Also, since object identity is impossible to @@ -185,8 +186,147 @@ This is the common read-modify-write problem in all databases. We will talk more about this in the @ref{Transactions} section. +@item @strong{Byte Ordering}. + The primitive elements such as integers are written to disk in +the native byte ordering of the machine on which the lisp runs. This +means that little endian machines cannot read values written by big +endian machines and vice a versa. + +@item @strong{Unicode codes and Serialized Strings}. + The characters and strings stored to disk can store and recover +lisp character codes that implement unicode, but the character maps +are the lisp character maps (produced by @code{char-code}) and not +strict unicode codes so lisps may not be able to interoperably read +characters unless they have identical character code maps for the +character sets you are reading and writing. All standard ASCII +strings should be portable. Here is what we know about specific +lisps, but this should not be taken as gospel. +@itemize +@item SBCL: In versions with the :sb-unicode feature (after 0.8.17) @code{char-code} + produces proper Unicode codes +@item Allegro: In the interational version, @code{char-code} produces proper Unicode codes for codes < 2^16 +@item OpenMCL: OpenMCL 1.1 supports unicode, we are unsure about earlier versions +@item Lispworks: Lispworks 5 does not, to our knowledge, produce proper Unicode characters. +(@emph{This can be fixed on request iff users ask for it and are willing to pay the performance hit}) +@end itemize + @end enumerate +@subsection{Atomic Types} + +Atomic types have no recursive substructure. That is they cannot +contain arbitrary objects and are of a bounded size. (Bignums are an +exception, but they have a predictable structure and cannot reference +or otherwise encapsulate other objects). The following is a list of +atoms and a discussion of how they are serialized. + +@itemize +@item @strong{@code{nil}}: + nil has it's own special tag in the serializer so it is easily +identifiable. @code{nil} is an awkward value as it is also a boolean. +The boolean value @code{t} is stored as the symbol 'T. +@item @strong{fixnums}: + The serializer will store both 32-bit and 64-bit fixnums. Both +types of fixnums are readable by a 32-bit or 64-bit lisp, but 64-bit +fixnums are only written if the underlying lisp is supports fixnums +between 32 and 64 bits. +@item @strong{bignums}: + Bignums are broken into a sequence of fixnum-sized chunks and +assembled by masking words onto the bignum. This is awfully +expensive, but it's always correct and fully portable. +@item @strong{small-float}: + Supported only on Lispworks 5 where type @code{small-float} is +not equivalent to type @code{single-float} as it is on all other +supported platforms. Written to disk and deserialized as a single +float so any memory footprint savings of @code{small-float} is lost. +@item @strong{single-float}: + 32-bit floating point numbers +@item @strong{double-float}: + 64-bit floating point numbers +@item @strong{rational}: + A rational is merely a ratio of two integers stored as fixnums or bignums. +@item @strong{complex}: + A complex is a pair of floating point values, rationals or integers. +@item @strong{char}: + Standalone chars are represented by their char-code and are +stored in 32-bit format to ensure that all lisps are stored correctly. +@item @strong{strings}: + Strings can be represented as 8, 16 or 32 bit sequences +depending on the character sizes used in the underlying lisp. Because +strings can be such a large percentage of on-disk space, Elephant uses +a peculiar method of encoding strings. Strings are converted from +their in-memory representation using @code{char-code}. The size of +the first character dictates the word width used for encoding. If a +character violates the word width, the string encoding is aborted and +the next larger width is chosen. The rationale here is that many +strings consist of Latin characters with codes less than 256. Strings +stored in other character sets tend to all be of codes > 256. +Therefore it is likely that the first character will properly +determine the word size of the string. (@emph{On request, we can easily make +a configuration option to fix the word width for encoding}) +@item @strong{pathname}: + A pathname is merely the @code{namestring} of the path object +stored as a string. The path object is reconstructed from the +namestring using @code{parse-namestring} during deserialization. +@item @strong{symbol}: + Symbols are stored as two strings, the package name and the symbol name in that package. When deserialized, the target package is searched for and the symbol is interned in that package. +@end itemize + +@subsection{Aggregate Types} + +The next list are @emph{aggregate} types, meaning that elements of +that type can contain references to elements of type @code{T}. That +means, in theory, that storing an aggregate type to disk that refers +to other objects can copy every reachable object! This is a direct +and dire consequence of the ``store-by-value'' restriction. +(@xref{Persistent Classes and Objects} for how to design around the +store-by-value restriction}). + +This list describes how aggregates are handled by the serializer. + +@itemize +@item @strong{cons}: + Cons is simply stored as a cons record containing two nested +elements. Linear lists are not treated specially (i.e. no cdr-coding) +by the serializer. +@item @strong{array}: + Arrays are stored as sequences of nested, serialized elements. +The array parameters are also stored so that arrays with fill +pointers, adjustable arrays can be stored and reconstructed. The only +arrays that cannot be reproduced are displaced arrays, which are +copied by value and reconstructed as standard arrays during +deserialization. +@item @strong{hash-table}: + Hash tables are stored as a sequence of key-value pairs, where +the key and value can be any serializable value. On deserialization, +the reconstructed key and value quantities are written incrementally +into the hash table. The hash table does remember it's test, rehash +size and threshold and it's total count. The final size of the new +hash table is set to @code{(* (/ size reshash-threshold) rehash-size)}. +@item @strong{struct}: + Structure objects are serialized using the metaprotocol. Each +slot where the value is bound is serialized by serializing the slot +name and the value in sequence. The underlying lisp must support the +@code{struct-constructor} method so that a new, empty instance of the +structure can be created and then populated by the stored keys and +values. +@item @strong{object}: + Instances of subclasses of standard-object are stored almost +identically to structs. The type of the object is stored and the +object slots with bound values are serialized as slotname-value pairs. +To read an object of this type, the lisp image must have the class +defined and it must have at least the slots that are stored on disk. +There is no good method for schema evolution (redefining objects to +have less slots) of ordinary classes. +@end itemize + + +One final strategic consideration is whether you plan on sharing the binary +database between machines or between different lisp platforms on the +same machine. This is almost possible today, but there are some +restrictions. In the section @ref{Repository Migration and Upgrade} +we will discuss possible ways of migrating an existing database across +platforms and lisps. @node Persistent Classes and Objects @comment node-name, next, previous, up

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 04 Apr '07

04 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv10431 Modified Files: tutorial.texinfo Log Message: Fix to docs after adding complex --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/04 15:28:28 1.14 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/04 15:35:44 1.15 @@ -204,14 +204,13 @@ @section Serialization What can you put into the store besides strings? Almost all lisp -values and objects can be stored: numbers (except for complexes), -symbols, strings, nil, characters, pathnames, conses, hash-tables, -arrays, CLOS objects and structs. Nested and circular things are -allowed. Nested and circular things are allowed. You can store -basically anything except compiled functions, closures, class objects, -packages and streams. Functions can be stored as uncompiled lambda -expressions. (Compiled functions and other kinds of objects may -eventually get supported too.) +values and objects can be stored: numbers, symbols, strings, nil, +characters, pathnames, conses, hash-tables, arrays, CLOS objects and +structs. Nested and circular things are allowed. Nested and circular +things are allowed. You can store basically anything except compiled +functions, closures, class objects, packages and streams. Functions +can be stored as uncompiled lambda expressions. (Compiled functions +and other kinds of objects may eventually get supported too.) Elephant needs to use a representation of data that is independant of a specific lisp or data store. Therefore all lisp values that are

1 0

[elephant-cvs] CVS elephant/tests
by ieslick 04 Apr '07

04 Apr '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv9316/tests Modified Files: testserializer.lisp Log Message: Added support for complex serialization (no sorting), latest doc changes and a preliminary GC wrapper --- /project/elephant/cvsroot/elephant/tests/testserializer.lisp 2007/02/25 20:02:32 1.19 +++ /project/elephant/cvsroot/elephant/tests/testserializer.lisp 2007/04/04 15:28:30 1.20 @@ -173,6 +173,18 @@ (in-out-equal (/ (expt 2 200) (expt 3 300))) (in-out-equal (/ (expt 2 200) (- (expt 3 300))))) t t t t t t t) + +(deftest complexes + (are-not-null + (in-out-equal (sqrt -1)) + (in-out-equal (complex 1)) + (in-out-equal (complex 1.0)) + (in-out-equal (complex (/ 1 2) (/ 2 3))) + (in-out-equal #C(1.0 0.0)) + (in-out-equal #C(2 3)) + (in-out-equal (complex most-positive-fixnum most-negative-fixnum)) + (in-out-equal (complex (expt 2 200) (expt 3 201)))) + t t t t t t t t) (deftest base-strings (are-not-null

1 0

[elephant-cvs] CVS elephant/src/db-bdb
by ieslick 04 Apr '07

04 Apr '07

Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv9316/src/db-bdb Modified Files: libberkeley-db.c Log Message: Added support for complex serialization (no sorting), latest doc changes and a preliminary GC wrapper --- /project/elephant/cvsroot/elephant/src/db-bdb/libberkeley-db.c 2007/03/18 20:40:50 1.9 +++ /project/elephant/cvsroot/elephant/src/db-bdb/libberkeley-db.c 2007/04/04 15:28:28 1.10 @@ -1030,6 +1030,7 @@ #define S2_ARRAY 19 #define S2_STRUCT 20 #define S2_CLASS 21 +#define S2_COMPLEX 22 #define S2_NIL 0x3F #define S2_FILL_POINTER_P 0x40

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 04 Apr '07

04 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv9316/src/elephant Modified Files: migrate.lisp serializer2.lisp Added Files: gc.lisp Log Message: Added support for complex serialization (no sorting), latest doc changes and a preliminary GC wrapper --- /project/elephant/cvsroot/elephant/src/elephant/migrate.lisp 2007/03/24 10:49:59 1.13 +++ /project/elephant/cvsroot/elephant/src/elephant/migrate.lisp 2007/04/04 15:28:29 1.14 @@ -42,8 +42,9 @@ ;; - Migrate keeps a memory-resident hash of all persistent objects; ;; this is not as bad as it sounds as an object is only an oid reference ;; and a pointer to the store controller it belongs to. However, you -;; may eventually run out of heap space for very large DB's. We can use -;; the old DB to store the mappings if this becomes a problem. +;; may eventually run out of heap space for very large DB's. You can use +;; a third store controller to get around this by setting set-oid-spec to +;; a valid, uncreated store specification. ;; ;; - Each top-level call to migration will be good about keeping track ;; of already copied persistent objects. However the hash is not --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/03/30 23:36:53 1.35 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/04 15:28:29 1.36 @@ -80,6 +80,7 @@ (defconstant +array+ 19) (defconstant +struct+ 20) (defconstant +class+ 21) +(defconstant +complex+ 22) ;; Lispworks support (defconstant +short-float+ 30) @@ -251,6 +252,10 @@ (let ((pstring (namestring frob))) (buffer-write-byte +pathname+ bs) (serialize-string pstring bs))) + (complex + (buffer-write-byte +complex+ bs) + (%serialize (realpart frob)) + (%serialize (imagpart frob))) (hash-table (buffer-write-byte +hash-table+ bs) (let ((idp (gethash frob circularity-hash))) @@ -459,6 +464,10 @@ (setf (car c) (%deserialize bs)) (setf (cdr c) (%deserialize bs)) c)))) + ((= tag +complex+) + (let ((rpart (%deserialize bs)) + (ipart (%deserialize bs))) + (complex rpart ipart))) ((= tag +hash-table+) (let* ((id (buffer-read-fixnum bs)) (maybe-hash (lookup-id id))) --- /project/elephant/cvsroot/elephant/src/elephant/gc.lisp 2007/04/04 15:28:29 NONE +++ /project/elephant/cvsroot/elephant/src/elephant/gc.lisp 2007/04/04 15:28:29 1.1 (in-package :elephant) (defgeneric stop-and-copy-gc (sc &key &allow-other-keys) (:documentation "Wrap all the migrate machinery in a simple top-level gc call. This will copy all the data in the source store to the target store")) (defmethod stop-and-copy-gc ((src store-controller) &key target mapspec replace-source delete-source) (when map-spec (set-oid-spec mapspec)) (let ((target (gc-open-target sc target-spec)) (src-spec (controller-spec src))) (migrate target src) (when map-spec (set-oid-spec nil) (delete-spec mapspec)) (when (or replace-source delete-source) (close-store src) (delete-spec src)) (when replace-source (copy-spec source target)))) (defun delete-spec (spec) "Delete the storage associated with spec" ) (defun copy-spec (src targ) "Copy files associated with spec from src to targ" )

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 04 Apr '07

04 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv9316/doc Modified Files: installation.texinfo scenarios.texinfo tutorial.texinfo user-guide.texinfo Log Message: Added support for complex serialization (no sorting), latest doc changes and a preliminary GC wrapper --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/02 00:51:06 1.7 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/04 15:28:28 1.8 @@ -175,31 +175,32 @@ @subsection Packages Now that Elephant has been loaded, you can call @code{use-package} in -the cl-user package or create a new package that imports the symbols -exported from package :elephant. +the cl-user package, @lisp CL-USER> (use-package :elephant) => T +@end lisp -OR +use a predefined user package, -(defpackage :elephant-user - (:use :common-lisp :elephant)) +@lisp +CL-USER> (in-package :elephant-user) +=> T + +ELE-USER> @end lisp -Beginners can skip to the end of this section. +or import the symbols into your own project package from :elephant. -Elephant has a common package called elephant that exports a set of -generic functions. It also contains a dispatcher based on the first -element of a specification list that calls the relevant backend -version of @code{open-controller}, the internal method that creates a -@code{store-controller}. Each backend has it's own subclass -implementing the abstract interface of @code{store-controller}. +@lisp +(defpackage :my-project + (:use :common-lisp :elephant)) +@end lisp @subsection Opening a Store -As discussed in the tutoral, you can now open a store to begin using +As discussed in the tutoral, you need to open a store to begin using Elephant: @lisp --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/02 13:09:46 1.2 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/04 15:28:28 1.3 @@ -5,39 +5,138 @@ @chapter Usage Scenarios @cindex Usage Scenarios -Sorry, haven't written this section yet. - -Simple file replacement and indexing -- Keep track of ordinary objects, ignore metaprotocol - -Persist system objects -- Intermingle persistent objects and regular objects -- Look up objects using class indices - -Full database system -- storage, rich data models, references, queries, etc - -Multithreaded web applications -- DB + multithreading - -Object-oriented data storage, large graph traversals - - -@node Commercial Applications - -Elephant is used by Konsenti(tm), a for-profit company of Robert L. Read, one of the maintainers of Elephant. It can be visited at -@uref{http://konsenti.com}. Konsenti uses the Data Collection Management (DCM) package, which can be -found in the contrib directory, under user rread. DCM provides prevalence-style in-memory write-through caching. -The most enjoyable feature about Elephant for this project is that new Business Layer objects can be created without having to -deal with an Object-Relational Mapping, which has allowed extremely rapid development. All Business objects are persisted via -a director in DCM (which sits on top of Elephant.) Many of these business objects are in fact finite state machines decorated -with functions. The functions are represented by their lambda expression stored in slots on the business objects. A complete -Message Factory and double-entry accounting system are also implemented as DCM objects. Binary objects, such as uploaded -PDFs that can be attached to objects as comments, are treated as simple objects and stored directly in Elephant. Konsenti -is completely based on utf-8, and unicode characters outside of the ISO-9959-1 character set are routinely stored in -Elephant. Konsenti uses Postgres as a backend; but Elephant makes it so easy to migrate between repositories that we -could change this decision at any time. - - +@menu +* File Replacement:: Simple deployment of Elephant as file replacement +* Persistent System Objects:: Making persistent objects a natural part of your system +* Crash Recovery:: How to recover application state from application or system crashes +* Elephant as Database:: Using Elephant as a database for records and user data instead of using a SQL relational Database +* Multithreaded Web Applications:: Elephant is a natural match for web applications +* Graph-oriented Applications:: Elephant is good, but not optimized, for graph-oriented applications. +* Real-World Application Examples:: See some real-world applications Elephant has been used for and a brief discussion of how it was used and any novel uses of Elephant. +@end menu + +@node File Replacement +@comment node-name, next, previous, up +@section File Replacement + +One of the annoying overheads in writing many programs is persisting +data between lisp sessions or invocations of that program. Elements +such as configuration files, raw data such as graphics and other +formats take time, attention and are a potential source of bugs. +Elephant can ease these concerns and allow you to work directly with +your natural in-memory representations with no work to encode/decode +formats or manage files in the file system. + +The simplest way to accomplish this is to simply open a store +controller and use the root btree as a key-value store instead of a +file system directory. You might hide some of the complexity sort of +like this: + +@lisp +(defmacro def-resource (name initializer) + (assert (symbolp name)) + `(defparameter name (list nil nil ,initializer))) + +(defun call-initializer (init) + (case init-stmt + (symbol (funcall (symbol-function init-stmt))) + (list (apply (first init) (rest init))))) + +(defun get-resource (name) + (if (and (symbol-value name) + (symbol-value-name) + (let ((newval (get-from-root name))) + (if newval + (setq name (add-to-root name newval)) + (setq name (add-to-root name (call-initializer +@end lisp + +@node Persistent System Objects +@comment node-name, next, previous, up +@section Persistent System Objects + +Persist system objects: + +@itemize +@item Intermingle persistent objects and regular objects +@item Look up objects using class indices +@end itemize + +@node Crash Recovery +@comment node-name, next, previous, up +@section Crash Recovery + +@node Elephant as Database +@comment node-name, next, previous, up +@section Elephant as Database + +Full database system: storage, rich data models, references, queries, etc + +@node Multithreaded Web Applications +@comment node-name, next, previous, up +@section Multithreaded Web Applications + +Multithreaded web applications: DB + multithreading + web objects + +@node Graph-oriented Applications +@comment node-name, next, previous, up +@section Graph-oriented Applications + +@node Real-World Applications +@comment node-name, next, previous, up +@section Real-World Application + +@subsection Konsenti + +Elephant is used by Konsenti(tm), a for-profit company of Robert +L. Read, one of the maintainers of Elephant. It can be visited at +@uref{http://konsenti.com}. + +Konsenti uses the Data Collection Management (DCM) package, found in +the @verbatim{src/contrib/rread directory}. DCM provides +prevalence-style in-memory write-through caching. The most enjoyable +feature about Elephant for this project is that new Business Layer +objects can be created without having to deal with an +Object-Relational Mapping, enabling extremely rapid development. + +All Business objects are persisted via a @code{director} in DCM (which +sits on top of Elephant.) Many of these business objects are in fact +finite state machines decorated with functions. The functions are +represented by lambda s-expressions stored in slots on the business +objects. A complete Message Factory and double-entry accounting +system are also implemented as DCM objects. Binary objects, such as +uploaded PDFs, can be attached to objects as comments and are stored +directly in Elephant. Konsenti is based on utf-8, and unicode +characters outside of the ISO-9959-1 character set are routinely +stored in Elephant. Konsenti uses Postgres as a backend for licensing +reasons; but use of other data stores is possible. + +@node Conceptminer +@subsection Conceptminer + +Conceptminer is an Elephant-based web-mining framework developed by +Ian Eslick (@uref{http://www.media.mit.edu/~eslick}) that performs +large-scale text analysis over the web to identify semantic +relationships such as ``PartOf'', ``DesireOf'' and ``EffectOf'' +between English phrases. + +Elephant's persistence capabilities is used to keep full records of +all source material, extracted relationships and search queries so +that it is always possible to trace the source of a learned relation +and to avoid repeated queries to web search engines. Conceptminer +used Elephant 0.6.0 and the development branch of Elephant 0.9 to +perform months of analysis consisting of millions of pages and a +page/query database of over ten gigabytes. + +There are several uses of Elephant in Conceptminer that bear +examination: + +@itemize +@item Process Components +@item Bulk storage of post-processed web data +@item Class indexes on strings +@item Cheap associations +@item Inverted document index +@end itemize --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/02 00:51:06 1.13 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/04 15:28:28 1.14 @@ -520,9 +520,9 @@ But what if we want to read out our friends from oldest to youngest? One way is to employ another btree that maps birthdays to names, but -this will require storing values multiple times for each update and -increases the burden on the programmer. Elephant provides a better -way. +this requires multiple @code{get-value} calls for each update, +increasing the burden on the programmer. Elephant provides several +better ways to do this. The next section @ref{Indexing Persistent Classes} shows you how to order and retrieve persistent classes by one or more slot values. @@ -547,20 +547,20 @@ (defmethod print-object ((f friend) stream) (format stream "#<~A>" (name f))) -(defun encode-birthday (dmy) +(defun encode-date (dmy) (apply #'encode-universal-time (append '(0 0 0) dmy))) (defmethod (setf birthday) (dmy (f friend)) (setf (slot-value f 'birthday) - (encode-birthday dmy)) + (encode-date dmy)) dmy) -(defun decode-birthday (utime) +(defun decode-date (utime) (subseq (multiple-value-list (decode-universal-time utime)) 3 6)) (defmethod birthday ((f friend)) - (decode-birthday (slot-value f 'birthday))) + (decode-date (slot-value f 'birthday))) @end lisp Notice the class argument ``:index t''. This tells Elephant to store @@ -579,9 +579,9 @@ (defun print-friend (friend) (format t " name: ~A birthdate: ~A~%" (name friend) (birthday friend))) -(make-instance 'friend :name "Carlos" :birthday (encode-birthday '(1 1 1972))) -(make-instance 'friend :name "Adriana" :birthday (encode-birthday '(24 4 1980))) -(make-instance 'friend :name "Zaid" :birthday (encode-birthday '(14 8 1976))) +(make-instance 'friend :name "Carlos" :birthday (encode-date '(1 1 1972))) +(make-instance 'friend :name "Adriana" :birthday (encode-date '(24 4 1980))) +(make-instance 'friend :name "Zaid" :birthday (encode-date '(14 8 1976))) (get-instances-by-class 'friends) => (#<Carlos> #<Adriana> #<Zaid>) @@ -629,14 +629,14 @@ (birthday :initarg :birthday :index t))) @end lisp -Notice the :index argument to the slots. Also notice that we dropped -the class :index argument. Specifying that a slot is indexed -automatically registers the class as indexed. While slot indices -increase the cost of writes and disk storage, each entry is only -slightly larger than the size of the slot value. Numbers, small -strings and symbols are good candidate types for indexed slots, but -any value may be used, even different types. Once a slot is indexed, -we can use the index to retrieve objects by slot values. +Notice the :index argument to the slots and that we dropped the class +:index argument. Specifying that a slot is indexed automatically +registers the class as indexed. While slot indices increase the cost +of writes and disk storage, each entry is only slightly larger than +the size of the slot value. Numbers, small strings and symbols are +good candidate types for indexed slots, but any value may be used, +even different types. Once a slot is indexed, we can use the index to +retrieve objects by slot values. @code{get-instances-by-value} will retrieve all instances that are equal to the value argument. @@ -652,7 +652,7 @@ (get-instances-by-range 'friends 'name "Adam" "Devin") => (#<Adriana> #<Carlos>) -(get-instances-by-range 'friend 'birthday (encode-birthday '(1 1 1974)) (encode-birthday '(31 12 1984))) +(get-instances-by-range 'friend 'birthday (encode-date '(1 1 1974)) (encode-date '(31 12 1984))) => (#<Zaid> #<Adriana>) (mapc #'print-friend *) @@ -676,8 +676,8 @@ @end lisp There are also functions for mapping over instances of a slot index. -To map over values, use the :value keyword argument. To map by range, -use the :start and :end arguments. +To map over duplicate values, use the :value keyword argument. To map +by range, use the :start and :end arguments. @lisp (map-class-index #'print-friend 'friend 'name :value "Carlos") @@ -690,21 +690,21 @@ => NIL (map-class-index #'print-friend 'friend 'birthday - :start (encode-birthday '(1 1 1974)) - :end (encode-birthday '(31 12 1984))) + :start (encode-date '(1 1 1974)) + :end (encode-date '(31 12 1984))) name: Zaid birthdate: (14 8 1976) name: Adriana birthdate: (24 4 1980) => NIL (map-class-index #'print-friend 'friend 'birthday :start nil - :end (encode-birthday '(10 10 1978))) + :end (encode-date '(10 10 1978))) name: Carlos birthdate: (1 1 1972) name: Zaid birthdate: (14 8 1976) => NIL (map-class-index #'print-friend 'friend 'birthday - :start (encode-birthday '(10 10 1975)) + :start (encode-date '(10 10 1975)) :end nil) name: Zaid birthdate: (14 8 1976) name: Adriana birthdate: (24 4 1980) @@ -827,27 +827,29 @@ @subsection Using @code{with-transaction} What is @code{with-transaction} really doing for us? It first starts -a new transaction, attempts to execute the body, and if successful -commit the transaction. If anywhere along the way there is a deadlock -with another thread, contention, or an error the transaction is +a new transaction, attempts to execute the body, and commits the +transaction if successful. If anytime during the dynamic extent of +this process there is a conflict with another thread's transaction, an +error, or other non-local transfer of control, the transaction is aborted. If it was aborted due to contention or deadlock, it attempts to retry the transaction a fixed number of times by re-executing the whole body. -And this brings us to two important caveats: nested transactions and -idempotent side-effects. +And this brings us to two important constraints on transaction bodies: +no dynamic nesting and idempotent side-effects. @subsection Nesting Transactions -In general, you want to avoid nesting @code{with-transaction} -statements. Nested transactions are valid for some data stores -(namely Berkeley DB), but typically only a single transaction can be -active at a time. The purpose of a nested transaction in data stores -that provide it, is break a long transaction into chunks. This way if -there is contention on a given subset of variables, only the inner -transaction is restarted while the larger transaction can continue. -When commit their results, those results become part of the outer -transaction until it in turn commits. +In general, you want to avoid nested uses of @code{with-transaction} +statements over multiple functions. Nested transactions are valid for +some data stores (namely Berkeley DB), but typically only a single +transaction can be active at a time. The purpose of a nested +transaction in data stores that support them is to break a long +transaction into subsets. This way if there is contention on a given +subset of variables, only the inner transaction is restarted while the +larger transaction can continue. When the inner transaction commits +its results, those results become part of the outer transaction but +are not written to disk until the outer transaction commits. If you have transaction protected primitive operations (such as @code{deposit} and @code{withdraw}) and you want to perform a group of @@ -922,13 +924,13 @@ (load-transients) (length *transient-objects*)) -(test-list) +(test-list 3) => 3 -(test-list) +(test-list 3) => 5 -(test-list) +(test-list 3) => 4 @end lisp @@ -936,39 +938,40 @@ parameters is atomic if the transaction completes. @lisp -(defun load-transients () +(defun load-transients (n) "This is a better way" (setq *transient-objects* (with-transaction () - (loop for i from 0 upto 3 collect + (loop for i from 0 upto n collect (get-from-root i))))) @end lisp -Of course we would need to use @code{nreverse} if we cared about the -order of instances in @code{*transient-objects*}. The best rule of -thumb is that transaction bodies should be purely functional as above, -except for side effects to the persistent store such as persistent -slot writes, adding to btrees, etc). - -If you do need side effects to lisp memory, such as writes to -transient slots, make sure they are idempotent and that other -processes will not be reading the written values until the transaction +(Of course we would need to use @code{nreverse} if we cared about the +order of instances in @code{*transient-objects*}) + +The best rule-of-thumb is to ensure that transaction bodies are purely +functional as above, except for side effects to persistent objects and +btrees. + +If you really do need to execute side-effects into lisp memory, such +as writes to transient slots, make sure they are idempotent and that +other processes cannot read the written values until the transaction completes. @subsection Transactions and Performance By now transactions almost look like more work than they are worth! -Well there are still some significant benefits to be had. Part of how -transactions are implemented is that they gather together all the -writes that are supposed to made to the database and store them until -the transaction commits, and then writes them atomically. - -The most time-intensive part of persistent operations is flushing -newly written data to disk. Using the default auto-committing -behavior requires a flush for every primitive write operation. This -can become very expensive! Because all the values read or written are -cached in memory until the transaction completes, the number of -flushes can be dramatically reduced. +Fortunately, there are also performance benefits to explicit use of +transactions. Transactions gather together all the writes that are +supposed to made to the database and store them in memory until the +transaction commits, and only then writes them to the disk. + +The most time-intensive component of a transaction is waiting while +flushing newly written data to disk. Using the default +auto-committing behavior requires a disk flush for every primitive +write operation. This is very, very expensive! Because all the +values read or written are cached in memory until the transaction +completes, the number of flushes can be dramatically reduced. But don't take my word for it, run the following statements and see for yourself the visceral impact transactions can have on system @@ -1003,12 +1006,11 @@ When we increase the number of objects within the transaction, the time cost does not go up linearly. This is because the total time to -write a hundred simple objects is still dominated by the final -synchronization step. +write a hundred simple objects is still dominated by the disk writes. These are huge differences in performance! However we cannot have -infinitely sized transactions due to the need to cache values in -working memory. Large operations (such as loading data into a +infinitely sized transactions due to the finite size of the data +store's memory cache. Large operations (such as loading data into a database) need to be split into a sequential set of smaller transactions. When dealing with persistent objects a good rule of thumb is to keep the number of objects touched in a transaction well @@ -1021,11 +1023,11 @@ as they only show up when transactions are interleaved within a larger, multi-threaded application. -In many cases, however, you can ignore transactions. For example, -when you don't have any other concurrent processes running. In this -case all operations are sequential and there is no chance of -conflicts. You would only want to use transactions for write -performance. +In many cases you can simply ignore transactions. For example, when +you don't have any other concurrent processes running. In this case +all operations are sequential and there is no chance of conflicts. +You would only want to use transactions to improve performance on +repeated sets of operations. You can also ignore transactions if your application can guarantee that concurrency won't generate any conflicts. For example, a web app @@ -1064,6 +1066,10 @@ features in the user guide that were not covered in this tutorial. @itemize +@item @strong{Using Multiple Threads and Processes} + What constraints must be accommodated to use Elephant data stores in +multiple threads? What capabilities are there to share data stores +among multiple processes or machines? @item @strong{Class Heirarchies and Queries} There are some subtle issues to take into account when querying persistent classes. For example, how do you query a base class of @@ -1072,34 +1078,26 @@ @item @strong{Derived Class Indices} You can create your own indices for classes that are arbitrary lisp functions of the persistent object. +@item @strong{Dynamic Class Index Management} + It is possible to add and remove indexes from classes at runtime. @item @strong{Class Definition/Database Conflict Resolution} When you startup lisp, there are potential conflicts between the class definition and the indexing records in the database. There are some constraints to account for and some facilities to manage how slots, class indices and -@item @strong{Dynamic Class Index Management} - It is possible to add and remove indexes from classes at runtime. +@item @strong{Indexed BTrees} + Indexed BTrees are just like BTrees, except it is possible to add +indexes which are BTrees who's values are primary keys in the parent +@code{indexed-btree}. This allows for multiple ordering and groupings +of the values of a BTree. @item @strong{BTree Cursors} If you need to do more than iterate over a collection, or you need to delete elements of the collection as you iterate cursors are an important data structure. They implement a variety of operators for moving backward and forward over a btree, including ranged operations and iterating of duplicate or unique values. -@item @strong{Indexed BTrees} - Indexed BTrees are just like BTrees, except it is possible to add -indexes which are BTrees who's values are primary keys in the parent -@code{indexed-btree}. This allows for multiple ordering and groupings -of the values of a BTree. @item @strong{Using the Map Operators} Mapping operators can be very efficient if properly utilized. -@item @strong{Handling Errors and Conditions} - There are a variety of errors that can occur in Elephant that need -to be dealt with by applications. -@item @strong{Deadlock Detection in Berkeley DB} - Berkeley DB requires an external process to detect deadlock -conditions among transactions. The :deadlock-detect keyword argument -to open-store for Berkeley DB specs will launch this process on most -lisps. @item @strong{Using Multiple Stores} Multiple store controllers can be open simultaneously. However it does make the code more complex and you need to be careful about how @@ -1108,10 +1106,14 @@ You can implement your own version of with-transaction using the underlying controller methods for starting, aborting and committing transactions. You had better know what you are doing, however! -@item @strong{Using Multiple Threads and Processes} - What constraints must be accommodated to use Elephant data stores in -multiple threads? What capabilities are there to share data stores -among multiple processes or machines? +@item @strong{Handling Errors and Conditions} + There are a variety of errors that can occur in Elephant that need +to be dealt with by applications. +@item @strong{Deadlock Detection in Berkeley DB} + Berkeley DB requires an external process to detect deadlock +conditions among transactions. The :deadlock-detect keyword argument +to open-store for Berkeley DB specs will launch this process on most +lisps. @end itemize Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/02 00:51:06 1.8 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/04 15:28:28 1.9 @@ -8,9 +8,9 @@ @menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. -* Persistent objects:: All the dirt on persistent objects. +* Persistent Classes and Objects:: All the dirt on persistent objects. * Class Indices:: In-depth discussion about indexing persistent indices. -* Querying persistent instances:: Retrieving instances of classes. +@c * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. @@ -21,67 +21,248 @@ * Repository Migration and Upgrade:: How to move objects from one repository to another. * Garbage Collection:: How to recover storage and OIDs in long-lived repositories. * Performance Tuning:: How to get the most from Elephant. +* Berkeley DB Data Store:: Commands and concerns specific to the :BDB data store +* CL-SQL Data Store:: Commands and concerns specific to the :CLSQL data store @end menu @node The Store Controller @comment node-name, next, previous, up @section The Store Controller -What is @code{open-store} doing? It creates a @code{store-controller} -object, and sets the special @code{*store-controller*} to point to it. -The store controller holds the handles to the database environment and -tables, and some other bookkeeping. If for some reason you need to -run recovery on the database (see sleepycat docs) you can specify that -with the @code{:recover} and @code{:recover-fatal} keys. +An instance of the @code{store-controller} class mediates interactions +between Lisp and a data store. All elephant operations are performed +in the context of a store controller. To be more specific, a data +store provides a subclass of @code{store-controller} specialized to +that data store. Typically this object contains pointers to the disk +files, foreign memory regions and any other necessary bookkeeping +information to support Elephant operations such as slot writes and +btree operations. The store also contains the root objects and other +bookeeping common to all data stores. + +To obtain a @code{store-controller} object, call the function +@code{open-store} with a store controller specification. The current +data store specification formats are: + +@itemize +@item Berkeley DB: '(:BDB "/path/to/datastore/directory/") +@item CL-SQL: '(:CLSQL (<sql-db-name> <sql-connect-command>)) +@end itemize + +Valid CLSQL database tags for @code{<sql-db-name>} are +@code{:SQLITE} and @code{:POSTGRESQL}. The @code{<sql-connect-command>} is +what you would pass to CLSQL's @code{connect} command. + +The open store function uses the first symbol in the specification +(i.e. :BDB or :CLSQL) to dispatch instance creation to the specified +data store which returns a specialized instance of +@code{store-controller}. @code{open-store} then initializes the store +using an internal call to @code{open-controller}. + +The final step of @code{open-store} is to set the global variable +@code{*store-controller*}. This special variable is used as a default +value in the optional or keyword arguments to number of operations +such as: + +@itemize +@item @code{make-instance} for persistent objects +@item @code{get-from-root} and @code{add-to-root} for accessing a store's root +@item @code{make-btree} for creating persistent index instances +@end itemize + +Each of these functions also accepts an explicit store controller +argument for use in multiple store environments. Normal applications +should only be aware that this global parameter is used. For further +discussion of @code{*store-controller*} @pxref{Multi-repository Operation}. + +Additionally, @code{open-store} accepts data store specific keyword +arguments. For example, you can force recovery to be run on Berkeley +DB data stores: + +@lisp +(open-store *my-spec* :recover t) +@end lisp + +The data store sections of the user guide (@ref{Berkeley DB Data +Store} and @ref{CL-SQL Data Store}) list all the data-store specific +options to various elephant functions. + +When you finish your application, @code{close-store} will close the +store controller. Failing to do this properly may lead to a need to +run recovery on the data store during the next session. Again, see +the relevant data store sections for more details. -To create one by hand one can do, +@node Serialization details +@comment node-name, next, previous, up +@section Serialization details -@lisp -* (setq *store-controller* (make-instance 'store-controller :path "testdb")) -=> #<STORE-CONTROLLER @{49252F75@}> +This section captures the details of how various types of objects are +serialized and some considerations to keep in mind when storing lisp +objects. + +The high level factors that you need to keep in mind are: + +@itemize +@item Circular References: +The serializer properly handles circular references to/from objects +such as cons cells, standard objects, arrays, etc. It accomplishes +this by assigning an ID to any non-atomic object and keeping a mapping +between previously serialized objects and these ids. +@end itemize + +Here is an introduction to + +@itemize +@item +@end itemize + +We will also review and add to the considerations outlined in the tutorial: -* (open-controller *store-controller*) -=> #<STORE-CONTROLLER @{49252F75@}> +@enumerate + + +@item @strong{Lisp identity can't be preserved}. Since this is a store which +persists across invocations of Lisp, this probably doesn't even make +sense. However if you get an object from the index, store it to a +lisp variable, then get it again - they will not be eq: + +@lisp +(setq foo (cons nil nil)) +=> (NIL) +(add-to-root "my key" foo) +=> (NIL) +(add-to-root "my other key" foo) +=> (NIL) +(eq (get-from-root "my key") + (get-from-root "my other key")) +=> NIL @end lisp -but +@item @strong{Nested aggregates are stored in one buffer}. +If you store an set of objects in a hash table you try to store a hash +table, all of those objects will get stored in one large binary buffer +with the hash keys. This is true for all other aggregates that can +store type T (cons, array, standard object, etc). + +@item @strong{Circular References}. +The serializer properly handles circular references to/from objects +such as cons cells, standard objects, arrays, etc. It accomplishes +this by assigning an ID to any non-atomic object and keeping a mapping +between previously serialized objects and these ids. + +@item @strong{Mutated substructure does not persist}. @lisp -* (open-store "testdb")) +(setf (car foo) T) +=> T +(get-from-root "my key") +=> (NIL) @end lisp -is the preferred mechanism. +This will affect all aggregate types: objects, conses, hash-tables, et +cetera. (You can of course manually re-store the cons.) In this sense +elephant does not automatically provide persistent collections. If you +want to persist every access, you have to use BTrees (@pxref{Using BTrees}). -This opens the environment and database. The @code{persistent-*} objects -reference the @code{*store-controller*} special. (This is in part because -slot accessors can't take additional arguments.) If for some reason -you want to operate on 2 store controllers, you'll have to do that by -flipping the @code{*store-controller*} special. - -@code{close-store} closes the store controller. Alternatively -@code{close-controller} can be called on a handle. Don't forget to do -this or else you may need to run recovery later. There is a -@code{with-open-controller} macro. Opening and closing a controller -is very expensive. +@item @strong{Storage limitations}. +The serializer writes sequentially into a foreign memory byte array +before passing that array to a given data store's API. There are +practical limits to the size of this buffer. Moreoever, in most data +stores there is a practical limit to the size of a transaction. +Either of these considerations should encourage you to plan to limit +the size of objects that you serialize to disk. A good rule of thumb +is to stay under a megabyte. -@node Serialization details -@comment node-name, next, previous, up -@section Serialization details +@item @strong{Serialization and deserialization can be costly}. While +serialization is pretty fast, but it is still expensive to store large +objects wholesale. Also, since object identity is impossible to +maintain, deserialization must re-cons or re-allocate the entire +object every time increasing the number of GCs the system does. This +eager allocation is contrary to how most people want to use a +database: one of the reasons to use a database is if your objects +can't fit into main memory all at once. + +@item @strong{Merge-conflicts in heavily multi-process/threaded situations}. +This is the common read-modify-write problem in all databases. We will talk +more about this in the @ref{Transactions} section. + +@end enumerate -Empty. -@node Persistent objects +@node Persistent Classes and Objects @comment node-name, next, previous, up -@section Persistent Objects +@section Persistent Classes and Objects + +Persistent classes are Elephant's answer to the limitations of +ordinary lisp object serialization, namely support for persistent +references. Any persistent object, when serialized, only serializes a +reference to the object and not the whole object. For example you can +serialize a node in the graph of persistent objects without worrying +about serializing the entire graph. + +@subsection{Persistent Class Definition} + +Other than specifying the metaclass or using @code{defpclass} the only +important differences in the @code{defclass} form is the specification +of a slot storage policy. Slot storage policy can be specified by a +boolean argument to the slot initargs @code{:persistent} or +@code{:transient}. Slots are @code{:persistent} by default + +@lisp +(defclass my-pclass () + ((pslot1 :accessor pslot1 :initarg :pslot1 :initform 'one) + (pslot2 :accessor pslot2 :initarg :pslot2 :initform 'two :persistent t) + (tslot1 :accessor tslot3 :initarg :tslot3 :initform nil :transient t)) + (:metaclass persistent-metaclass)) +@end lisp + +The :index options to persistent classes are discussed in persistent +indices. + +Slot storage class implications are straightforward. Persistent slot +writes are durably stored to disk in an automatic or encompassing +transaction. Transient slots are initialized on instance creation +according to initforms or to initargs. They are never stored to nor +loaded from the database. + + +@subsection{Instance Creation} + +Persistent objects are instances of the persistent classes defined +above. All persistent objects inherit from the class +@code{persistent} and share two properties: a unique ID and a +reference to the specification of the @code{store-controller} in which +they reside. This is ensured by the instance creation protocol +implemented by @code{persistent-metaclass}. + +Instances are created as normal, with a call to make-instance and +appropriate initargs. + + +The two properties of @code{persistent} can be specified explicitly +during instance creation: + +@lisp +(make-instance 'my-pclass :from-oid 100 :sc *store-controller*) +@end lisp + +These three elements, class, oid and store controller is all that is +needed to create a new instance + +If you do make an instance with a specified OID which already exists +in the database, @code{initargs} to @code{make-instanc} take +precedence over values in the database, which take precedences over +any @code{initforms} defined in the class. + +* Default store controller & instance creation +* What happens to persistent objects when store-controller is closed? + + -Finally, if you for some reason make an instance with a specified OID -which already exists in the database, @code{initargs} take precedence -over values in the database, which take precedences over -@code{initforms}. - -Also currently there is a bug where -@code{initforms} are always evaluated, so beware. -(What is the current model here?) +:: User-defined persistent objects + +* slot types +* caching +* slot access protocol Readers, writers, accessors, and @code{slot-value-using-class} are employed in redirecting slot accesses to the database, so override @@ -90,6 +271,20 @@ the specification to work properly with persistent slots. However the proper behavior has been verified on SBCL, Allegro and Lispworks. +:: Initialization + +Also currently there is a bug where @code{initforms} are always +evaluated, so beware. (What is the current model here?) + +:: Class Redefinition and Evolution + +* What happens when you redefine a class online? +* Drop & add slots? Change slot status? +* What if you connect to an old database with a new class specification? + (ref to class indicies behavior) + +:: Storage and Performance Considerations + @node Class Indices @comment node-name, next, previous, up @section Class Indices @@ -137,112 +332,111 @@ somewhat user customizable; documentation for this exists in the source file referenced above. -@node Querying persistent instances -@comment node-name, next, previous, up -@section Querying persistent instances - -A SQL select-like interface is in the works, but for now queries are -limited to manual mapping over class instances or doing small queries -with @code{get-instances-*} functions. One advantage of this is that -it is easy to estimate the performance costs of your queries and to -choose standard and derived indices that give you the ordering and -performance you want. - -There is, however, a quick and dirty query API example that is not -officially supported in the release but is intended to invite comment. -This is an example of a full query system that would automatically -perform joins, use the appropriate indices and perhaps even adaptively -suggest or add indices to facilitate better performance on common -queries. - -There are two functions @ref{Function elephant:get-query-instances,,,includes/fun-elephant-get-query-instance } -and @ref{Function elephant:map-class-query,,,includes/fun-elephant-map-class-query} which accept a set of -constraints instead of the familiar value or range arguments. - -We'll use the classes @code{person} and @code{department} to -illustrate how to perform queries over a set of objects that may be -constrainted by their relationships to other objects. - -@lisp -(defpclass person () - ((name :initarg :name :index t) - (salary :initarg :salary :index t) - (department :initarg :dept))) - -(defmethod print-object ((p person) stream) - (format stream "#<PERS: ~A>" (slot-value p 'name))) - -(defun print-name (inst) - (format t "Name: ~A~%" (slot-value inst 'name))) - -(defpclass department () - ((name :initarg :name) - (manager :initarg :manager))) - -(defmethod print-object ((d department) stream) - (format stream "#<DEPT ~A, mgr = ~A>" - (slot-value d 'name) - (when (slot-boundp d 'manager) - (slot-value (slot-value d 'manager) 'name)))) -@end lisp - -Here we have a simple employee database with managers (also of type -person) and departments. This simple system will provide fodder for -some reasonably complex constraints. Let's create a few departments. - -@lisp -(setf marketing (make-instance 'department :name "Marketing")) -(setf engineering (make-instance 'department :name "Engineering")) -(setf sales (make-instance 'department :name "Sales")) -@end lisp - -And manager @code{people} for the departments. - -@lisp -(make-instance 'person :name "George" :salary 140000 :department marketing) -(setf (slot-value marketing 'manager) *) - -(make-instance 'person :name "Sally" :salary 140000 :department engineering) -(setf (slot-value engineering 'manager) *) - -(make-instance 'person :name "Freddy" :salary 180000 :department sales) -(setf (slot-value sales 'manager) *) -@end lisp - [156 lines skipped]

1 0

[elephant-cvs] CVS elephant/doc
by rread 02 Apr '07

02 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv2667 Modified Files: scenarios.texinfo Log Message: Added a blurb about Konsenti. --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 1.1 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/02 13:09:46 1.2 @@ -23,3 +23,21 @@ Object-oriented data storage, large graph traversals +@node Commercial Applications + +Elephant is used by Konsenti(tm), a for-profit company of Robert L. Read, one of the maintainers of Elephant. It can be visited at +@uref{http://konsenti.com}. Konsenti uses the Data Collection Management (DCM) package, which can be +found in the contrib directory, under user rread. DCM provides prevalence-style in-memory write-through caching. +The most enjoyable feature about Elephant for this project is that new Business Layer objects can be created without having to +deal with an Object-Relational Mapping, which has allowed extremely rapid development. All Business objects are persisted via +a director in DCM (which sits on top of Elephant.) Many of these business objects are in fact finite state machines decorated +with functions. The functions are represented by their lambda expression stored in slots on the business objects. A complete +Message Factory and double-entry accounting system are also implemented as DCM objects. Binary objects, such as uploaded +PDFs that can be attached to objects as comments, are treated as simple objects and stored directly in Elephant. Konsenti +is completely based on utf-8, and unicode characters outside of the ISO-9959-1 character set are routinely stored in +Elephant. Konsenti uses Postgres as a backend; but Elephant makes it so easy to migrate between repositories that we +could change this decision at any time. + + + +

1 0

[elephant-cvs] CVS elephant/doc
by rread 02 Apr '07

02 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv13016 Modified Files: data-store-reference.texinfo installation.texinfo intro.texinfo make-ref.lisp tutorial.texinfo user-guide.texinfo Log Message: Sprucing up Chapter 3. --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/02 00:51:06 1.6 @@ -138,10 +138,8 @@ @include includes/class-elephant-indexed-btree.texinfo To create the backend-appropriate type of btree, the backend -implements these methods aginst their store-controller. - -@include includes/fun-elephant-build-btree.texinfo -@include includes/fun-elephant-build-indexed-btree.texinfo +implements this method (and possibly related methods) aginst their store-controller. +@include includes/fun-elephant-backend-build-btree.texinfo Most of the user-visible operations over BTrees must be implemented. Class indexing functions such as @code{map-class} and --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/02 00:51:06 1.7 @@ -66,7 +66,9 @@ test other combinations, but practically these configurations will be the most stable and reliable. Elephant is becoming quite stable in general, so don't be afriad to try an unemphasized combination - -chances are it is just a little more work to bring it up. +chances are it is just a little more work to bring it up. In particular, +Elephant can probably work with MySQL or Oracle with just a little work, +but nobody has asked for this yet. @subsection Library dependencies @@ -223,12 +225,53 @@ @comment node-name, next, previous, up @section Berkeley DB Data Store +The Berkeley DB Data Store started out as a very simple data dictionary in the +Berkeley Unix operating system. There are many ``Xdb'' systems that use the +same API, or a similarly one. A commercial version of the BDB was provided by +Sleepycat Software, and this product has since been purchased by Oracle corporation, +and is currently distributed under a similar licensing arrangement, with commercial +support also available. Please follow the download and installation procedures +defined here: +http://www.oracle.com/technology/products/berkeley-db/db/index.html + +Elephant only works with version 4.5 of BerkeleyDB. @node Berkeley DB Example @comment node-name, next, previous, up @section Setting up Berkeley DB +Beyond ensuring that the file ``my-config.sexp'' points correctly +to your BDB installation, nothing else should be required to configure +the example that uses a local ``testdb'' directory as a dabase (under ``tests'') +in the top-level Elephant directory. + +On one Fedora based system, the ``my-config.sexp'' file looked like this: + +@lisp +((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:pthread-lib . nil) + (:clsql-lib . "/usr/local/share/common-lisp/") + (:compiler . :gcc)) +@end lisp + +The @ref{Test Suites} give a nice example of using BDB by running the test using +the specification: +@lisp +'(:BDB "/home/me/db/testdb/")) +@end lisp + +Once you start working on an application, you will want to change this path +to a directory that make sense for you application, and use that as +the specification passed to +@lisp open-store +@end lisp +in your application. + + @node CL-SQL Data Store @comment node-name, next, previous, up @section CL-SQL Data Store @@ -237,7 +280,7 @@ the original Elephant system has been experimenetally extended to support the use of relational database management systems as the implementation of the persistent store. This relies on Kevin Rosenberg's -CL-SQL interface to relational systems. +CL-SQL interface to a large number of relational systems. Although the BerkeleyDB system is an ideal object store for LISP objects, one might prefer the licensing of a different system. For example, at @@ -246,13 +289,10 @@ http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source. -Neither the PostGres DBMS nor SQLite 3 has any such restriction. Elephant itself is released -under the GPL. It is somewhat debatable if the GPL allows one to construct -to construct a non-open-source web application but the preponderance of -opinion appears to be that it does. Thefore using Elephant and the other -GPLed software that it depends upon allows one to host a a non open-source -web application. This might be a reason to use Elephant on PostGres of SQLite rather -than Elephant on BerkeleyDB. +Neither the PostGres DBMS nor SQLite 3, nor Elephant itself, imposes +any such restriction. (Elephant is released under the LLGPL +(http://opensource.franz.com/preamble.html .) Older versions were +released under the GPL.) Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application @@ -260,27 +300,22 @@ the tools associated with those systems, etc. The SQL back-end extention of Elephant provides a function for migrating -data seamlessly between repositories. That is, one can quite easily move +data seamlessly between repositories. One can quite easily move data from a BerkeleyDB repository to a PostGres repository, and vice versa. -In fact, one of the most important aspects of the extention is that it -makes Elephant a multi-repository system, rather than a single repository -system, as addition to allowing different implementation strategies for -those repositories. This offers at least the possiblity than once -can develop using one backend, for example BerkeleyDB, and then later -move to MySQL. +This offers at least the possiblity than once can develop using one backend, +for example BerkeleyDB, and then later move to Postgres, or vice versa. +One could even operate simultaneously out of multiple repositories, if +there were a good reason to do so. At the time of this writing, the basic strategy for the SQL implementation is quite simple. The same serializer used for the Sleepycat implementation is employed, the byte-string is base64 encoded, and placed in a single table which is managed by Elephant. -As of Elephant 0.3, Elephant has been tested to work with both Postgres, and -SQLite 3, thanks do Dan Knapp. - -As far as is known at this writing, all functionality except nested transaction -support and cursor-puts supported by the BerkeleyDB backend is supported by the -CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested -well for the CL-SQL based system. +All functionality except nested transaction support and cursor-puts +supported by the BerkeleyDB backend is supported by the CL-SQL +back-end. Concurrency and transaction atomicity have not been stress +tested well for the CL-SQL based system. Additionally, it is NOT the case that the Elephant system currently provides transaction support across multiple repositories; it provides the transaction @@ -290,13 +325,7 @@ The PostGres backend is as currently employed is about 5 times slower than the BerkeleyDB backend. This could probably change with continued development. -CL-SQL supports a lot of DBMS systems, but only PostGres has been tested. - -The SQL back-end extention has only been tested under SBCL 0.8.18. - -The SQL back-end is as easy to use as the BerkeleyDB back-end. However, -the multi-repository version somewhat complicates the underlying -persistent object management. +CL-SQL supports a lot of DBMS systems, but only PostGres and SqlLite 3 have been tested. @node CL-SQL Example @comment node-name, next, previous, up @@ -306,6 +335,9 @@ @enumerate @item Install postgres and make sure postmaster is running. +Postgres may be installed on your system; you may be able to use a package manager to +install it, or you can install it quite easily from the PostgresSQL site directly +(http://www.postgresql.org/). @item Create a database called ``test'' and set its permissions to be reached by whatever connection specification you intend to use. The @@ -377,34 +409,39 @@ The text of this file is included here to give the casual reader an idea of how elepant test can be run in general: @lisp -;; This file is an example of how to perform the -;; migration tests. You will have to modify it -;; slightly depending on the systems that want to test... -;; You can test migration even between two BDB respositories if you wish +;; If you are only using one back-end, you may prefer: +;; SQLDB-test.lisp or BerkeleyDB-tests.lisp (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) -(asdf:operate 'asdf:load-op :clsql-postgresql-socket) (asdf:operate 'asdf:load-op :ele-bdb) +(asdf:operate 'asdf:load-op :ele-sqlite3) + (asdf:operate 'asdf:load-op :elephant-tests) -;; For sqlite-3.. -;; (asdf:operate 'asdf:load-op :ele-sqlite3) (in-package "ELEPHANT-TESTS") -;; The primary and secondary test-paths are -;; use for the migration tests. - -;; This this configuration for testing between BDB and SQL.... -(setq *test-path-primary* *testpg-path*) -;; (setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path*) - -;; This this configuration for testing from one BDB repository to another... -(setq *test-path-primary* *testdb-path*) -;; (setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path2*) +;; Test Postgres backend +(setq *default-spec* *testpg-spec*) +(do-backend-tests) + +;; Test BDB backend +(setq *default-spec* *testbdb-spec*) +(do-backend-tests) + +;; Test SQLite 3 +(setq *default-spec* *testsqlite3-spec*) +(do-backend-tests) + +;; Test a Migration of data from BDB to postgres +(do-migration-tests *testbdb-spec* *testpg-spec*) + +;; An example usage. +(open-store *testpg-spec*) +(add-to-root "x1" "y1") +(get-from-root "x1") -(do-migrate-test-spec *test-path-primary*) +(add-to-root "x2" '(a 4 "spud")) +(get-from-root "x2") @end lisp The appropriate test should execute for you with no errors. @@ -440,5 +477,29 @@ @comment node-name, next, previous, up @section Documentation +If you are getting the documentation as a released tar file, you will probably find +the documenation in .html or .pdf form in the release, or can find it at the +Elephant website. + +If you want to compile the documentation youself, for example, if you can +think of a way to improve this manual, then you will do something similar +to this in a shell or command-line prompt: +@code +cd doc +make +make pdf +@end code + +This process will populate the ``./includes'' directory with references +automatically extracted from the list code. It will then compile the +texinfo documenation source into both HTML and a PDF which will be left +in the ``doc/elephant'' directory. + +Don't edit anything in the ``doc/elephant'' directory or the +``doc/includes'' directories, as everything in these directories is +generated. Instead, edit the ``.texinfo'' files in the doc directory. + + + --- /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/03/24 12:16:02 1.5 +++ /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/04/02 00:51:06 1.6 @@ -28,7 +28,8 @@ Elephant has been extended to provide support for multiple backends, specifically a relational database backend based on CL-SQL which has -been tested with Postgres and SQLite 3. It supports, with some care, +been tested with Postgres and SQLite 3, and probably support +other relational systems easily. It supports, with some care, multi-repository operation and enables convenient migration of data between repositories. --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/30 23:36:52 1.6 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/02 00:51:06 1.7 @@ -30,7 +30,8 @@ (make-instance 'elephant::persistent-collection :sc sc :from-oid 10) (make-instance 'elephant::secondary-cursor) (make-instance 'elephant::indexed-btree :sc sc :from-oid 10) - (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" +;; (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" + (sb-texinfo:generate-includes #p"." (find-package :elephant) (find-package :elephant-backend) (find-package :elephant-memutil) --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 20:22:24 1.12 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/02 00:51:06 1.13 @@ -32,6 +32,11 @@ Elephant provides a persistent index which maintains an ordered collection of lisp values or persistent object references. +The use of persistent objects makes coding concise, convenient, and +powerful, and makes persistence almost invisible to the programmer. +However, Elephant also allows the same basic data dictionary of +key/value retrieval that BerkeleyDB provides. + When someone says "database," most people think of SQL Relational Data Base Management Systems (e.g. Oracle, Postgresql, MySql). Those systems store data in statically typed tables with unique shared @@ -202,9 +207,11 @@ values and objects can be stored: numbers (except for complexes), symbols, strings, nil, characters, pathnames, conses, hash-tables, arrays, CLOS objects and structs. Nested and circular things are -allowed. You can store basically anything except lambdas, closures, -class objects, packages and streams. (These may eventually get -supported too.) +allowed. Nested and circular things are allowed. You can store +basically anything except compiled functions, closures, class objects, +packages and streams. Functions can be stored as uncompiled lambda +expressions. (Compiled functions and other kinds of objects may +eventually get supported too.) Elephant needs to use a representation of data that is independant of a specific lisp or data store. Therefore all lisp values that are --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:56:19 1.7 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/02 00:51:06 1.8 @@ -155,8 +155,8 @@ suggest or add indices to facilitate better performance on common queries. -There are two functions @ref{Function elephant:get-query-instances} -and @ref{Function elephant:map-class-query} which accept a set of +There are two functions @ref{Function elephant:get-query-instances,,,includes/fun-elephant-get-query-instance } +and @ref{Function elephant:map-class-query,,,includes/fun-elephant-map-class-query} which accept a set of constraints instead of the familiar value or range arguments. We'll use the classes @code{person} and @code{department} to

1 0

[elephant-cvs] CVS elephant/tests
by rread 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv27892 Modified Files: RUNTEST.lisp Log Message: Modernization of this file. --- /project/elephant/cvsroot/elephant/tests/RUNTEST.lisp 2006/02/05 23:46:41 1.1 +++ /project/elephant/cvsroot/elephant/tests/RUNTEST.lisp 2007/04/01 23:28:35 1.2 @@ -4,60 +4,44 @@ ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; -;;; Copyright (c) 2005,2006 by Robert L. Read +;;; Copyright (c) 2005,2006,2007 by Robert L. Read ;;; <rread(a)common-lisp.net> ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html) also known as the LLGPL. - - -;; This file is now obsolete... -;; Please use SQLDB-test.lisp or BerkeleyDB-tests.lisp - +;; If you are only using one back-end, you may prefer: +;; SQLDB-test.lisp or BerkeleyDB-tests.lisp (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) -(asdf:oos 'asdf:load-op :clsql-postgresql-socket) (asdf:operate 'asdf:load-op :ele-bdb) -(asdf:operate 'asdf:load-op :elephant-tests) - (asdf:operate 'asdf:load-op :ele-sqlite3) +(asdf:operate 'asdf:load-op :elephant-tests) (in-package "ELEPHANT-TESTS") -(do-all-tests) -(do-all-tests-spec *testpg-path*) -(do-migrate-test-spec *testpg-path*) -(do-all-tests-spec *testdb-path*) -(do-all-tests-spec *testsqlite3-path*) -;; The primary and secondary test-paths are -;; use for the migration tests. -(setq *test-path-primary* *testpg-path*) -(setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path*) +;; Test Postgres backend +(setq *default-spec* *testpg-spec*) +(do-backend-tests) + +;; Test BDB backend +(setq *default-spec* *testbdb-spec*) +(do-backend-tests) + +;; Test SQLite 3 +(setq *default-spec* *testsqlite3-spec*) +(do-backend-tests) -(setq *test-path-primary* *testdb-path*) -(setq *test-path-secondary* nil) +;; Test a Migration of data from BDB to postgres +(do-migration-tests *testbdb-spec* *testpg-spec*) -(do-all-tests-spec *test-path-primary*) - - -(use-package :sb-profile) - -(profile "CLSQL") -(profile "POSTGRESQL-SOCKET") -(profile "ELEPHANT") - -(use-package "SB-PROFILE") - -(open-store *testpg-path*) -(open-store *testdb-path*) +;; An example usage. +(open-store *testpg-spec*) (add-to-root "x1" "y1") (get-from-root "x1") - (add-to-root "x2" '(a 4 "spud")) (get-from-root "x2")

1 0