April 2007 - elephant-cvs@common-lisp.net

[elephant-cvs] CVS elephant/doc
by rread 02 Apr '07

02 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv13016 Modified Files: data-store-reference.texinfo installation.texinfo intro.texinfo make-ref.lisp tutorial.texinfo user-guide.texinfo Log Message: Sprucing up Chapter 3. --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/02 00:51:06 1.6 @@ -138,10 +138,8 @@ @include includes/class-elephant-indexed-btree.texinfo To create the backend-appropriate type of btree, the backend -implements these methods aginst their store-controller. - -@include includes/fun-elephant-build-btree.texinfo -@include includes/fun-elephant-build-indexed-btree.texinfo +implements this method (and possibly related methods) aginst their store-controller. +@include includes/fun-elephant-backend-build-btree.texinfo Most of the user-visible operations over BTrees must be implemented. Class indexing functions such as @code{map-class} and --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/02 00:51:06 1.7 @@ -66,7 +66,9 @@ test other combinations, but practically these configurations will be the most stable and reliable. Elephant is becoming quite stable in general, so don't be afriad to try an unemphasized combination - -chances are it is just a little more work to bring it up. +chances are it is just a little more work to bring it up. In particular, +Elephant can probably work with MySQL or Oracle with just a little work, +but nobody has asked for this yet. @subsection Library dependencies @@ -223,12 +225,53 @@ @comment node-name, next, previous, up @section Berkeley DB Data Store +The Berkeley DB Data Store started out as a very simple data dictionary in the +Berkeley Unix operating system. There are many ``Xdb'' systems that use the +same API, or a similarly one. A commercial version of the BDB was provided by +Sleepycat Software, and this product has since been purchased by Oracle corporation, +and is currently distributed under a similar licensing arrangement, with commercial +support also available. Please follow the download and installation procedures +defined here: +http://www.oracle.com/technology/products/berkeley-db/db/index.html + +Elephant only works with version 4.5 of BerkeleyDB. @node Berkeley DB Example @comment node-name, next, previous, up @section Setting up Berkeley DB +Beyond ensuring that the file ``my-config.sexp'' points correctly +to your BDB installation, nothing else should be required to configure +the example that uses a local ``testdb'' directory as a dabase (under ``tests'') +in the top-level Elephant directory. + +On one Fedora based system, the ``my-config.sexp'' file looked like this: + +@lisp +((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:pthread-lib . nil) + (:clsql-lib . "/usr/local/share/common-lisp/") + (:compiler . :gcc)) +@end lisp + +The @ref{Test Suites} give a nice example of using BDB by running the test using +the specification: +@lisp +'(:BDB "/home/me/db/testdb/")) +@end lisp + +Once you start working on an application, you will want to change this path +to a directory that make sense for you application, and use that as +the specification passed to +@lisp open-store +@end lisp +in your application. + + @node CL-SQL Data Store @comment node-name, next, previous, up @section CL-SQL Data Store @@ -237,7 +280,7 @@ the original Elephant system has been experimenetally extended to support the use of relational database management systems as the implementation of the persistent store. This relies on Kevin Rosenberg's -CL-SQL interface to relational systems. +CL-SQL interface to a large number of relational systems. Although the BerkeleyDB system is an ideal object store for LISP objects, one might prefer the licensing of a different system. For example, at @@ -246,13 +289,10 @@ http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source. -Neither the PostGres DBMS nor SQLite 3 has any such restriction. Elephant itself is released -under the GPL. It is somewhat debatable if the GPL allows one to construct -to construct a non-open-source web application but the preponderance of -opinion appears to be that it does. Thefore using Elephant and the other -GPLed software that it depends upon allows one to host a a non open-source -web application. This might be a reason to use Elephant on PostGres of SQLite rather -than Elephant on BerkeleyDB. +Neither the PostGres DBMS nor SQLite 3, nor Elephant itself, imposes +any such restriction. (Elephant is released under the LLGPL +(http://opensource.franz.com/preamble.html .) Older versions were +released under the GPL.) Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application @@ -260,27 +300,22 @@ the tools associated with those systems, etc. The SQL back-end extention of Elephant provides a function for migrating -data seamlessly between repositories. That is, one can quite easily move +data seamlessly between repositories. One can quite easily move data from a BerkeleyDB repository to a PostGres repository, and vice versa. -In fact, one of the most important aspects of the extention is that it -makes Elephant a multi-repository system, rather than a single repository -system, as addition to allowing different implementation strategies for -those repositories. This offers at least the possiblity than once -can develop using one backend, for example BerkeleyDB, and then later -move to MySQL. +This offers at least the possiblity than once can develop using one backend, +for example BerkeleyDB, and then later move to Postgres, or vice versa. +One could even operate simultaneously out of multiple repositories, if +there were a good reason to do so. At the time of this writing, the basic strategy for the SQL implementation is quite simple. The same serializer used for the Sleepycat implementation is employed, the byte-string is base64 encoded, and placed in a single table which is managed by Elephant. -As of Elephant 0.3, Elephant has been tested to work with both Postgres, and -SQLite 3, thanks do Dan Knapp. - -As far as is known at this writing, all functionality except nested transaction -support and cursor-puts supported by the BerkeleyDB backend is supported by the -CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested -well for the CL-SQL based system. +All functionality except nested transaction support and cursor-puts +supported by the BerkeleyDB backend is supported by the CL-SQL +back-end. Concurrency and transaction atomicity have not been stress +tested well for the CL-SQL based system. Additionally, it is NOT the case that the Elephant system currently provides transaction support across multiple repositories; it provides the transaction @@ -290,13 +325,7 @@ The PostGres backend is as currently employed is about 5 times slower than the BerkeleyDB backend. This could probably change with continued development. -CL-SQL supports a lot of DBMS systems, but only PostGres has been tested. - -The SQL back-end extention has only been tested under SBCL 0.8.18. - -The SQL back-end is as easy to use as the BerkeleyDB back-end. However, -the multi-repository version somewhat complicates the underlying -persistent object management. +CL-SQL supports a lot of DBMS systems, but only PostGres and SqlLite 3 have been tested. @node CL-SQL Example @comment node-name, next, previous, up @@ -306,6 +335,9 @@ @enumerate @item Install postgres and make sure postmaster is running. +Postgres may be installed on your system; you may be able to use a package manager to +install it, or you can install it quite easily from the PostgresSQL site directly +(http://www.postgresql.org/). @item Create a database called ``test'' and set its permissions to be reached by whatever connection specification you intend to use. The @@ -377,34 +409,39 @@ The text of this file is included here to give the casual reader an idea of how elepant test can be run in general: @lisp -;; This file is an example of how to perform the -;; migration tests. You will have to modify it -;; slightly depending on the systems that want to test... -;; You can test migration even between two BDB respositories if you wish +;; If you are only using one back-end, you may prefer: +;; SQLDB-test.lisp or BerkeleyDB-tests.lisp (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) -(asdf:operate 'asdf:load-op :clsql-postgresql-socket) (asdf:operate 'asdf:load-op :ele-bdb) +(asdf:operate 'asdf:load-op :ele-sqlite3) + (asdf:operate 'asdf:load-op :elephant-tests) -;; For sqlite-3.. -;; (asdf:operate 'asdf:load-op :ele-sqlite3) (in-package "ELEPHANT-TESTS") -;; The primary and secondary test-paths are -;; use for the migration tests. - -;; This this configuration for testing between BDB and SQL.... -(setq *test-path-primary* *testpg-path*) -;; (setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path*) - -;; This this configuration for testing from one BDB repository to another... -(setq *test-path-primary* *testdb-path*) -;; (setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path2*) +;; Test Postgres backend +(setq *default-spec* *testpg-spec*) +(do-backend-tests) + +;; Test BDB backend +(setq *default-spec* *testbdb-spec*) +(do-backend-tests) + +;; Test SQLite 3 +(setq *default-spec* *testsqlite3-spec*) +(do-backend-tests) + +;; Test a Migration of data from BDB to postgres +(do-migration-tests *testbdb-spec* *testpg-spec*) + +;; An example usage. +(open-store *testpg-spec*) +(add-to-root "x1" "y1") +(get-from-root "x1") -(do-migrate-test-spec *test-path-primary*) +(add-to-root "x2" '(a 4 "spud")) +(get-from-root "x2") @end lisp The appropriate test should execute for you with no errors. @@ -440,5 +477,29 @@ @comment node-name, next, previous, up @section Documentation +If you are getting the documentation as a released tar file, you will probably find +the documenation in .html or .pdf form in the release, or can find it at the +Elephant website. + +If you want to compile the documentation youself, for example, if you can +think of a way to improve this manual, then you will do something similar +to this in a shell or command-line prompt: +@code +cd doc +make +make pdf +@end code + +This process will populate the ``./includes'' directory with references +automatically extracted from the list code. It will then compile the +texinfo documenation source into both HTML and a PDF which will be left +in the ``doc/elephant'' directory. + +Don't edit anything in the ``doc/elephant'' directory or the +``doc/includes'' directories, as everything in these directories is +generated. Instead, edit the ``.texinfo'' files in the doc directory. + + + --- /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/03/24 12:16:02 1.5 +++ /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/04/02 00:51:06 1.6 @@ -28,7 +28,8 @@ Elephant has been extended to provide support for multiple backends, specifically a relational database backend based on CL-SQL which has -been tested with Postgres and SQLite 3. It supports, with some care, +been tested with Postgres and SQLite 3, and probably support +other relational systems easily. It supports, with some care, multi-repository operation and enables convenient migration of data between repositories. --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/30 23:36:52 1.6 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/02 00:51:06 1.7 @@ -30,7 +30,8 @@ (make-instance 'elephant::persistent-collection :sc sc :from-oid 10) (make-instance 'elephant::secondary-cursor) (make-instance 'elephant::indexed-btree :sc sc :from-oid 10) - (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" +;; (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" + (sb-texinfo:generate-includes #p"." (find-package :elephant) (find-package :elephant-backend) (find-package :elephant-memutil) --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 20:22:24 1.12 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/02 00:51:06 1.13 @@ -32,6 +32,11 @@ Elephant provides a persistent index which maintains an ordered collection of lisp values or persistent object references. +The use of persistent objects makes coding concise, convenient, and +powerful, and makes persistence almost invisible to the programmer. +However, Elephant also allows the same basic data dictionary of +key/value retrieval that BerkeleyDB provides. + When someone says "database," most people think of SQL Relational Data Base Management Systems (e.g. Oracle, Postgresql, MySql). Those systems store data in statically typed tables with unique shared @@ -202,9 +207,11 @@ values and objects can be stored: numbers (except for complexes), symbols, strings, nil, characters, pathnames, conses, hash-tables, arrays, CLOS objects and structs. Nested and circular things are -allowed. You can store basically anything except lambdas, closures, -class objects, packages and streams. (These may eventually get -supported too.) +allowed. Nested and circular things are allowed. You can store +basically anything except compiled functions, closures, class objects, +packages and streams. Functions can be stored as uncompiled lambda +expressions. (Compiled functions and other kinds of objects may +eventually get supported too.) Elephant needs to use a representation of data that is independant of a specific lisp or data store. Therefore all lisp values that are --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:56:19 1.7 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/02 00:51:06 1.8 @@ -155,8 +155,8 @@ suggest or add indices to facilitate better performance on common queries. -There are two functions @ref{Function elephant:get-query-instances} -and @ref{Function elephant:map-class-query} which accept a set of +There are two functions @ref{Function elephant:get-query-instances,,,includes/fun-elephant-get-query-instance } +and @ref{Function elephant:map-class-query,,,includes/fun-elephant-map-class-query} which accept a set of constraints instead of the familiar value or range arguments. We'll use the classes @code{person} and @code{department} to

1 0

[elephant-cvs] CVS elephant/tests
by rread 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv27892 Modified Files: RUNTEST.lisp Log Message: Modernization of this file. --- /project/elephant/cvsroot/elephant/tests/RUNTEST.lisp 2006/02/05 23:46:41 1.1 +++ /project/elephant/cvsroot/elephant/tests/RUNTEST.lisp 2007/04/01 23:28:35 1.2 @@ -4,60 +4,44 @@ ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; -;;; Copyright (c) 2005,2006 by Robert L. Read +;;; Copyright (c) 2005,2006,2007 by Robert L. Read ;;; <rread(a)common-lisp.net> ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html) also known as the LLGPL. - - -;; This file is now obsolete... -;; Please use SQLDB-test.lisp or BerkeleyDB-tests.lisp - +;; If you are only using one back-end, you may prefer: +;; SQLDB-test.lisp or BerkeleyDB-tests.lisp (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) -(asdf:oos 'asdf:load-op :clsql-postgresql-socket) (asdf:operate 'asdf:load-op :ele-bdb) -(asdf:operate 'asdf:load-op :elephant-tests) - (asdf:operate 'asdf:load-op :ele-sqlite3) +(asdf:operate 'asdf:load-op :elephant-tests) (in-package "ELEPHANT-TESTS") -(do-all-tests) -(do-all-tests-spec *testpg-path*) -(do-migrate-test-spec *testpg-path*) -(do-all-tests-spec *testdb-path*) -(do-all-tests-spec *testsqlite3-path*) -;; The primary and secondary test-paths are -;; use for the migration tests. -(setq *test-path-primary* *testpg-path*) -(setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path*) +;; Test Postgres backend +(setq *default-spec* *testpg-spec*) +(do-backend-tests) + +;; Test BDB backend +(setq *default-spec* *testbdb-spec*) +(do-backend-tests) + +;; Test SQLite 3 +(setq *default-spec* *testsqlite3-spec*) +(do-backend-tests) -(setq *test-path-primary* *testdb-path*) -(setq *test-path-secondary* nil) +;; Test a Migration of data from BDB to postgres +(do-migration-tests *testbdb-spec* *testpg-spec*) -(do-all-tests-spec *test-path-primary*) - - -(use-package :sb-profile) - -(profile "CLSQL") -(profile "POSTGRESQL-SOCKET") -(profile "ELEPHANT") - -(use-package "SB-PROFILE") - -(open-store *testpg-path*) -(open-store *testdb-path*) +;; An example usage. +(open-store *testpg-spec*) (add-to-root "x1" "y1") (get-from-root "x1") - (add-to-root "x2" '(a 4 "spud")) (get-from-root "x2")

1 0

[elephant-cvs] CVS elephant/src/db-clsql
by rread 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv8188 Modified Files: package.lisp Log Message: Not sure what happened here but this file is definitely needed for the test on the sql side. --- /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/02/07 22:54:12 1.2 +++ /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/04/01 22:01:51 1.3 @@ -21,7 +21,7 @@ (defpackage db-clsql (:use :common-lisp :uffi :cl-base64 :elephant :elephant-memutil :elephant-backend -;; :elephant-utils + :elephant-utils #+sbcl :sb-thread ))

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv27196/doc Modified Files: user-guide.texinfo Log Message: Fixed package bug. Minor tweaks elsewhere --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:22:24 1.6 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:56:19 1.7 @@ -141,8 +141,6 @@ @comment node-name, next, previous, up @section Querying persistent instances - - A SQL select-like interface is in the works, but for now queries are limited to manual mapping over class instances or doing small queries with @code{get-instances-*} functions. One advantage of this is that

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv27196/src/elephant Modified Files: backend.lisp package.lisp query-example.lisp query.lisp Log Message: Fixed package bug. Minor tweaks elsewhere --- /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 23:36:53 1.15 +++ /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/04/01 20:56:19 1.16 @@ -56,6 +56,10 @@ #:controller-deserialize #:root #:spec #:class-root + ;; Collections + #:build-btree + #:build-indexed-btree + ;; Serializer tools/api's #:serialize #:deserialize #:deserialize-from-base64-string --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/03/30 14:55:54 1.27 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/01 20:56:19 1.28 @@ -206,9 +206,9 @@ #:persistent #:persistent-object #:persistent-metaclass #:defpclass #:persistent-collection #:drop-pobject - #:btree #:build-btree + #:btree #:make-btree #:get-value #:remove-kv #:existsp - #:indexed-btree #:build-indexed-btree + #:indexed-btree #:make-indexed-btree #:btree-index #:add-index #:get-index #:remove-index #:map-indices #:get-primary-key #:primary #:key-form #:key-fn @@ -260,6 +260,9 @@ #:lookup-persistent-symbol-id #:struct-constructor + + #:map-class-query + #:get-query-instances ) ) --- /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/03/01 02:45:45 1.1 +++ /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/04/01 20:56:19 1.2 @@ -61,7 +61,7 @@ (slot-value (slot-value person 'department) 'name))) (defun example-query1 () - "Performs a query against a single class. Trivial string & integer matchingA" + "Performs a query against a single class. Trivial string & integer matching" (map-class-query #'print-person '((person name = "Greg") (person salary >= 100000)))) --- /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/03/30 14:34:35 1.2 +++ /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/04/01 20:56:19 1.3 @@ -41,6 +41,7 @@ (defun get-query-instances (constraints) "Get a list of instances according to the query constraints" + (declare (dynamic-extent constraints)) (let ((list nil)) (flet ((collect (inst) (push inst list))) @@ -52,6 +53,7 @@ slot indices (for last query) and stack allocated test closures. This is a minimally optimizing version that uses the first index it finds, and then does a nested loop join on the rest of the parameters." + (declare (dynamic-extent constraints)) (assert (not (null constraints))) (destructuring-bind (class slot relation &rest values) (first constraints) (flet ((filter-by-relation (inst)

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv22287 Modified Files: tutorial.texinfo user-guide.texinfo Log Message: Documentation changes, mostly to transaction section of tutorial --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 20:22:24 1.12 @@ -732,6 +732,8 @@ transaction that performs all the updates atomically and thus enforcing consistency. +@subsection Why do we need Transactions? + Most real applications will need to use explicit transactions rather than relying on the primitives alone because you will want multiple read-modify-update operations act as an atomic unit. A good example @@ -815,6 +817,8 @@ And presto, we have an ACID compliant, thread-safe, persistent banking system! +@subsection Using @code{with-transaction} + What is @code{with-transaction} really doing for us? It first starts a new transaction, attempts to execute the body, and if successful commit the transaction. If anywhere along the way there is a deadlock @@ -823,14 +827,145 @@ to retry the transaction a fixed number of times by re-executing the whole body. -The other value transactions provide is the capability to delay -flushing dirty data to disk. The most time-intensive part of -persistent operations is flushing newly written data to disk. Using -the default auto-commit behavior requires a flush on every operation -which can become very expensive. Because a transaction caches values, -all the values read or written are cached in memory until the -transaction completes, dramatically decreasing the number of flushes -and the total time taken. +And this brings us to two important caveats: nested transactions and +idempotent side-effects. + +@subsection Nesting Transactions + +In general, you want to avoid nesting @code{with-transaction} +statements. Nested transactions are valid for some data stores +(namely Berkeley DB), but typically only a single transaction can be +active at a time. The purpose of a nested transaction in data stores +that provide it, is break a long transaction into chunks. This way if +there is contention on a given subset of variables, only the inner +transaction is restarted while the larger transaction can continue. +When commit their results, those results become part of the outer +transaction until it in turn commits. + +If you have transaction protected primitive operations (such as +@code{deposit} and @code{withdraw}) and you want to perform a group of +such transactions, for example a transfer between accounts, you can +use the macro @code{ensure-transaction} instead of @code{with-transaction}. + +@lisp +(defun deposit (account amount) + "Wrap the balance read and the setf with the new balance" + (ensure-transaction () + (let ((balance (balance account))) + (setf (balance account) + (+ balance amount))))) + +(defun deposit (account amount) + "A more concise version with decf doing both read and write" + (ensure-transaction () + (decf (balance account) amount))) + +(defun withdraw (account amount) + (ensure-transaction () + (decf (balance account) amount))) + +(defun transfer (src dst amount) + "There are four primitive read/write operations + grouped together in this transaction" + (with-transaction () + (withdraw src amount) + (deposit dst amount))) +@end lisp + +@code{ensure-transaction} is exactly like @code{with-transaction} +except it will reuse an existing transaction, if there is one, or +create a new one. There is no harm, in fact, in using this macro all +the time. + +Notice the use of @code{decf} and @code{incf} above. The primary +reason to use Lisp is that it is good at hiding complexity using +shorthand constructs just like this. This also means it is also going +to be good at hiding data dependencies that should be captured in a +transaction! + +@subsection Idempotent Side Effects + +Within the body of a with-transaction, any non database operations +need to be @emph{idempotent}. That is the side effects of the body +must be the same no matter how many times the body is executed. This +is done automatically for side effects on the database, but not for +side effects like pushing a value on a lisp list, or creating a new +standard object. + +@lisp +(defparameter *transient-objects* nil) + +(defun load-transients (n) + "This is the wrong way!" + (with-transaction () + (loop for i from 0 upto n do + (push (get-from-root i) *transient-objects*)))) +@end lisp + +In this contrived example we are pulling a set of standard objects +from the database using an integer key and pushing them onto a list +for later use. However, if there is a conflict where some other +process writes a key-value pair to a matching key, the whole +transaction will abort and the loop will be run again. In a heavily +contended system you might see results like the following. + +@lisp +(defun test-list () + (setf *transient-objects* nil) + (load-transients) + (length *transient-objects*)) + +(test-list) +=> 3 + +(test-list) +=> 5 + +(test-list) +=> 4 +@end lisp + +So the solution is to make sure that the operation on the lisp +parameters is atomic if the transaction completes. + +@lisp +(defun load-transients () + "This is a better way" + (setq *transient-objects* + (with-transaction () + (loop for i from 0 upto 3 collect + (get-from-root i))))) +@end lisp + +Of course we would need to use @code{nreverse} if we cared about the +order of instances in @code{*transient-objects*}. The best rule of +thumb is that transaction bodies should be purely functional as above, +except for side effects to the persistent store such as persistent +slot writes, adding to btrees, etc). + +If you do need side effects to lisp memory, such as writes to +transient slots, make sure they are idempotent and that other +processes will not be reading the written values until the transaction +completes. + +@subsection Transactions and Performance + +By now transactions almost look like more work than they are worth! +Well there are still some significant benefits to be had. Part of how +transactions are implemented is that they gather together all the +writes that are supposed to made to the database and store them until +the transaction commits, and then writes them atomically. + +The most time-intensive part of persistent operations is flushing +newly written data to disk. Using the default auto-committing +behavior requires a flush for every primitive write operation. This +can become very expensive! Because all the values read or written are +cached in memory until the transaction completes, the number of +flushes can be dramatically reduced. + +But don't take my word for it, run the following statements and see +for yourself the visceral impact transactions can have on system +performance. @lisp (defpclass test () @@ -872,52 +1007,42 @@ thumb is to keep the number of objects touched in a transaction well under 1000. -And this brings us to the last caveat we'll introduce in this -introductory tutorial: nested transactions. - -In general, avoid nesting transactions. Nested transactions are valid -for some data stores (namely Berkeley DB), but typically only a single -transaction is valid at a time. The purpose of a nested transaction -is to allow a long transaction to be broken up into chunks. This way -if there is contention on a given subset of variables, only the -subtransaction is restarted while the larger transaction can continue. -Subtransactions commit their results and they become part of the -outer transaction until it in turn commits. - -If you have transaction protected primitive operations (such as -@code{deposit} and @code{withdraw}) and you want to perform a group of -such transactions, for example a transfer between accounts, you can -use the macro @code{ensure-transaction} instead of @code{with-transaction}. - -@lisp -(defun deposit (account amount) - (ensure-transaction () - (let ((balance (balance account))) - (setf (balance account) - (+ balance amount))))) - -(defun withdraw (account amount) - (ensure-transaction () - (decf (balance account) amount))) - -(defun transfer (src dst amount) - (with-transaction () - (withdraw src amount) - (deposit dst amount))) -@end lisp - -@code{ensure-transaction} is exactly like @code{with-transaction} -except it will reuse an existing transaction, if there is one, or -create a new one. There is no harm, in fact, in using this macro all -the time. +@subsection Transactions and Applications Designing and tuning a transactional architecture can become quite -complicated. The best strategy at the beginning is a conservative -one, break things up into the smallest logical sets of primitive -operations and only wrap higher level functions in transactions when -they absolutely have to commit together. See @ref{Transaction Details} -for the full details and @pxref{Usage Scenarios} for more examples of -how systems can be designed and tuned using transactions. +complex. Moreover, bugs in your system can be very difficult to find +as they only show up when transactions are interleaved within a +larger, multi-threaded application. + +In many cases, however, you can ignore transactions. For example, +when you don't have any other concurrent processes running. In this +case all operations are sequential and there is no chance of +conflicts. You would only want to use transactions for write +performance. + +You can also ignore transactions if your application can guarantee +that concurrency won't generate any conflicts. For example, a web app +that guarantees only one thread will write to objects in a particular +session can avoid transactions altogether. However, it is good to be +careful about making these assumptions. In the above example, a +reporting function that iterates over sessions, users or other objects +may still see partial updates (i.e. a user's id was written prior to +the query, but not the name). However, if you don't care about these +infrequent glitches, this case would still hold. + +If these cases don't apply to your application, or you aren't sure, +you will fare best by programming defensively. Break your system into +the smallest logical sets of primitive operations +(i.e. @code{withdraw} and @code{deposit}) using +@code{ensure-transaction} and then wrap the highest level calls made +to your system in with-transaction when the operations absolutely have +to commit together or you need the extra performance. Try not to have +more than two levels of transactional accesses with the top using +with-transaction and the bottom using ensure-transaction. + +@xref{Transaction Details} for more details and @pxref{Usage +Scenarios} for examples of how systems can be designed and tuned using +transactions. @node Advanced Topics @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:22:24 1.6 @@ -23,26 +23,6 @@ * Performance Tuning:: How to get the most from Elephant. @end menu -@node Persistent objects -@comment node-name, next, previous, up -@section Persistent Objects - -Finally, if you for some reason make an instance with a specified OID -which already exists in the database, @code{initargs} take precedence -over values in the database, which take precedences over -@code{initforms}. - -Also currently there is a bug where -@code{initforms} are always evaluated, so beware. -(What is the current model here?) - -Readers, writers, accessors, and @code{slot-value-using-class} are -employed in redirecting slot accesses to the database, so override -these with care. Because @code{slot-value, slot-boundp, -slot-makunbound} are not generic functions, they are not guaranteed by -the specification to work properly with persistent slots. However the -proper behavior has been verified on SBCL, Allegro and Lispworks. - @node The Store Controller @comment node-name, next, previous, up @section The Store Controller @@ -90,6 +70,26 @@ Empty. +@node Persistent objects +@comment node-name, next, previous, up +@section Persistent Objects + +Finally, if you for some reason make an instance with a specified OID +which already exists in the database, @code{initargs} take precedence +over values in the database, which take precedences over +@code{initforms}. + +Also currently there is a bug where +@code{initforms} are always evaluated, so beware. +(What is the current model here?) + +Readers, writers, accessors, and @code{slot-value-using-class} are +employed in redirecting slot accesses to the database, so override +these with care. Because @code{slot-value, slot-boundp, +slot-makunbound} are not generic functions, they are not guaranteed by +the specification to work properly with persistent slots. However the +proper behavior has been verified on SBCL, Allegro and Lispworks. + @node Class Indices @comment node-name, next, previous, up @section Class Indices @@ -141,6 +141,111 @@ @comment node-name, next, previous, up @section Querying persistent instances + + +A SQL select-like interface is in the works, but for now queries are +limited to manual mapping over class instances or doing small queries +with @code{get-instances-*} functions. One advantage of this is that +it is easy to estimate the performance costs of your queries and to +choose standard and derived indices that give you the ordering and +performance you want. + +There is, however, a quick and dirty query API example that is not +officially supported in the release but is intended to invite comment. +This is an example of a full query system that would automatically +perform joins, use the appropriate indices and perhaps even adaptively +suggest or add indices to facilitate better performance on common +queries. + +There are two functions @ref{Function elephant:get-query-instances} +and @ref{Function elephant:map-class-query} which accept a set of +constraints instead of the familiar value or range arguments. + +We'll use the classes @code{person} and @code{department} to +illustrate how to perform queries over a set of objects that may be +constrainted by their relationships to other objects. + +@lisp +(defpclass person () + ((name :initarg :name :index t) + (salary :initarg :salary :index t) + (department :initarg :dept))) + +(defmethod print-object ((p person) stream) + (format stream "#<PERS: ~A>" (slot-value p 'name))) + +(defun print-name (inst) + (format t "Name: ~A~%" (slot-value inst 'name))) + +(defpclass department () + ((name :initarg :name) + (manager :initarg :manager))) + +(defmethod print-object ((d department) stream) + (format stream "#<DEPT ~A, mgr = ~A>" + (slot-value d 'name) + (when (slot-boundp d 'manager) + (slot-value (slot-value d 'manager) 'name)))) +@end lisp + +Here we have a simple employee database with managers (also of type +person) and departments. This simple system will provide fodder for +some reasonably complex constraints. Let's create a few departments. + +@lisp +(setf marketing (make-instance 'department :name "Marketing")) +(setf engineering (make-instance 'department :name "Engineering")) +(setf sales (make-instance 'department :name "Sales")) +@end lisp + +And manager @code{people} for the departments. + +@lisp +(make-instance 'person :name "George" :salary 140000 :department marketing) +(setf (slot-value marketing 'manager) *) + +(make-instance 'person :name "Sally" :salary 140000 :department engineering) +(setf (slot-value engineering 'manager) *) + +(make-instance 'person :name "Freddy" :salary 180000 :department sales) +(setf (slot-value sales 'manager) *) +@end lisp + +And of course we need some folks to manage + +@lisp +(defparameter *names* + '("Jacob" "Emily" "Michael" "Joshua" "Andrew" "Olivia" "Hannah" "Christopher")) + +(defun random-element (list) + "Choose a random element from the list and return it" + (nth (random (length list)) list)) + +(with-transaction () + (loop for i from 0 upto 40 do + (make-instance 'person + :name (format nil "~A~A" (random-elephant *names*) i) + :salary (floor (+ (* (random 1000) 100) 30000)) + :department (case (random 3) + (0 marketing) + (1 engineering) + (2 sales))))) +@end lisp + +Due to the random allocation of +In the follwoing examples below, the results will be different due to the random +allocation of employee names, etc. However, these examples are +illustrative of what you should see if you run the same code. + + + +For those familiar with SQL, if an instance of @code{person} has a +pointer to an instance of @code{department} then that relation can be +used to perform a join. Of course joins in the object world won't +return a table, instead they will return conjunctions of objects that +satisfy a mutual set of constraints. + + @node Using BTrees @comment node-name, next, previous, up @section Using BTrees @@ -174,6 +279,14 @@ @comment node-name, next, previous, up @section Transaction Details +You can trace @code{elephant::execute-transaction} to see the sequence +of calls to @code{execute-transaction} that occur dynamically and +detect where transactions are and are not happening. We may add some +transaction diagnosis and tracing tools in the future, such as +throwing a condition when @code{with-transaction} forms are nested +dynamically. + + ;; Transaction architecture: ;; ;; User and designer considerations:

1 0

[elephant-cvs] CVS elephant/src/elephant
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv29083/src/elephant Modified Files: classes.lisp controller.lisp metaclasses.lisp serializer.lisp Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/30 23:36:53 1.25 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/01 14:33:34 1.26 @@ -39,9 +39,11 @@ (defclass persistent-object (persistent) () (:metaclass persistent-metaclass) (:documentation - "Superclass of all user-defined persistent classes. This is + "Superclass for all user-defined persistent classes. This is automatically inherited if you use the persistent-metaclass - metaclass.")) + metaclass. This allows specialization of functions for user + objects that would not be appropriate for Elephant objects + such as persistent collections")) ;; ================================================ ;; METACLASS INITIALIZATION AND CHANGES --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:42:35 1.44 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/01 14:33:34 1.45 @@ -48,11 +48,13 @@ (defvar *dbconnection-spec* (make-hash-table :test 'equal)) (defvar *dbconnection-lock* (ele-make-lock)) -(defmethod get-con ((instance persistent) &optional (sc *store-controller*)) - "This is used to find and validate the connection spec +(defgeneric get-con (instance &optional sc) + (:documentation "This is used to find and validate the connection spec maintained for in-memory persistent objects. Should we re-open the controller from the spec if it's not - cached? That might be dangerous so for now we error" + cached? That might be dangerous so for now we error")) + +(defmethod get-con ((instance persistent) &optional (sc *store-controller*)) (declare (ignore sc)) (let ((con (gethash (dbcn-spc-pst instance) *dbconnection-spec*))) (cond ((not con) --- /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/03/30 23:36:53 1.14 +++ /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/04/01 14:33:34 1.15 @@ -29,7 +29,7 @@ :documentation "Persistent objects use a spec pointer to identify which store they are connected to")) (:documentation "Abstract superclass for all persistent classes (common - to user-defined classes and collections.)")) + to both user-defined classes and Elephant-defined objects such as collections.)")) (defclass persistent-metaclass (standard-class) ((%persistent-slots :accessor %persistent-slots) --- /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/03/30 14:34:35 1.26 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/04/01 14:33:34 1.27 @@ -47,6 +47,8 @@ ;; (defun serialize-to-base64-string (x sc) + "Encode object using the store controller's serializer format, + but encoded in a base64" (with-buffer-streams (out-buf) (cl-base64::usb8-array-to-base64-string (elephant-memutil::buffer-read-byte-vector @@ -62,6 +64,7 @@ ) (defun deserialize-from-base64-string (x sc) + "Decode a base64-string using the store controller's deserialize method" (with-buffer-streams (other) (deserialize (elephant-memutil::buffer-write-byte-vector

1 0

[elephant-cvs] CVS elephant/doc
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv29083/doc Modified Files: Makefile data-store-reference.texinfo elephant-design.texinfo elephant.texinfo installation.texinfo reference.texinfo tutorial.texinfo user-guide.texinfo Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/30 23:36:52 1.5 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/01 14:33:24 1.6 @@ -9,4 +9,4 @@ makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo pdf: includes-stuff - texi2dvi --pdf elphant.texinfo + texi2dvi --pdf elephant.texinfo --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 23:36:52 1.4 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5 @@ -3,9 +3,8 @@ @node Data Store API Reference @comment node-name, next, previous, up @chapter Data Store API Reference -@cindex Data Store API Reference @cindex Data Store -@cindex API Reference +@cindex API This reference includes functions that need to be overridden, classes inherited from or other action taken to implement support for a new @@ -26,16 +25,16 @@ relevant to them. @menu -* Registration:: Register the backend to parse controller specifications. -* Store Controllers:: Subclassing the store controller. -* Handling Serialization:: Available facilities for serializing objects. -* C Utilities:: Writing primitive C types. -* Slot access:: Support for metaprotocol slot access. -* Collections:: BTrees and indices. -* Cursors:: Traversing BTrees. -* Transactions:: Transaction implementation. -* Multithreading:: Multithreading considerations. -* Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries +* Registration: DSR Registration. Register the backend to parse controller specifications. +* Store Controllers: DSR Store Controllers. Subclassing the store controller. +* Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects. +* Memory Utilities: DSR Memory Utilities. Writing primitive C types. +* Persistent Objects and Slot access: DSR Persistent Objects and Slot Access. Support for metaprotocol slot access. +* Collections: DSR Collections. BTrees and indices. +* Cursors: DSR Cursors. Traversing BTrees. +* Transactions: DSR Transactions. Transaction implementation. +* Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations. +* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries @end menu @node DSR Registration @@ -96,17 +95,28 @@ @section Slot Access @cindex Persistent Objects and Slot Access +Persistence is implement with a metaclass and several required base +classes. + +@include includes/class-elephant-persistent-metaclass.texinfo @include includes/class-elephant-persistent.texinfo +@include includes/class-elephant-persistent-object.texinfo + +Persistent objects can be queries for their home store controller so +that functions such as map-btree do not need a store-controller +argument. (NOTE: Should this function be user visible?) + @include includes/fun-elephant-backend-get-con.texinfo -@c @include includes/fun-elephant-backend-oid.texinfo -All objects require a unique id. During new object creation the -backend is asked to produce a unique id. +All objects require a unique object identifier. During new object +creation the backend is asked to produce a unique id. @include includes/fun-elephant-backend-next-oid.texinfo -These functions are called by the metaclass protocol to support -operations on persistent class slots. +These functions are called by the metaclass protocol to implement the +appropriate operations on persistent class slots. Unless protected by +a transaction, the side effects of these functions should be atomic, +persistent and visible to other threads on completion. @include includes/fun-elephant-backend-persistent-slot-writer.texinfo @include includes/fun-elephant-backend-persistent-slot-reader.texinfo @@ -121,7 +131,9 @@ To support collections, the data store must subclass the following classes. -@include includes/class-elephant-btree.texinfo.texinfo +@include includes/class-elephant-persistent-collection.texinfo + +@include includes/class-elephant-btree.texinfo @include includes/class-elephant-btree-index.texinfo @include includes/class-elephant-indexed-btree.texinfo @@ -131,42 +143,48 @@ @include includes/fun-elephant-build-btree.texinfo @include includes/fun-elephant-build-indexed-btree.texinfo -And every btree needs accessors, these must be implemented for btree, -indexed-btree and btree-index. +Most of the user-visible operations over BTrees must be implemented. +Class indexing functions such as @code{map-class} and +@code{get-instances-by-value} and related functions are all +implemented using map-btree and map-index. + +@itemize +@item @ref{Generic-Function elephant:get-value} +@item @ref{Generic-Function (setf elephant:get-value)} +@item @ref{Generic-Function elephant:existsp} +@item @ref{Generic-Function elephant:remove-kv} +@item @ref{Generic-Function elephant:get-index} +@item @ref{Generic-Function elephant:remove-index} +@item @ref{Generic-Function elephant:map-btree} +@item @ref{Generic-Function elephant:map-index} +@end itemize -@include includes/fun-elephant-get-value.texinfo -@include includes/fun-elephant-setf-get-value.texinfo -@include includes/fun-elephant-existsp.texinfo -@include includes/fun-elephant-remove-kv.texinfo +Mapping over the indices of a btree is important to derived facilities +such as class indexing and the query subsystem. @include includes/fun-elephant-map-indices.texinfo -@include includes/fun-elephant-get-index.texinfo -@include includes/fun-elephant-remove-index.texinfo - -Critical to indexing and queries are the map operators for collections - -@include includes/fun-elephant-map-btree.texinfo -@include includes/fun-elephant-map-index.texinfo @node DSR Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors -@include includes/class-cursor.texinfo -@c #:cursor-btree -@c #:cursor-oid -@c #:cursor-initialized-p +Data stores must subclass these cursor classes and implement all the +methods described in @ref{Cursors} except @ref{Macro +elephant:with-btree-cursor}. + +@include includes/class-elephant-cursor.texinfo +@include includes/class-elephant-secondary-cursor.texinfo @node DSR Transactions @comment node-name, next, previous, up @section Transactions @cindex Transactions -These functions must be implemented or stubbed in any -backend. +These functions must be implemented or stubbed by all data stores. @include includes/fun-elephant-backend-execute-transaction.texinfo + @include includes/fun-elephant-backend-controller-start-transaction.texinfo @include includes/fun-elephant-backend-controller-commit-transaction.texinfo @include includes/fun-elephant-backend-controller-abort-transaction.texinfo @@ -179,33 +197,77 @@ @include includes/fun-elephant-backend-transaction-store.texinfo @include includes/fun-elephant-backend-transaction-object.texinfo +;; Designer considerations: +;; - with-transaction passes *current-transaction* or the user parameter to execute-transaction +;; in the parent keyword argument. Backends allowing nested transactions can treat the transaction +;; as a parent, otherwise they can reuse the current transaction by ignoring it (inheriting the dynamic +;; value of *current-transaction*) or rebinding the dynamic context (whatever makes coding easier). +;; - ensure-transaction uses *current-transaction* to determine if there is a current transaction +;; in progress (not null). If so, it jumps to the body directly. Otherwise it executes the body in a +;; new transaction by calling ... +;; - execute-transaction contract: +;; - Backends must dynamically bind *current-transaction* to a meaningful identifier for the +;; transaction in progress and execute the provided closure in that context +;; - All non-local exists result in an abort; only regular return values result in a commit +;; - If a transaction is aborted due to a deadlock or read conflict, execute-transaction should +;; automatically retry with an appropriate default amount +;; - execute-transaction can take any number of backend-defined keywords, although designers should +;; make sure there are no semantic conflicts if there is a name overlap with existing backends +;; - A typical design approach is to make sure that the most primitive interfaces to the backend +;; database look at *current-transaction* to determine whether a transaction is active. Users code can also +;; access this parameter to check whether a transaction is active. + @node DSR Multithreading Considerations @comment node-name, next, previous, up @section Multithreading Considerations @cindex Multithreading -@c utils locks -@c utils thread-vars +Generic locking utility functions + +Variable behavior in multithreading situations @node DSR Handling Serialization @comment node-name, next, previous, up @section Handling Serialization @cindex Serialization -@c #:deserialize #:serialize -@c #:serialize-symbol-complete -@c #:deserialize-from-base64-string -@c #:serialize-to-base64-string +Backends must initialize @ref{Class elephant:store-controller} with +internal serializer functions. Packages @code{elephant-serializer1} +and @code{elephant-serializer2} contains serialize and deserialize +methods on buffer-streams as defined in @code{elephant-memutil}. The +elephant functions @code{serialize} and @code{deserialize} dispatch on +the appropriate slot values of the store-controller. + +@verbatim +NOTE: This should perhaps become entirely the job of the data store to +decide how to serialize values and for a specific version, what +serializer to use. The elphant main package can define serializers +for use by different backends. +@end verbatim + +@include includes/fun-elephant-backend-serialize.texinfo +@include includes/fun-elephant-backend-deserialize.texinfo + +These utility functions are useful if a data store does not have the +ability to store variable length binary data. They are based on the +@code{cl-base64} library. + +@include includes/fun-elephant-backend-serialize-to-base64-string.texinfo +@include includes/fun-elephant-backend-deserialize-from-base64-string.texinfo -@node DSR Memory utilities +@node DSR Memory Utilities @comment node-name, next, previous, up @section Memory utilities @cindex Memory utilities -@node DSR Foreign libraries +Details about memory utilities here. + +@node DSR Foreign Libraries @comment node-name, next, previous, up @section Foreign libraries @cindex Foreign libraries +How foreign libraries are built and used via UFFI. What functions are +in the .asd files or main lisp code to build & load libraries? --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/30 14:34:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/01 14:33:29 1.3 @@ -1,9 +1,37 @@ +@c -*-texinfo-*- @node Elephant Design @comment node-name, next, previous, up -@section Elephant Design +@chapter Elephant Design @cindex design +Elephant's early architecture was tightly coupled to the Berkeley DB +API. Over time we've moved towards a more modular architecture to +support easy upgrading, repository migration, shared functionality +between data stores and general hygene. + +The current architecture is modularized into the following pieces: + +@verbatim +[Picture describing] Metaclass, Store controller, persistent objects, +data stores, serializer & memutils. Derived features such as class +indexing, migration, query system and root operations can also be +illustrated? +@end verbatim + +To get a feeling for what is happening inside elephant, it is probably +most useful to walk through the various major protocols: + +@itemize +@item Initialization of a store controller +@item Creating a persistent object +@item Operations on persistent slots +@item Operations on persistent collections +@item Implementing @code{with-transaction} +@end itemize + +@section Initializing a store controller + When the main elephant @code{open-store} function is called, it calls @code{get-controller} which grabs an existing store controller if the spec is identical, or builds a new controller. Building the @@ -13,4 +41,14 @@ that returns a @code{store-controller} subclass instance specific to that backend. -Elephant than calls open-controller +Elephant than calls open-controller to actually establish a connection +to or create the files of the data store. + +@section Persistent Object Creation +@section Persistent Slot Protocol +@section Persistent Slot Protocol +@section Persistent Collection Protocol +@section Implementing Transactions + + + --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/30 14:34:34 1.6 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/01 14:33:29 1.7 @@ -38,6 +38,7 @@ @insertcopying @end ifnottex +@ifhtml @menu * Table of Contents:: @end menu @@ -49,7 +50,7 @@ * Tutorial:: A basic ``getting started'' tutorial. * Installation:: Installation and test-suite procedures. * User Guide:: In depth discussion of all Elephant facilities and features. -* Usage scenarios:: Design scenarios for Elephant applications. +* Usage Scenarios:: Design scenarios for Elephant applications. * User API Reference:: Function and class documentation of the user API. * Elephant Design:: An overview of elephant's internal architecture. * Data Store API Reference:: Function level documentation for data store implementors. @@ -66,6 +67,8 @@ * Colophon:: @end menu +@end ifhtml + @node Table of Contents @unnumbered @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/30 14:34:34 1.5 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6 @@ -10,9 +10,9 @@ * Configuring Elephant:: Setting up Elephant and the configuration file. * Loading Elephant:: Loading Elephant and the data store loading protocol. * Berkeley DB Data Store:: Installing support for the Berkeley DB data store -* Berkeley DB Examples:: An example of installing and running the Berkeley DB data store. +* Berkeley DB Example:: An example of installing and running the Berkeley DB data store. * CL-SQL Data Store:: Install and connecting to the CL-SQL data store -* CL-SQL Examples:: An example of using the CL-SQL data store. +* CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows * Test Suites:: How to run and interpret the output of the regression test suite * Documentation:: Building documentation from texinfo sources. @@ -277,10 +277,6 @@ As of Elephant 0.3, Elephant has been tested to work with both Postgres, and SQLite 3, thanks do Dan Knapp. -@node Extension Status -@comment node-name, next, previous, up -@section Extension Status - As far as is known at this writing, all functionality except nested transaction support and cursor-puts supported by the BerkeleyDB backend is supported by the CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested @@ -302,13 +298,9 @@ the multi-repository version somewhat complicates the underlying persistent object management. -@node PostGres Examples -@comment node-name, next, previous, up -@section Setting up PostGres - -@node Setting up PostGres +@node CL-SQL Example @comment node-name, next, previous, up -@section Setting up PostGres +@section CL-SQL Example To set up a PostGres based back end, you should: @@ -351,6 +343,10 @@ sequences, and so on needed by the Elephant system will be created in the schema named ``test'' automatically. +@node Elephant on Windows +@comment node-name, next, previous, up +@section Elephant on Windows + @node Test Suites @comment node-name, next, previous, up @section Test Suites --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 23:36:52 1.8 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/01 14:33:29 1.9 @@ -22,14 +22,20 @@ @section Store Controllers @cindex Store Controllers -Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller. +Store controllers provide the persistent storage for CLOS objects and +BTree collections. Any persistent operations must be done in the +context of a store controller. The default store-controller is stored +in a global variable. -@include includes/class-elephant-store-controller.texinfo @include includes/var-elephant-star-store-controller-star.texinfo +@c @include includes/class-elephant-store-controller.texinfo +@ref{Class elephant:store-controller} is associated with the following +user methods and macros: + +@include includes/macro-elephant-with-open-store.texinfo @include includes/fun-elephant-open-store.texinfo @include includes/fun-elephant-close-store.texinfo -@include includes/macro-elephant-with-open-store.texinfo @include includes/fun-elephant-get-from-root.texinfo @include includes/fun-elephant-add-to-root.texinfo @@ -42,12 +48,12 @@ @section Persistent Objects @cindex Persistent Objects -@include includes/class-elephant-persistent-metaclass.texinfo -@include includes/class-elephant-persistent.texinfo -@include includes/class-elephant-persistent-object.texinfo +@ref{Class elephant:persistent-metaclass} can be used as the +:metaclass argument in a defclass form to create a persistent object. +Slots of the metaclass take the :index and :transient keyword +arguments and the class accepts the :index keyword argument. @include includes/macro-elephant-defpclass.texinfo - @include includes/fun-elephant-drop-pobject.texinfo @node Persistent Object Indexing @@ -94,22 +100,26 @@ interface is provided for reference only, a new system is under development for the 0.7 release. -@include includes/fun-elephant-get-query-results.texinfo -@include includes/fun-elephant-map-class-query.texinfo +@c @include includes/fun-elephant-get-query-results.texinfo +@c @include includes/fun-elephant-map-class-query.texinfo @node Collections @comment node-name, next, previous, up @section Collections @cindex Collections -@include includes/class-elephant-persistent-collection.texinfo -@include includes/class-elephant-btree.texinfo -@include includes/class-elephant-indexed-btree.texinfo -@include includes/class-elephant-btree-index.texinfo +Persistent collections inherit from @ref{Class +elephant:persistent-collection} and consist of the @ref{Class +elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class +elephant:btree-index} classes. The following operations are defined +on most of these classes. More information can be found in @ref{Using +BTrees} and @ref{Secondary Indices}. @include includes/fun-elephant-get-value.texinfo @include includes/fun-elephant-setf-get-value.texinfo @include includes/fun-elephant-remove-kv.texinfo +@include includes/fun-elephant-existsp.texinfo + @include includes/fun-elephant-map-btree.texinfo @include includes/fun-elephant-map-index.texinfo @@ -118,17 +128,17 @@ @include includes/fun-elephant-get-primary-key.texinfo @include includes/fun-elephant-remove-index.texinfo - @node Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors -@include includes/class-elephant-cursor.texinfo -@include includes/class-elephant-secondary-cursor.texinfo +Cursors are objects of type cursor (@pxref{Class elephant:cursor}) +which provide methods for complex traversals of BTrees. + +@include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo -@include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-cursor-current.texinfo @include includes/fun-elephant-cursor-delete.texinfo --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/30 14:34:34 1.10 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11 @@ -15,6 +15,7 @@ * Persistent collections:: Keep track of collections of values. * Indexing Persistent Classes:: Simple way to keep track of persistent instances. * Using Transactions:: Providing ACID database properties. +* Advanced Topics:: Additional Elephant features covered in the User Guide. @end menu @node Overview @@ -704,14 +705,14 @@ @end lisp The @ref{User Guide} contains a descriptions of the advanced features -of @ref{Class indices} such as ``derived indicies'' that allow you to +of @ref{Class Indices} such as ``derived indicies'' that allow you to order classes according to an arbitrary function, a dynamic API for adding and removing slots and how to set a policy for resolving conflicts between the code image and a database where the indexing specification differs. This same facility is also available for your own use. For more -information @pxref{Using Indexed BTrees}. +information @pxref{Secondary Indices}. @node Using Transactions @@ -914,8 +915,71 @@ complicated. The best strategy at the beginning is a conservative one, break things up into the smallest logical sets of primitive operations and only wrap higher level functions in transactions when -they absolutely have to commit together. See @ref{Transaction details} -for the full details and @pxref{Usage scenarios} for more examples of +they absolutely have to commit together. See @ref{Transaction Details} +for the full details and @pxref{Usage Scenarios} for more examples of how systems can be designed and tuned using transactions. +@node Advanced Topics +@comment node-name, next, previous, up +@section Advanced Topics + +The tutorial covers the essential topics and concepts for using +Elephant. Many people will find that these features are the ones that +are most often needed and used in ordinary applications. + +More sophisticated uses of Elephant may require additional features +that are covered in the user guide. The following is a list of major +features in the user guide that were not covered in this tutorial. + +@itemize +@item @strong{Class Heirarchies and Queries} + There are some subtle issues to take into account when querying +persistent classes. For example, how do you query a base class of +type people to get subclass instances such as employee, manager, +consultant, etc? +@item @strong{Derived Class Indices} + You can create your own indices for classes that are arbitrary +lisp functions of the persistent object. +@item @strong{Class Definition/Database Conflict Resolution} + When you startup lisp, there are potential conflicts between the +class definition and the indexing records in the database. There are +some constraints to account for and some facilities to manage +how slots, class indices and +@item @strong{Dynamic Class Index Management} + It is possible to add and remove indexes from classes at runtime. +@item @strong{BTree Cursors} + If you need to do more than iterate over a collection, or you need +to delete elements of the collection as you iterate cursors are an +important data structure. They implement a variety of operators for +moving backward and forward over a btree, including ranged operations +and iterating of duplicate or unique values. +@item @strong{Indexed BTrees} + Indexed BTrees are just like BTrees, except it is possible to add +indexes which are BTrees who's values are primary keys in the parent +@code{indexed-btree}. This allows for multiple ordering and groupings +of the values of a BTree. +@item @strong{Using the Map Operators} + Mapping operators can be very efficient if properly utilized. +@item @strong{Handling Errors and Conditions} + There are a variety of errors that can occur in Elephant that need +to be dealt with by applications. +@item @strong{Deadlock Detection in Berkeley DB} + Berkeley DB requires an external process to detect deadlock +conditions among transactions. The :deadlock-detect keyword argument +to open-store for Berkeley DB specs will launch this process on most +lisps. +@item @strong{Using Multiple Stores} + Multiple store controllers can be open simultaneously. However it +does make the code more complex and you need to be careful about how +you use them to avoid crashes and other unpleasant side effects. +@item @strong{Custom Transaction Architecture} + You can implement your own version of with-transaction using the +underlying controller methods for starting, aborting and committing +transactions. You had better know what you are doing, however! +@item @strong{Using Multiple Threads and Processes} + What constraints must be accommodated to use Elephant data stores in +multiple threads? What capabilities are there to share data stores +among multiple processes or machines? +@end itemize +Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/30 14:34:34 1.4 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5 @@ -8,18 +8,19 @@ @menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. -* Reachability:: Determining liveness in a store. * Persistent objects:: All the dirt on persistent objects. -* Class indices:: In-depth discussion about indexing persistent indices. +* Class Indices:: In-depth discussion about indexing persistent indices. * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. -* Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls. +* Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Multi-repository Operation:: Specifying repositories. +* Multi-threaded Applications:: What considerations are required for safe multi-threading +* Multiple Processes and Distributed Applications:: Can elephant be run on multiple CPUs and multiple machines? * Repository Migration and Upgrade:: How to move objects from one repository to another. -* Garbage collection:: How to recover storage and OIDs in long-lived repositories. -* Performance tuning:: How to get the most from Elephant. +* Garbage Collection:: How to recover storage and OIDs in long-lived repositories. +* Performance Tuning:: How to get the most from Elephant. @end menu @node Persistent objects @@ -83,9 +84,15 @@ @code{with-open-controller} macro. Opening and closing a controller is very expensive. +@node Serialization details +@comment node-name, next, previous, up +@section Serialization details + +Empty. + @node Class Indices @comment node-name, next, previous, up -@section Class Indicies +@section Class Indices You can enable/disable class indexing for an entire class. When you disable indexing all references to instances of that class are lost. If you re-enable @@ -130,6 +137,10 @@ somewhat user customizable; documentation for this exists in the source file referenced above. +@node Querying persistent instances +@comment node-name, next, previous, up +@section Querying persistent instances + @node Using BTrees @comment node-name, next, previous, up @section Using BTrees @@ -147,10 +158,127 @@ blocks of data is relatively inexpensive after a seek and comparisons on objects that are stored in memory is cheap. +@node Secondary Indices +@comment node-name, next, previous, up +@section Secondary Indices + +Empty. + +@node Using Cursors +@comment node-name, next, previous, up +@section Using Cursors + +Empty. -@node Repository Migration +@node Transaction Details @comment node-name, next, previous, up -@section Repository Migration +@section Transaction Details + +;; Transaction architecture: +;; +;; User and designer considerations: +;; - *current-transaction* is reserved for use by dynamic transaction context. The default global +;; value must always be null (no transaction). Each backend can set it to a different parameter +;; within the dynamic context of an execute-transaction. +;; - Any closures returned from within a transaction cannot bind *current-transaction* +;; - Only a normal return value will result in the transaction being committed, any non-local exit +;; results in a transaction abort. If you want to do something more sophisticated, roll your own +;; using controller-start-transaction, etc. +;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc) +;; knowing that the transaction will be aborted +;; + +@node Multi-threaded Applications +@comment node-name, next, previous, up +@section Multi-threaded Applications + +Sleepycat plays well with threads and processes. The store controller +is thread-safe by default, that is, can be shared amongst threads. +This is set by the @code{:thread} key. Transactions may not be shared +amongst threads except serially. One thing which is NOT thread and +process safe is recovery, which should be run when no one is else is +talking to the database environment. Consult the Sleepycat docs for +more information. + +Elephant uses some specials to hold parameters and buffers. If you're +using a natively threaded lisp, you can initialize these specials to +thread-local storage by using the @code{run-elephant-thread} function, +assuming your lisp creates thread-local storage for let-bound +specials. (This functionality is currently broken) + +Persisting ordinary aggregate types (e.g. NOT persistent classes or +btrees) suffers from something called "merge-conflicts." Since +updating one value of an aggregate object requires the entire object +to be written to the database, in heavily threaded situations you may +overwrite changes another thread or process has committed. This is +not protected by transactions! + +Consider two processes operating on the same cons: + +@code{ +-----start---read--update-car--write--commit---------------- +-start------read--update-cdr-----------------write--commit-- +} + +Although the first process successfully committed its transaction, its +work (writing to the car) will be erased by the second transaction +(which writes both the car and cdr.) + +Persistent classes and persistent collections do not suffer from +merge-conflicts, since each slot / entry is a separate database entry. + +@node Multi-repository Operation +@comment node-name, next, previous, up +@section Multi-repository Operation + +Elephant now keeps a small hashtables that maps ``database specifications'' into +actual database connections. + +If a database spec is a string, it is assumed to be a BerkeleyDB path. +If it is a list, it is a assumed to be a CL-SQL connection specification. +For example: +@lisp +ELE-TESTS> *testdb-path* +"/home/read/projects/elephant/elephant/tests/testdb/" +ELE-TESTS> *testpg-path* +(:postgresql "localhost.localdomain" "test" "postgres" "") +ELE-TESTS> +@end lisp + +The tests now have a function @code{do-all-tests-spec} that take a spec and +based on its type attempt to open the correct kind of store controller and +perform the tests. + +The routine @code{get-controller} takes this specifiation. + +The basic strategy is that the ``database specification'' object is stored in +every persistent object and collection so that the repository can be found. + +In this way, objects that reside in different repositories can coexist within +the LISP object space, allowing data migration. + +;; Multiple stores + +;; Multiple store considerations: +;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to +;; work around: how to avoid writing one store with a transaction created in the context of another. +;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct; +;; this requirement may relax in the future +;; - The following macros accomodate multiple stores by requiring that execute-transaction return a +;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller +;; is the store instance it is associated with. A nested or ensured transaction is only indicated +;; in the call to execute transaction if the store controllers match, otherwise a new transaction +;; for that store is created + +@node Multiple Processes and Distributed Applications +@comment node-name, next, previous, up +@section Multiple Processes and Distributed Applications + +Can we do this? What do we have to say about this? + +@node Repository Migration and Upgrade +@comment node-name, next, previous, up +@section Repository Migration and Upgrade This version of Elephant supports migration between store controllers of any backend type. @@ -227,49 +355,17 @@ @code{*inhibit-slot-writes*} in your user method using @code{with-inhibited-slot-copy} a convenience macro. - -@node Threading +@node Garbage Collection @comment node-name, next, previous, up -@section Threading - -Sleepycat plays well with threads and processes. The store controller -is thread-safe by default, that is, can be shared amongst threads. -This is set by the @code{:thread} key. Transactions may not be shared -amongst threads except serially. One thing which is NOT thread and -process safe is recovery, which should be run when no one is else is -talking to the database environment. Consult the Sleepycat docs for -more information. - -Elephant uses some specials to hold parameters and buffers. If you're -using a natively threaded lisp, you can initialize these specials to -thread-local storage by using the @code{run-elephant-thread} function, -assuming your lisp creates thread-local storage for let-bound -specials. (This functionality is currently broken) - -Persisting ordinary aggregate types (e.g. NOT persistent classes or -btrees) suffers from something called "merge-conflicts." Since -updating one value of an aggregate object requires the entire object -to be written to the database, in heavily threaded situations you may -overwrite changes another thread or process has committed. This is -not protected by transactions! - -Consider two processes operating on the same cons: - -@code{ ------start---read--update-car--write--commit---------------- --start------read--update-cdr-----------------write--commit-- -} - -Although the first process successfully committed its transaction, its -work (writing to the car) will be erased by the second transaction -(which writes both the car and cdr.) +@section Garbage Collection -Persistent classes and persistent collections do not suffer from -merge-conflicts, since each slot / entry is a separate database entry. +GC is not implemented, but migration (@pxref{Repository Migration and +Upgrade}) will consolidate storage and recover OIDs which emulates GC. +No online solution is currently supported. -@node Performance Tips +@node Performance Tuning @comment node-name, next, previous, up -@section Performance Tips +@section Performance Performance is usually measured in transactions per second. Database reads are cheap. To get more transactions throughput, consider @@ -290,8 +386,8 @@ things. It is fast but consing with floats and doubles. YMMV with other values, though I've tried to make them fast. -Using @code{*auto-commit*} and not @code{with-transactions} is a great -way to have a huge number of transactions. You'll find that +Use @code{with-transactions} to avoid many automatic transactions, for +example you'll find that this construct @lisp (dotimes (i 1000) (add-to-root "key" "value")) @@ -300,9 +396,8 @@ is much slower than @lisp -(let ((*auto-commit* nil)) - (with-transaction () - (dotimes (i 1000) (add-to-root "key" "value")))) +(with-transaction () + (dotimes (i 1000) (add-to-root "key" "value")))) @end lisp since there's only 1 transaction in the latter. However storing @@ -319,32 +414,3 @@ mode is not suitable for use in web servers or other typically multi-threaded applications. -@node Multi-repository Operation -@comment node-name, next, previous, up -@section Multi-repository Operation - -Elephant now keeps a small hashtables that maps ``database specifications'' into -actual database connections. - -If a database spec is a string, it is assumed to be a BerkeleyDB path. -If it is a list, it is a assumed to be a CL-SQL connection specification. -For example: -@lisp -ELE-TESTS> *testdb-path* -"/home/read/projects/elephant/elephant/tests/testdb/" -ELE-TESTS> *testpg-path* -(:postgresql "localhost.localdomain" "test" "postgres" "") -ELE-TESTS> -@end lisp - -The tests now have a function @code{do-all-tests-spec} that take a spec and -based on its type attempt to open the correct kind of store controller and -perform the tests. - -The routine @code{get-controller} takes this specifiation. - -The basic strategy is that the ``database specification'' object is stored in -every persistent object and collection so that the repository can be found. - -In this way, objects that reside in different repositories can coexist within -the LISP object space, allowing data migration.

1 0

[elephant-cvs] CVS elephant
by ieslick 01 Apr '07

01 Apr '07

Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv29083 Modified Files: TODO elephant.asd Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/TODO 2007/03/30 14:34:34 1.76 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/01 14:33:23 1.77 @@ -18,7 +18,9 @@ - Verify db_deadlock for other lisps (launch and kill background program I/F) Bugs: +- Fix awkward serializer API - Support for asdf-install? +- Edi's patches & suggestions for windows Test coverage: - Clean up interface to tests @@ -26,14 +28,13 @@ - Multi-threading stress tests? Ensure that there are conflicts and lots of serialization happening concurrently to make sure that multi-threading is in good shape (Henrik's code) - Unicode tests - - Ensure that variable length UTF-8 reps are automatically stored as UTF-16 + - Ensure that variable length UTF-8 reps are automatically stored as UTF-16? - Class / DB sychronization tests Documentation: ~ License and copyright file headers -- Redo tutorial -- Proper user guide - Update install, build and test procedures +- Proper user guide - Upgrade, migration and other system level issues - Performance and design issues - More notes about transaction performance @@ -42,7 +43,6 @@ - Add notes about optimize-storage - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) -- Add document section about backend interface & developer decisions 0.6.1 - Features COMPLETED to date ---------------------------------- --- /project/elephant/cvsroot/elephant/elephant.asd 2007/03/30 23:36:52 1.40 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/01 14:33:23 1.41 @@ -304,6 +304,7 @@ (:file "serializer2") ;; 0.6.1 db's (:file "unicode2") (:file "migrate") + (:file "query") (:file "backend")) :serial t :depends-on (memutil utils)))))

1 0