Update of /project/elephant/cvsroot/elephant/doc In directory common-lisp:/tmp/cvs-serv874
Modified Files: elephant.texinfo tutorial.texinfo Added Files: installation.texinfo Log Message: Improving the documentation with some stuff about the tests, I hope.
--- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2005/11/23 17:51:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2006/01/24 20:37:43 1.3 @@ -43,7 +43,7 @@ * Introduction:: Introducing Elephant! * Tutorial:: A leisurely walk-through. * Reference:: API documentation. -* SQL back-end:: CL-SQL based implementation +* Installation:: Installation and test-suite procedures and issues * Design Notes:: Internals. * Copying:: Your rights and freedoms. * Concept Index:: @@ -57,7 +57,7 @@ @include tutorial.texinfo @include reference.texinfo @include notes.texinfo -@include sql-backend.texinfo +@include installation.texinfo @include copying.texinfo
@node Concept Index --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2006/01/24 18:25:00 1.3 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2006/01/24 20:37:43 1.4 @@ -49,10 +49,12 @@ your store! We'll assume in this tutorial you created a folder @code{testdb} in the current directory.
-Assuming you've managed to install Elephant properly, +Assuming you've managed to install Elephant properly, and +are using a BerkeleyDB installation.
@lisp * (asdf:operate 'asdf:load-op :elephant) +* (asdf:operate 'asdf:load-op :ele-bdb) @end lisp
will load the relevant files.
--- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2006/01/24 20:37:43 NONE +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2006/01/24 20:37:43 1.1 @c -*-texinfo-*-
@node Installation @comment node-name, next, previous, up @chapter Installation @cindex Installation
@menu * Installation Basics:: Basic installation * Test-Suites:: Running the test suites * SQL-Introduction:: The design and status of the SQL back-end extention. * Extention Status:: The current status of the SQL back-end extention. * Multi-repository Operation:: Specifying repositories * Setting up PostGres:: An example * Repository Migration:: How to move objects from one repository to another @end menu
@node Installation Basics @comment node-name, next, previous, up @section Installation
Please see the file ``INSTALL'' in the source distribution for more precise information; this is an overview.
Installation of Elephant itself is easy because of the asdf system. Just execute: @lisp (asdf:operate 'asdf:load-op :elephant) @end lisp
However, Elephant cannot function without a back-end repository. Elephant presents exactly the same API no matter what you choose as a repository. However, you have to use asdf to load the code that interfaces to particular repository system.
The basic choices are to use the BerkeleyDB system or a SQL based system. You must perform one of these: @lisp (asdf:operate 'asdf:load-op :ele-clsql) (asdf:operate 'asdf:load-op :ele-bdb) @end lisp
If you choose a SQL based system, you may have to load a specific package for that system, such as:
@lisp (asdf:operate 'asdf:load-op :ele-sqlite3) @end lisp or, for Postgres, @lisp (asdf:oos 'asdf:load-op :clsql-postgresql-socket) @end lisp
You will have to have the CL-SQL package installed. Following the documentation for CL-SQL under the section ``How CLSQL finds and loads foreign libraries'' you may need to do something like: @lisp (clsql:push-library-path "/usr/lib/") @end lisp
before doing @lisp (asdf:oos 'asdf:load-op :clsql-postgresql-socket) @end lisp
in order for clsql to find the PostGres library libpq.so, for example.
Without modifcation, Elephant uses this as it's lib path: @lisp /usr/local/share/common-lisp/elephant-0.3/ @end lisp
So you could put a symbolic link to libpq.so there, where libmemutil.so and libsleepycat.so will also reside.
Elephant is designed to allow multi-repository operation; so you could concievably use two or more repositories at the same time. More particularly, you can seamlessly migrate your data from one repository to a different one at a later date. In a long duration project, this might occur because of a licensing or performance issue with a particular respository.
@node Test-Suites @comment node-name, next, previous, up @section Test-Suites
Elephant is moderately mature. Hopefully, it will work out-of-the-box for you.
However, if you are using an LISP implementation different than the ones on which it is developed and maintained (currently OpenMCL, SBCL, and ACL), or as the repositories evolve, or just because of mistakes, you may need to run the test suites. If you report a bug, we will ask you to run these tests and report the output. Running them when you first install things may give you a sense of confidence and understanding that makes it worth the trouble.
There are three files that execute the tests. You should choose one as a starting point based on what backend(s) you are using. If using BerekleyDB, use @lisp BerkeleyDB-tests.lisp @end lisp
If using both, use both of the above and also use: @lisp MigrationTests.lisp @end lisp
The text of this file is included here to give the casual reader an idea of how elepant test can be run in general: @lisp ;; This file is an example of how to perform the ;; migration tests. You will have to modify it ;; slightly depending on the systems that want to test... ;; You can test migration even between two BDB respositories if you wish (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) (asdf:operate 'asdf:load-op :clsql-postgresql-socket) (asdf:operate 'asdf:load-op :ele-bdb) (asdf:operate 'asdf:load-op :elephant-tests) ;; For sqlite-3.. ;; (asdf:operate 'asdf:load-op :ele-sqlite3)
(in-package "ELEPHANT-TESTS")
;; The primary and secondary test-paths are ;; use for the migration tests.
;; This this configuration for testing between BDB and SQL.... (setq *test-path-primary* *testpg-path*) ;; (setq *test-path-primary* *testsqlite3-path*) (setq *test-path-secondary* *testdb-path*)
;; This this configuration for testing from one BDB repository to another... (setq *test-path-primary* *testdb-path*) ;; (setq *test-path-primary* *testsqlite3-path*) (setq *test-path-secondary* *testdb-path2*)
(do-migrate-test-spec *test-path-primary*) @end lisp
The appropriate test should execute for you with no errors. If you get errors, you may wish to report it the @code{ elephant-devel at common-lisp.net} email list.
@node SQL-Introduction @comment node-name, next, previous, up @section SQL-Introduction
Although originally designed as an interface to the BerkeleyDB system, the original Elephant system has been experimenetally extended to support the use of relational database management systems as the implementation of the persistent store. This relies on Kevin Rosenberg's CL-SQL interface to relational systems.
Although the BerkeleyDB system is an ideal object store for LISP objects, one might prefer the licensing of a different system. For example, at the time of this writing, it is my interpretation that one cannot use the BerkeleyDB system behind a public website http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source.
Neither the PostGres DBMS nor SQLite 3 has any such restriction. Elephant itself is released under the GPL. It is somewhat debatable if the GPL allows one to construct to construct a non-open-source web application but the preponderance of opinion appears to be that it does. Thefore using Elephant and the other GPLed software that it depends upon allows one to host a a non open-source web application. This might be a reason to use Elephant on PostGres of SQLite rather than Elephant on BerkeleyDB.
Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application needs to use the truly relational aspects of those systems, preference for the tools associated with those systems, etc.
The SQL back-end extention of Elephant provides a function for migrating data seamlessly between repositories. That is, one can quite easily move data from a BerkeleyDB repository to a PostGres repository, and vice versa. In fact, one of the most important aspects of the extention is that it makes Elephant a multi-repository system, rather than a single repository system, as addition to allowing different implementation strategies for those repositories. This offers at least the possiblity than once can develop using one backend, for example BerkeleyDB, and then later move to MySQL.
At the time of this writing, the basic strategy for the SQL implementation is quite simple. The same serializer used for the Sleepycat implementation is employed, the byte-string is base64 encoded, and placed in a single table which is managed by Elephant.
As of Elephant 0.3, Elephant has been tested to work with both Postgres, and SQLite 3, thanks do Dan Knapp.
@node Extention Status @comment node-name, next, previous, up @section Extention Status
As far as is known at this writing, all functionality except nested transaction support and cursor-put's that is supported by the BerkeleyDB backend is supported by the CL-SQL based back-end. Concurrency and transaction atomicity has not been tested well for the CL-SQL based system.
Additionally, it is NOT the case that the Elephant system currently provides transaction support across multiple repositories; it provides the transaction support provided by the underlying repository to the user in a per-repository basis.
The PostGres backend is as currently employed is about 5 times slower than the BerkeleyDB backend. This could probably change with continued development.
CL-SQL supports a lot of DBMS systems, but only PostGres has been tested.
The SQL back-end extention has only been tested under SBCL 0.8.18.
The SQL back-end is as easy to use as the BerkeleyDB back-end. However, the multi-repository version somewhat complicates the underlying persistent object management. At the time of this writing, the community has not decided if this extention will be a part of Elephant proper or a separate branch; if it is not made a part of Elephant proper, a user might prefer the simpler (and better maintained?) system if they only want to use the BerkeleyDB back-end.
@node Multi-repository Operation @comment node-name, next, previous, up @section Multi-repository Operation
Elephant now keeps a small hashtables that maps ``database specifications'' into actual database connections.
If a database spec is a string, it is assumed to be a BerkeleyDB path. If it is a list, it is a assumed to be a CL-SQL connection specification. For example: @lisp ELE-TESTS> *testdb-path* "/home/read/projects/elephant/elephant/tests/testdb/" ELE-TESTS> *testpg-path* (:postgresql "localhost.localdomain" "test" "postgres" "") ELE-TESTS> @end lisp
The tests now have a function @code{do-all-tests-spec} that take a spec and based on its type attempt to open the correct kind of store controller and perform the tests.
The routine @code{get-controller} takes this specifiation.
The basic strategy is that the ``database specification'' object is stored in every persistent object and collection so that the repository can be found.
In this way, objects that reside in different repositories can coexist within the LISP object space, allowing data migration.
@node Setting up PostGres @comment node-name, next, previous, up @section Setting up PostGres
To set up a PostGres based back end, you should:
@enumerate @item Install postgres and make sure postmaster is running.
@item Create a database called ``test'' and set its permissions to be reached by whatever connection specification you intend to use. The tests use:
@lisp (defvar *testpg-path* '(:postgreql "localhost.localdomain" "test" "postgres" "")) @end lisp
meaning that connections must be allowed to the database test, user ``postgres'', no password, connected from the same machine ``localhost.localdomain''. (This would be changed to something more secure in a real application.) Typically you edit the file : pg_hba.conf to enable various kinds of connections in postgres.
@item Be sure to enable socket connection to postgres when you invoke the postmaster.
@item Test that you can connect to the database with these credentials by running:
@code{ psql -h 127.0.0.1 -U postgres test}
Before you attempt to connect with Elephant. @end enumerate
meaning that connections must be allowed to the database test, user ``postgres'', no password, connected from the same machine ``localhost.localdomain''. (This would be changed to something more secure in a real application.)
Furthermore, you must grant practically all creation/read/write privileges to the user postgres on this schema, so that it can construct the tables it needs.
Upon first opening a CL-SQL based store controller, the tables, indexes, sequences, and so on needed by the Elephant system will be created in the schema named ``test'' automatically.
To run the tests, execute:
@lisp (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) (asdf:oos 'asdf:load-op :clsql-postgresql-socket) (in-package "ELEPHANT-TESTS") (do-all-tests-spec *testpg-path*) @end lisp
This should produce a small number of errors (about 7) for those test having to do with migration and the BerkeleyDB system specifically.
If you execute:
@lisp (asdf:operate 'asdf:load-op :ele-bdb) @end lisp
Then connection to the BerkeleyDB system will be enabled, and you should be able to execute both
@lisp (do-all-tests-spec *testpg-path*) (do-all-tests-spec *testdb-path*) @end lisp
with no errors in either case.
At present the system has only been tested under PostGres. Some code parametrization would be required to work with other databases.
Setting up SQLite3 is even easier. Install SQLite3 (I had to use the source rather than the binary install, in order to get the dynamic libraries constructed.)
An example use of SQLLite3 would be: @lisp (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) (asdf:operate 'asdf:load-op :ele-sqlite3) (in-package "ELEPHANT-TESTS") (setq *test-path-primary* '(:sqlite3 "testdb")) (do-all-tests-spec *test-path-primary*) @end lisp
The file RUNTESTS.lisp, although possibly not exactly what you want, contains useful example code.
You can of course migrate between the three currently supported repository strategies in any combination: BDB, Postgresql, and SQLite3.
In all probability, other relational datbases would be very easy to support but have not yet been tested. The basic pattern of the ``path'' specifiers is (cons clsqal-database-type-symbol (normal-clsql-connection-specifier)).
@node Repository Migration @comment node-name, next, previous, up @section Repository Migration
This version of Elephant supports migration betwen store controllers, whether of the same implementation strategy or not.
The tests @code{migrate1} - @code{migrate5} are demonstrations of this techinque.
The functions for performing these migrations are:
@code{migraten-pobj}
The name of this function is meant to imply that it is destructive of the object in question, mutating it to
[23 lines skipped]