Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv14802

Modified Files: Makefile TODO config.lisp Added Files: UPGRADE-BDB Log Message: Berkeley DB Backend upgrade & compact API fn, bug fixes

--- /project/elephant/cvsroot/elephant/Makefile 2006/02/19 17:25:52 1.11 +++ /project/elephant/cvsroot/elephant/Makefile 2006/09/04 00:09:10 1.12 @@ -26,13 +26,17 @@ $(UTILSRC)/libmemutil.$(EXT): $(UTILSRC)/libmemutil.c gcc $(SHARED) -Wall -fPIC -O3 -o $@ $< -lm

+clean: + rm $(UTILSRC)/libmemutil.$(EXT) + rm $(BDBSRC)/libsleepycat.$(EXT) + # # NON BDB PLATFORMS SHOULDN'T NEED TO COMPILE LIBSLEEPYCAT # SO ONLY EDIT THIS IF YOU WANT TO USE BDB! #

# But I will assume that Linux is more common? -DB43DIR=/usr/local/BerkeleyDB.4.3/ +DB43DIR=/usr/local/BerkeleyDB.4.4/ # Dan Knapp contributed this line, which came form OS X? # DB43DIR=/sw # Other example paths --- /project/elephant/cvsroot/elephant/TODO 2006/04/26 17:53:43 1.22 +++ /project/elephant/cvsroot/elephant/TODO 2006/09/04 00:09:10 1.23 @@ -1,33 +1,19 @@

-April 23, 2006 +September 1st, 2006

-Ongoing release plan notes +Ongoing release plan notes:

-Features completed in 0.6.0: ------------------------------------ -x Add a class-indexing class option to the metaclass so we can maintain class instances - index without any secondary indices or indexed slots -x Upgrade solution to 0.6.0, DB properties & version tag for future upgrades -x Validate migration -x Documentation update -x Indexing tutorial and tutorial review - - -0.6.1 - performance, safety and portability --------------------------------------------------- +0.6.1 - performance, safety and portability (end of Summer?) +-----------------------------------------------------------

Bugs or Observations: -- BDB overwrite of values makes DB grow - [So far I can only find that it grows on the 2nd write, but not after that...artifact of - page allocation or caching of memory pools?] -- FEATURE: Investigate BDB record size; it's 2x larger than expected? - [Need a good test for this to follow up]

Multi-threading operation: -- Make elephant thread bound variables dynamic and modifiable by backends +- Make elephant thread-bound variables dynamic and modifiable by backends +- Ensure serialization is multi-threaded - Verify that operations such as indexing are thread safe

-Stability +Stability: - Review all the NOTE comments in the code - Remove build gensym warnings in sleepycat - Port elephant to closer-to-MOP to make it easier to support additional lisps (Both) @@ -62,10 +48,9 @@ *auto-index* is false and we're not in a transaction to help users avoid lockups in bdb? Should be able to turn off for performance but it will help catch missing with-transaction statemetns in user code. (Both) -- Determine how to detect deadlock conditions as an optional run-safe mode? -- Automatically run db_deadlock when opening a bdb backend? Requires path to +~ Automatically run db_deadlock when opening a bdb backend? Requires path to functions and ability to launch shell command. Closing the store stops the - sub-process. + sub-process. - Always support locks that timeout? Tradeoffs?

Performance: @@ -75,23 +60,35 @@ - Add dependency information into secondary index callback functions so that we can more easily compute which indices need to be updated to avoid the global remove/add in order to maintain consistency (Ian) -- Improve SQL serializer performance (Robert) +- Improve SQL serializer performance (Robert/Ian)

Indexing features: - Add :inverse-reader to slot options to create a named method that indexes into objects based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in?

-0.6.3 - Query & indexing expansion +Compliance and Efficiency: + - Update to support BDB 4.4 + - Add ability from within lisp to reclaim DB space after deleting btree key-value pairs + - Reclaim table storage on index drop + +0.6.1 - Features COMPLETED to date +---------------------------------- +x Determine how to detect deadlock conditions as an optional run-safe mode? +x BDB overwrite of values makes DB grow + [So far I can only find that it grows on the 2nd write, but not after that...artifact of + page allocation or caching of memory pools?] +x FEATURE: Investigate BDB record size; it's 2x larger than expected? + [Ditto above] + +0.6.2 - Query & indexing expansion (Fall '06) -------------------------------------------------- - - simple object query language (Ian - orthogonal, on main branch) + - Simple object query language (Ian - orthogonal, on main branch) - Add needed support (if any) for persistent graph structures & queries (Ian on a branch) - A wrapper around migration that emulates a stop-and-copy GC + - Fast serializer port & upgrade strategy

-0.6.4 - Compliance & Documentation +0.6.3 - Documentation & Tools (Winter '06) -------------------------------------------------- - - Update to support BDB 4.4 - - Add ability from within lisp to reclaim DB space after deleting btree key-value pairs - - Reclaim table storage on index drop - Tutorial example rethink: update the blog tutorial using indexed objects to create different views as well as integrating something like logging for admin or version control purposes. @@ -104,17 +101,17 @@ (a simple REPL tool to see what classes are in a repository and what state they're in...useful for long-lived repositories)

-0.6.5 - Additional datastructures? +0.6.4 - Additional datastructures (?) -------------------------------------------------- - Native BDB persistent hashes (easy; can do on SQL backends?) - Support for cheap persistent sets (medium? can do on SQL?)

Some placeholders & dreams features below... :)

-0.7+: Major features +0.7+: Major features (Winter '07) -------------------------------------------------- - - A backend controller for AllegroCache (Ian) - - Prevalence-like in-memory database system (Robert?) + - A native lisp backend controller (Ian) + - Integrate prevalence-like in-memory database system (Robert?) - Richer controller modes: - Single-user mode (cache values in instance slots for fast reads, write-through) - Prevalence mode (read/write to normal slots except on object creation or synch) @@ -124,20 +121,25 @@ - Controller 'switches' - NoSynch - allow transactions to be lost on failure but maintains consistency instead of performance

-0.8 - Lisp Backend? --------------------------------------------------- - - A native BTree implementation in CommonLisp (prototype on Allegro's BTree implementation for ACache) - - ======================================================== ========================================================

-Resolved issues: -- On class change, new slots should have their initform values pushed +June 20, 2006: + +Features completed in 0.6.0: +----------------------------------- +x Add a class-indexing class option to the metaclass so we can maintain class instances + index without any secondary indices or indexed slots +x Upgrade solution to 0.6.0, DB properties & version tag for future upgrades +x Validate migration +x Documentation update +x Indexing tutorial and tutorial review +x On class change, new slots should have their initform values pushed into the slot value as if the slot was being created the first time (currently this doesn't happen) [fixed in 0.6.0-rc1]

-Feb. 4, 2006 + +Feb. 4, 2006:

As of 0.5.0, we have seem to have a stable suite on ACL, SBCL, and OpenMCL. @@ -150,18 +152,14 @@ 3) The SQL serializer could be made more efficiently very easily, greatly enhancing the speed of that as a back-end.

- In general, Elephant is very usable and modestly robust, but heavier use, better documentation, and a good example application would help it a lot.

- - - - Merge in the todos from the source and the NOTES!

-October 19, 2005 + +October 19, 2005:

The SQL back-end stuff has only been tested with Postgress and SBCL. @@ -176,16 +174,15 @@ make things MUCH faster.

-new counters in 4.3 (october) - +Notes from original developers at time of handoff: +(minus stuff recorded elsewhwere) +-------------------------------------------------- understand the profiler / timer, tweak performance of CLOS stuff

tweak performance of transactions! dynamic-extent in CMUCL / SBCL.

-more documentation: texinfo NOTES. - equality joins have to be done on the lisp side: end-of-table is not the same as end-of-btree.

@@ -202,14 +199,6 @@ openmcl lispy pointer arithmetic (profile sap-alien, etc). profile CMUCL / SBCL sap arithmetic.

-performance hacks: class / slot to ID - -tests tests tests - -this is not particularly a bug but: if you redefine the -persistent-object class, you will mess up any existing -persistent classes you've made. - CMUCL, SBCL, Allegro? (NOT OpenMCL) can directly pass memory like foreign arrays. Use these instead of foreign arrays?

@@ -220,6 +209,3 @@ cursor-put : move the cursor after insert.

change :transient flag to an allocation type (fix CMUCL!) - -make update-class-for-redefined-class work. (persistent -slots are class allocated, this is bad.) --- /project/elephant/cvsroot/elephant/config.lisp 2006/04/30 01:03:48 1.3 +++ /project/elephant/cvsroot/elephant/config.lisp 2006/09/04 00:09:11 1.4 @@ -36,7 +36,7 @@ ;; for Fink (OS X) -- but I will assume Linux more common... ;; "/sw/lib/libdb-4.3.dylib" ;; a possible manual install - "/usr/local/BerkeleyDB.4.3/lib/libdb.dylib") + "/usr/local/BerkeleyDB.4.4/lib/libdb.dylib")

(defparameter *sleepycat-pthreads-path* #-linux nil ;; don't open on non-linux environments

--- /project/elephant/cvsroot/elephant/UPGRADE-BDB 2006/09/04 00:09:12 NONE +++ /project/elephant/cvsroot/elephant/UPGRADE-BDB 2006/09/04 00:09:12 1.1 Version 0.6.0 of Elephant using the BDB backend depends on Berkeley DB 4.3. As for September 3rd, 2006, the current CVS is now dependant on Berkeley DB 4.4. If you already have a database based on 0.6.0 or a HEAD prior to September 3rd, you'll need to take the following steps to upgrade your BDB databases to work with my latest checkin.

(NOTE: Allegro users may have additional work to do, please check e-mail logs on this topic)

1) Install BDB 4.4.20 or later just as you installed BDB 4.3

2) Pull the latest HEAD from CVS

3) Update config.lisp and Makefile in elephant root to point to the appropriate directories

4) Rebuild elephant C libraries

In the root directory:

make clean make make bdb

5) Upgrade your database directory (only log files need updating)

From Sleepycat documentation:

1. Shut down the old version of the application. 2. Run recovery on the database environment using the DB_ENV->open method or the db_recover utility. 3. Remove any Berkeley DB environment using the DB_ENV->remove method or an appropriate system utility. 4. Archive the database environment for catastrophic recovery. See Archival procedures for more information. 5. Recompile and install the new version of the application. 6. Force a checkpoint using the DB_ENV->txn_checkpoint method or the db_checkpoint utility. 7. Restart the application.

A known good procedure:

1. Cleanly exit lisp/elephant application 2. Run 'db_recover' in database directory using 4.3 tools 3. This will remove the environment 3.b (optional) Run 'db_checkpoint -1' and 'db_archive -d' to checkpoint and update db files to latest log. This will snapshot the DB and allow you to backup less data, but it makes catastrophic recovery to any time before the snapshot impossible as you are deleting history with the '-d' option so exclude that if you are conservative or aren't sure what you're doing. 4. Copy your database files and all log files to a backup 5. Run 'db_checkpoint -1' using 4.4 tools (ignore error message) 6. Restart lisp, reload application and ensure that the latest elephant source has been fully recompiled

6) Connect to your DB, all should be well!

This procedure worked for my Mac OS X upgrade from BDB 4.3 using the latest HEAD on a very large, complex DB.