Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv2451
Modified Files: TODO UPGRADE-BDB Log Message: Checkpoint checkin of db-lisp
--- /project/elephant/cvsroot/elephant/TODO 2007/02/05 19:33:10 1.45 +++ /project/elephant/cvsroot/elephant/TODO 2007/02/08 23:07:18 1.46 @@ -9,12 +9,12 @@ ALPHA RELEASE TASKS
Bug and feature fixes: +- Validate migration 0.6.0->0.6.1 +- Fix cur-del2 failure under SBCL (robert to reproduce and fix) ~ Resolve duplicate sorting guarantee in btree interface; currently supported - by BDB but not SQL and it is not tested in the regression suite (Robert) + by BDB but not SQL and it is not tested in the regression suite (Robert to comment) - Fix *dbconnection-spec* to support multiple controllers for multiple threads - for CLSQL backend -- Validate migration 0.6.0->0.6.1 -- Fix cur-del2 failure under SBCL + for CLSQL backend (Robert)
BDB Features/Cleanup: - Trace all paths to db-put or db-delete and ensure that there is a check or a @@ -57,7 +57,6 @@ - Ensure that variable length UTF-8 is automatically stored as UTF-16
Documentation: -- Migrate code base to Darcs and create tickets in TRAC - Add notes about with/ensure-transaction usage (abort & commit behavior on exit) - Add notes about optimize-storage - Add notes about deadlock-detect @@ -128,6 +127,9 @@
0.6.2 - Advanded work, low-hanging fruit (Summer '07) -------------------------------------------------- + +Migrate code base to Darcs and create feature/bug tickets in TRAC + Storage and Indexing: - Add :inverse-reader to slot options to create a named method that indexes into objects based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in? @@ -175,7 +177,22 @@ 0.7.0: Native Lisp Backend (beta), Fast In-Memory Operations ------------------------------------------------------------ - Full support for DCM or integration of DCM functionality - - Integrate prevalence-like in-memory database system for single image, multiple-thread operation + - Integrate prevalence-like in-memory database system for single image, + multiple-thread operation + - Richer set of policy decisions on per-class basis + - Concurrent mode (for backends that allow multiple processes to connect, current default) + - Single-user mode (cache values in instance slots for fast reads, write-through) + - Backing store mode (read/write to normal slots except on object creation or synch) + (in-memory slot indexing, on disk class) + (works for any backend) + - Backing-store mode + - Controller 'switches' + - NoSynch - allow transactions to be lost on failure but maintains consistency + instead of performance + - Upgrade overall functionality + - Solid garbage collection strategy + - 64-bit oids / 64-bit file sizes + - class templates stored and cached - (From Ben's e-mail) We are storing persistent objects incorrectly. They should be stored only as OIDs, and we should have a separate OID->class table. This way change-class can be handled correctly. This also non-trivially compresses storage @@ -183,24 +200,16 @@ [Ian comment: only problem with this is an extra access to oid table each time a class is deserialized and overall storage is constant. Would make it easy to invalidate objects though!] - - Richer set of policy decisions on per-class basis - - Concurrent mode (for backends that allow multiple processes to connect, current default) - - Single-user mode (cache values in instance slots for fast reads, write-through) - - Backing store mode (read/write to normal slots except on object creation or synch) - (in-memory slot indexing, on disk class) - (works for any backend) - - Backing-store mode - - Controller 'switches' - - NoSynch - allow transactions to be lost on failure but maintains consistency instead of performance - Usage model examples
-0.7.1 - Elephant/BDB/SQL Production Release +0.7.1 - Elephant BDB/SQL/Lisp Production Release -------------------------------------------------- - More work on testing, examples and documentation - Intent is for this to be a major, long-term supported release prior to work on the new backend (i.e. patches against this release for bugs rather than only available in latest development tree)
+ 0.8.0 - Supporting Tools Release -------------------------------------------------- - Add special support (if any) for persistent graph structures & queries --- /project/elephant/cvsroot/elephant/UPGRADE-BDB 2006/09/04 00:09:11 1.1 +++ /project/elephant/cvsroot/elephant/UPGRADE-BDB 2007/02/08 23:07:18 1.2 @@ -1,46 +1,42 @@ -Version 0.6.0 of Elephant using the BDB backend depends on Berkeley DB 4.3. As for September 3rd, 2006, -the current CVS is now dependant on Berkeley DB 4.4. If you already have a database based on 0.6.0 or a HEAD prior to September 3rd, you'll need to take the following steps to upgrade your BDB databases to work with my latest checkin. +Elephant 0.6.1 depends on Berkeley DB 4.5.
-(NOTE: Allegro users may have additional work to do, please check e-mail logs on this topic) +------------------------------------ +Upgrading from 0.6 / Berkeley DB 4.3 +------------------------------------
-1) Install BDB 4.4.20 or later just as you installed BDB 4.3 +1) Install BDB 4.5 (keep 4.3 around for now)
-2) Pull the latest HEAD from CVS +2) Update my-config.sexp to point to the appropriate BDB 4.5 directories
-3) Update config.lisp and Makefile in elephant root to point to the appropriate directories +3) Upgrade your database directory to 4.5
-4) Rebuild elephant C libraries +3a) Run db43_recover in your 0.6 database +3b) Optional: run db43_archive -d to remove all logs not part of a checkpoint + This will make catastrophic recovery impossible, but reduces the amount of data you + have to backup. +3c) Backup your db files and remaining logs +3d) Run db45_checkpoint -1 in your 0.6 database directory
-In the root directory: -> make clean -> make -> make bdb +4) Migrate 0.6 data to a new 0.6.1 database
-5) Upgrade your database directory (only log files need updating) +4a) Open your old database: (setf sc (open-store '(:BDB "/Users/me/db/ele060/"))) +4b) Run upgrade: (upgrade sc '(:BDB "/Users/me/db/ele061/"))
-From Sleepycat documentation: +5) Test your new application and report any bugs that arise to elephant-devel@common-lisp.net
- 1. Shut down the old version of the application. - 2. Run recovery on the database environment using the DB_ENV->open method or the db_recover utility. - 3. Remove any Berkeley DB environment using the DB_ENV->remove method or an appropriate system utility. - 4. Archive the database environment for catastrophic recovery. See Archival procedures for more information. - 5. Recompile and install the new version of the application. - 6. Force a checkpoint using the DB_ENV->txn_checkpoint method or the db_checkpoint utility. - 7. Restart the application. +NOTE 1: close-store may fail when closing the old 0.6 database, this is OK +NOTE 2: 64-bit lisps will not successfully upgrade 32-bit 0.6 databases. Use a 32-bit + version of your lisp to update to 0.6.1 and then open that database in your 64-bit + lisp. There should be no compatibility problems. Best to test your application on + a 32-bit lisp if you can, just to be sure.
-A known good procedure: +------------------------------------ +Upgrading from 0.5 / Berkeley DB 4.3 +------------------------------------
- 1. Cleanly exit lisp/elephant application - 2. Run 'db_recover' in database directory using 4.3 tools - 3. This will remove the environment - 3.b (optional) Run 'db_checkpoint -1' and 'db_archive -d' to checkpoint and update db files to latest log. This will - snapshot the DB and allow you to backup less data, but it makes catastrophic recovery to any time - before the snapshot impossible as you are deleting history with the '-d' option so exclude that if you - are conservative or aren't sure what you're doing. - 4. Copy your database files and all log files to a backup - 5. Run 'db_checkpoint -1' using 4.4 tools (ignore error message) - 6. Restart lisp, reload application and ensure that the latest elephant source has been fully recompiled +Follow the upgrade procedures outlined in Elephant 0.6.0 to migrate your database +from 0.5 to 0.6. Then follow the above procedures for upgradeing from an 0.6 database.
-6) Connect to your DB, all should be well! - -This procedure worked for my Mac OS X upgrade from BDB 4.3 using the latest HEAD on a very large, complex DB. +NOTE: It may not take much work to make 0.6.1 upgrade directly from 0.5.0. However +there are so few 0.5.0 users that it wasn't deemed worth the work given that +there's an upgrade path available. \ No newline at end of file