Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv12311

Modified Files: NOTES TODO elephant.asd Log Message:

Significant additions to the 0.6.0 release on the trunk. Updates to documentation, 0.5.0 compliance, final on 0.6.0 features. There are one or two BDB interactions on migration to work out but this should be a nearly code complete 0.6.0 release. Please start testing and evaluating this - especially the ability to open and tag 0.5.0 databases.

Features: - Database version tagging - Support for 0.5.0 namespaces & databases - New migration system - class indexing without slot indexing - various bug fixes - reverted fast allegro/sbcl string support to allow 0.5.0 databases to work correctly. I couldn't find a good way to work around this without creating infinite headaches - validated that running db_deadlock will stop all lisp freezes that I've encountered. This has to be run each time a DB environment is opened/created so eventually should be made part of the open-controller functionality for the BDB backend

--- /project/elephant/cvsroot/elephant/NOTES 2006/01/25 21:52:54 1.6 +++ /project/elephant/cvsroot/elephant/NOTES 2006/04/26 17:53:43 1.7 @@ -3,10 +3,9 @@ GENERAL -------

-this has been optimized for use with CMUCL / SBCL. it has -been tested and somewhat optimized for allegro. OpenMCL is -definitely also a target. Lispworks is a target as well but -less so: i don't have access to it. +this has been optimized for use with CMUCL / SBCL / Allegro. +OpenMCL has been minimally supported. Lispworks is a target as well +but less so as the developers don't have access to it.

Theoretically one can port this to any lisp with a decent FFI and MOP. However since those are two of the less @@ -19,7 +18,7 @@ persistent meta-object, persistent collections controller serializer -SLEEPYCAT package +memutils package UFFI / implementation specific stuff libsleepycat.so Sleepycat 4.2/3 --- /project/elephant/cvsroot/elephant/TODO 2006/03/07 14:12:22 1.21 +++ /project/elephant/cvsroot/elephant/TODO 2006/04/26 17:53:43 1.22 @@ -1,13 +1,27 @@

-Feb 23, 2006 +April 23, 2006

-Release plan in-discussion with Robert and Ian +Ongoing release plan notes + +Features completed in 0.6.0: +----------------------------------- +x Add a class-indexing class option to the metaclass so we can maintain class instances + index without any secondary indices or indexed slots +x Upgrade solution to 0.6.0, DB properties & version tag for future upgrades +x Validate migration +x Documentation update +x Indexing tutorial and tutorial review

-0.6.0 - Adding default class/slot indexing; modularize backends -- Documentation update -- Tutorial review

0.6.1 - performance, safety and portability +-------------------------------------------------- + +Bugs or Observations: +- BDB overwrite of values makes DB grow + [So far I can only find that it grows on the 2nd write, but not after that...artifact of + page allocation or caching of memory pools?] +- FEATURE: Investigate BDB record size; it's 2x larger than expected? + [Need a good test for this to follow up]

Multi-threading operation: - Make elephant thread bound variables dynamic and modifiable by backends @@ -21,55 +35,63 @@ stored only as OIDs, and we should have a separate OID->class table. This way change-class can be handled correctly. This also non-trivially compresses storage in the database as we only need to store oids rather than serialized class names. + [Ian comment: only problem with this is an extra access to oid table each time a + class is deserialized and overall storage is constant. Would make it easy to + invalidate objects though!] - Delete persistent slot values from the slot store with remove-kv to ensure that there's no data left lying around if you define then redefine a class and add back a persistent slot name that you thought was deleted and it gets the old - value + value by default.

Stores: +- Think through default *store-controller* vs. explicit parameter passing + referencing all over the APIs - Think about dynamic vs. object based store & transaction resolution? - - Error checking when mixed + - Perform error checking when mixed - Current store specific *current-transaction* stack -- Think through default vs. explicit store referencing all over the APIs (Both) - Throw condition when store spec is invalid, etc

Transactionalism: - Cleaner failure modes if operations are performed without repository or without transaction or auto-commit (Both) + +BDB Features: - Trace all paths to db-put or db-delete and ensure that there is a check or a default with-transaction around the primitive components - write a document clarifying transaction design & assumptions in the backend] Add asserts if *auto-index* is false and we're not in a transaction to help users avoid lockups in bdb? Should be able to turn off for performance but it will help catch missing with-transaction statemetns in user code. (Both) -- BDB: determine how to detect deadlock conditions as an optional run-safe mode? (?) - Does BDB have timeouts enabled on select? (Ian) +- Determine how to detect deadlock conditions as an optional run-safe mode? +- Automatically run db_deadlock when opening a bdb backend? Requires path to + functions and ability to launch shell command. Closing the store stops the + sub-process. +- Always support locks that timeout? Tradeoffs?

Performance: - Metering and understanding locking issues. Large transactions seem to use a lot of locks. In general understanding how to use Sleepycat efficiently seems like a good thing. (From Ben) -- Reclaim table storage on index drop (Ian) - Add dependency information into secondary index callback functions so that we can more easily compute which indices need to be updated to avoid the global remove/add in order to maintain consistency (Ian) +- Improve SQL serializer performance (Robert)

Indexing features: -- Add a class-indexing class option to the metaclass so we can maintain class instances - index without any secondary indices or indexed slots (Ian) - Add :inverse-reader to slot options to create a named method that indexes into objects based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in?

-Bugs: -- anything else reported against 0.5.0/0.6.0 - 0.6.3 - Query & indexing expansion +-------------------------------------------------- - simple object query language (Ian - orthogonal, on main branch) - Add needed support (if any) for persistent graph structures & queries (Ian on a branch) - A wrapper around migration that emulates a stop-and-copy GC

0.6.4 - Compliance & Documentation +-------------------------------------------------- - Update to support BDB 4.4 + - Add ability from within lisp to reclaim DB space after deleting btree key-value pairs + - Reclaim table storage on index drop - Tutorial example rethink: update the blog tutorial using indexed objects to create different views as well as integrating something like logging for admin or version control purposes. @@ -83,12 +105,14 @@ what state they're in...useful for long-lived repositories)

0.6.5 - Additional datastructures? +-------------------------------------------------- - Native BDB persistent hashes (easy; can do on SQL backends?) - Support for cheap persistent sets (medium? can do on SQL?)

Some placeholders & dreams features below... :)

0.7+: Major features +-------------------------------------------------- - A backend controller for AllegroCache (Ian) - Prevalence-like in-memory database system (Robert?) - Richer controller modes: @@ -101,9 +125,13 @@ - NoSynch - allow transactions to be lost on failure but maintains consistency instead of performance

0.8 - Lisp Backend? +-------------------------------------------------- - A native BTree implementation in CommonLisp (prototype on Allegro's BTree implementation for ACache)

+======================================================== +======================================================== + Resolved issues: - On class change, new slots should have their initform values pushed into the slot value as if the slot was being created the first time --- /project/elephant/cvsroot/elephant/elephant.asd 2006/02/22 20:18:51 1.16 +++ /project/elephant/cvsroot/elephant/elephant.asd 2006/04/26 17:53:43 1.17 @@ -94,7 +94,7 @@ :serial t) (:module elephant :components - ((:file "elephant") + ((:file "package") (:file "config" :pathname "../../config.lisp") (:file "variables") #+cmu (:file "cmu-mop-patches")