Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv17940

Modified Files: TODO Log Message: Further rethink of roadmap and TODO tasks

--- /project/elephant/cvsroot/elephant/TODO 2006/09/04 05:01:05 1.24 +++ /project/elephant/cvsroot/elephant/TODO 2006/09/04 05:42:43 1.25 @@ -8,28 +8,19 @@

Bugs or Observations:

-Multi-threading operation: -- Make elephant thread-bound variables dynamic and modifiable by backends -- Ensure serialization is multi-threaded -- Verify that operations such as indexing are thread safe - Stability: - Review all the NOTE comments in the code - Remove build gensym warnings in sleepycat -- Port elephant to closer-to-MOP to make it easier to support additional lisps (Both) -- (From Ben's e-mail) We are storing persistent objects incorrectly. They should be - stored only as OIDs, and we should have a separate OID->class table. This way - change-class can be handled correctly. This also non-trivially compresses storage - in the database as we only need to store oids rather than serialized class names. - [Ian comment: only problem with this is an extra access to oid table each time a - class is deserialized and overall storage is constant. Would make it easy to - invalidate objects though!] +- Remove sleepycat name. Change sleepycat to db-bdb to reflect oracle ownership and avoid + confusion for new users - Delete persistent slot values from the slot store with remove-kv to ensure that there's no data left lying around if you define then redefine a class and add back a persistent slot name that you thought was deleted and it gets the old value by default. +- Cleaner failure modes if operations are performed without repository or without + transaction or auto-commit (Both)

-Stores: +Store variables: - Think through default *store-controller* vs. explicit parameter passing referencing all over the APIs - Think about dynamic vs. object based store & transaction resolution? @@ -37,57 +28,74 @@ - Current store specific *current-transaction* stack - Throw condition when store spec is invalid, etc

-Transactionalism: -- Cleaner failure modes if operations are performed without repository or without - transaction or auto-commit (Both) +Multi-threading operation: +- Make elephant threads appropriately bind dynamic variables +- Verify that operations such as indexing are thread safe

BDB Features: +~ Automatically run db_deadlock when opening a bdb backend? Requires path to + functions and ability to launch shell command. Closing the store stops the + sub-process. +- Always support locks that timeout? Tradeoffs? +- Roll deprecation of *auto-commit* through code base so leaf functions stop referring to it - Trace all paths to db-put or db-delete and ensure that there is a check or a default with-transaction around the primitive components - write a document clarifying transaction design & assumptions in the backend] Add asserts if *auto-index* is false and we're not in a transaction to help users avoid lockups in bdb? Should be able to turn off for performance but it will help catch missing with-transaction statemetns in user code. (Both) -~ Automatically run db_deadlock when opening a bdb backend? Requires path to - functions and ability to launch shell command. Closing the store stops the - sub-process. -- Always support locks that timeout? Tradeoffs? +- Figure out how to compact a specific btree and/or key-range using optimize-storage. + Probably need to update keyword part of the API + +Indexing efficiency and policies: +- Add :inverse-reader to slot options to create a named method that indexes into objects + based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in? +- Reclaim table storage on index drop? It's nice to be able to reconnect sometimes! + Perhaps an API command that allows explicit dropping of tables for a class and a policy + parameter that determines if this is the default? +- Should we delete slot-values in the db when redefining classes, currently those values + stay around - probably indefinitely unless we GC

Performance: - Metering and understanding locking issues. Large transactions seem - to use a lot of locks. In general understanding how to use Sleepycat + to use a lot of locks. In general understanding how to use Berkeley DB efficiently seems like a good thing. (From Ben) - Add dependency information into secondary index callback functions so that we can more easily compute which indices need to be updated to avoid the global remove/add in order to maintain consistency (Ian) - Improve SQL serializer performance (Robert/Ian)

-Indexing features: -- Add :inverse-reader to slot options to create a named method that indexes into objects - based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in? - -Compliance and Efficiency: - - Update to support BDB 4.4 - - Add ability from within lisp to reclaim DB space after deleting btree key-value pairs - - Reclaim table storage on index drop - - Should we delete slot-values in the db when redefining classes, currently those values - stay around - probably indefinitely unless we GC +Test coverage: +- Test for optimize storage method (just add probe-file methods to get file size) +- Multi-threading stress tests? Ensure that there are conflicts and lots of serialization + happening concurrently to make sure that multi-threading is in good shape + +Documentation: +- Add notes about with-transaction usage (abort & commit behavior on exit) +- Add notes about optimize-storage +- Add notes about new BDB 4.4 *auto-commit* behavior. Default for entire store-controller, + will auto create a transaction if none is active if open with :auto-commit t or will + never auto-commit (regardless of operator flags) if it is not. Make sure open-store + defaults to auto-commit and there is a flag to turn it off.

0.6.1 - Features COMPLETED to date ---------------------------------- +x Ensure serialization is multi-threaded and efficient x Determine how to detect deadlock conditions as an optional run-safe mode? x BDB overwrite of values makes DB grow [So far I can only find that it grows on the 2nd write, but not after that...artifact of page allocation or caching of memory pools?] x FEATURE: Investigate BDB record size; it's 2x larger than expected? [Ditto above] +x Update to support BDB 4.4 + x Add ability from within lisp to reclaim DB space after deleting btree key-value pairs

-0.6.2 - Query & indexing expansion (Fall '06) + +0.6.2 - Advanded work, low-hanging fruit (Fall '06) -------------------------------------------------- - - Simple object query language (Ian - orthogonal, on main branch) - - Add needed support (if any) for persistent graph structures & queries (Ian on a branch) + - Port elephant to closer-to-MOP to make it easier to support additional lisps and to + seriously clean up metaclasses.lisp and classes.lisp protocols - A wrapper around migration that emulates a stop-and-copy GC - - Fast serializer port & upgrade strategy

0.6.3 - Documentation & Tools (Winter '06) -------------------------------------------------- @@ -99,21 +107,19 @@ - A guide to dealing with multiple open stores - A guide to performance - An overview of licensing issues... - - Repository browser (Ian - orthogonal, on main branch) - (a simple REPL tool to see what classes are in a repository and - what state they're in...useful for long-lived repositories)

-0.6.4 - Additional datastructures (?) +0.7.0: Fast In-Memory Database (Not backwards compatible) -------------------------------------------------- - - Native BDB persistent hashes (easy; can do on SQL backends?) - - Support for cheap persistent sets (medium? can do on SQL?) - -Some placeholders & dreams features below... :) - -0.7+: Major features (Winter '07) --------------------------------------------------- - - A native lisp backend controller (Ian) - - Integrate prevalence-like in-memory database system (Robert?) + - Integrate prevalence-like in-memory database system + - Fast serializer port w/ upgrade strategy and prevalence like storage solution + - Further improve SQL 64-bit serialization performance (if possible) + - (From Ben's e-mail) We are storing persistent objects incorrectly. They should be + stored only as OIDs, and we should have a separate OID->class table. This way + change-class can be handled correctly. This also non-trivially compresses storage + in the database as we only need to store oids rather than serialized class names. + [Ian comment: only problem with this is an extra access to oid table each time a + class is deserialized and overall storage is constant. Would make it easy to + invalidate objects though!] - Richer controller modes: - Single-user mode (cache values in instance slots for fast reads, write-through) - Prevalence mode (read/write to normal slots except on object creation or synch) @@ -122,6 +128,31 @@ - Concurrent mode (for backends that allow multiple processes to connect, current default) - Controller 'switches' - NoSynch - allow transactions to be lost on failure but maintains consistency instead of performance + - Usage model examples + +0.7.1 - Elephant/BDB/SQL Production Release +-------------------------------------------------- + - More work on testing, examples and documentation + - Intent is for this to be a major, long-term supported release prior + to work on the new backend + +0.7.2 - Additional Tools +-------------------------------------------------- + - Add needed support (if any) for persistent graph structures & queries (Ian on a branch) + - Simple object query language (Ian - orthogonal, on main branch) + - Repository browser (Ian - orthogonal, on main branch) + (a simple REPL tool to see what classes are in a repository and + what state they're in...useful for long-lived repositories) + +0.8.0 - Native Backend & Datastructure Library ( +-------------------------------------------------- + - A native lisp backend controller (Ian) + - Native BDB persistent hashes (easy; can do on SQL backends?) + - Support for cheap persistent sets (medium? can do on SQL?) + - Usage model examples + + +

======================================================== ========================================================