Update of /project/elephant/cvsroot/elephant
In directory clnet:/tmp/cvs-serv17940
Modified Files:
TODO
Log Message:
Further rethink of roadmap and TODO tasks
--- /project/elephant/cvsroot/elephant/TODO 2006/09/04 05:01:05 1.24
+++ /project/elephant/cvsroot/elephant/TODO 2006/09/04 05:42:43 1.25
@@ -8,28 +8,19 @@
Bugs or Observations:
-Multi-threading operation:
-- Make elephant thread-bound variables dynamic and modifiable by backends
-- Ensure serialization is multi-threaded
-- Verify that operations such as indexing are thread safe
-
Stability:
- Review all the NOTE comments in the code
- Remove build gensym warnings in sleepycat
-- Port elephant to closer-to-MOP to make it easier to support additional lisps (Both)
-- (From Ben's e-mail) We are storing persistent objects incorrectly. They should be
- stored only as OIDs, and we should have a separate OID->class table. This way
- change-class can be handled correctly. This also non-trivially compresses storage
- in the database as we only need to store oids rather than serialized class names.
- [Ian comment: only problem with this is an extra access to oid table each time a
- class is deserialized and overall storage is constant. Would make it easy to
- invalidate objects though!]
+- Remove sleepycat name. Change sleepycat to db-bdb to reflect oracle ownership and avoid
+ confusion for new users
- Delete persistent slot values from the slot store with remove-kv to ensure that
there's no data left lying around if you define then redefine a class and add
back a persistent slot name that you thought was deleted and it gets the old
value by default.
+- Cleaner failure modes if operations are performed without repository or without
+ transaction or auto-commit (Both)
-Stores:
+Store variables:
- Think through default *store-controller* vs. explicit parameter passing
referencing all over the APIs
- Think about dynamic vs. object based store & transaction resolution?
@@ -37,57 +28,74 @@
- Current store specific *current-transaction* stack
- Throw condition when store spec is invalid, etc
-Transactionalism:
-- Cleaner failure modes if operations are performed without repository or without
- transaction or auto-commit (Both)
+Multi-threading operation:
+- Make elephant threads appropriately bind dynamic variables
+- Verify that operations such as indexing are thread safe
BDB Features:
+~ Automatically run db_deadlock when opening a bdb backend? Requires path to
+ functions and ability to launch shell command. Closing the store stops the
+ sub-process.
+- Always support locks that timeout? Tradeoffs?
+- Roll deprecation of *auto-commit* through code base so leaf functions stop referring to it
- Trace all paths to db-put or db-delete and ensure that there is a check or a
default with-transaction around the primitive components - write a document
clarifying transaction design & assumptions in the backend] Add asserts if
*auto-index* is false and we're not in a transaction to help users avoid lockups
in bdb? Should be able to turn off for performance but it will help catch
missing with-transaction statemetns in user code. (Both)
-~ Automatically run db_deadlock when opening a bdb backend? Requires path to
- functions and ability to launch shell command. Closing the store stops the
- sub-process.
-- Always support locks that timeout? Tradeoffs?
+- Figure out how to compact a specific btree and/or key-range using optimize-storage.
+ Probably need to update keyword part of the API
+
+Indexing efficiency and policies:
+- Add :inverse-reader to slot options to create a named method that indexes into objects
+ based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in?
+- Reclaim table storage on index drop? It's nice to be able to reconnect sometimes!
+ Perhaps an API command that allows explicit dropping of tables for a class and a policy
+ parameter that determines if this is the default?
+- Should we delete slot-values in the db when redefining classes, currently those values
+ stay around - probably indefinitely unless we GC
Performance:
- Metering and understanding locking issues. Large transactions seem
- to use a lot of locks. In general understanding how to use Sleepycat
+ to use a lot of locks. In general understanding how to use Berkeley DB
efficiently seems like a good thing. (From Ben)
- Add dependency information into secondary index callback functions so that
we can more easily compute which indices need to be updated to avoid the
global remove/add in order to maintain consistency (Ian)
- Improve SQL serializer performance (Robert/Ian)
-Indexing features:
-- Add :inverse-reader to slot options to create a named method that indexes into objects
- based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in?
-
-Compliance and Efficiency:
- - Update to support BDB 4.4
- - Add ability from within lisp to reclaim DB space after deleting btree key-value pairs
- - Reclaim table storage on index drop
- - Should we delete slot-values in the db when redefining classes, currently those values
- stay around - probably indefinitely unless we GC
+Test coverage:
+- Test for optimize storage method (just add probe-file methods to get file size)
+- Multi-threading stress tests? Ensure that there are conflicts and lots of serialization
+ happening concurrently to make sure that multi-threading is in good shape
+
+Documentation:
+- Add notes about with-transaction usage (abort & commit behavior on exit)
+- Add notes about optimize-storage
+- Add notes about new BDB 4.4 *auto-commit* behavior. Default for entire store-controller,
+ will auto create a transaction if none is active if open with :auto-commit t or will
+ never auto-commit (regardless of operator flags) if it is not. Make sure open-store
+ defaults to auto-commit and there is a flag to turn it off.
0.6.1 - Features COMPLETED to date
----------------------------------
+x Ensure serialization is multi-threaded and efficient
x Determine how to detect deadlock conditions as an optional run-safe mode?
x BDB overwrite of values makes DB grow
[So far I can only find that it grows on the 2nd write, but not after that...artifact of
page allocation or caching of memory pools?]
x FEATURE: Investigate BDB record size; it's 2x larger than expected?
[Ditto above]
+x Update to support BDB 4.4
+ x Add ability from within lisp to reclaim DB space after deleting btree key-value pairs
-0.6.2 - Query & indexing expansion (Fall '06)
+
+0.6.2 - Advanded work, low-hanging fruit (Fall '06)
--------------------------------------------------
- - Simple object query language (Ian - orthogonal, on main branch)
- - Add needed support (if any) for persistent graph structures & queries (Ian on a branch)
+ - Port elephant to closer-to-MOP to make it easier to support additional lisps and to
+ seriously clean up metaclasses.lisp and classes.lisp protocols
- A wrapper around migration that emulates a stop-and-copy GC
- - Fast serializer port & upgrade strategy
0.6.3 - Documentation & Tools (Winter '06)
--------------------------------------------------
@@ -99,21 +107,19 @@
- A guide to dealing with multiple open stores
- A guide to performance
- An overview of licensing issues...
- - Repository browser (Ian - orthogonal, on main branch)
- (a simple REPL tool to see what classes are in a repository and
- what state they're in...useful for long-lived repositories)
-0.6.4 - Additional datastructures (?)
+0.7.0: Fast In-Memory Database (Not backwards compatible)
--------------------------------------------------
- - Native BDB persistent hashes (easy; can do on SQL backends?)
- - Support for cheap persistent sets (medium? can do on SQL?)
-
-Some placeholders & dreams features below... :)
-
-0.7+: Major features (Winter '07)
---------------------------------------------------
- - A native lisp backend controller (Ian)
- - Integrate prevalence-like in-memory database system (Robert?)
+ - Integrate prevalence-like in-memory database system
+ - Fast serializer port w/ upgrade strategy and prevalence like storage solution
+ - Further improve SQL 64-bit serialization performance (if possible)
+ - (From Ben's e-mail) We are storing persistent objects incorrectly. They should be
+ stored only as OIDs, and we should have a separate OID->class table. This way
+ change-class can be handled correctly. This also non-trivially compresses storage
+ in the database as we only need to store oids rather than serialized class names.
+ [Ian comment: only problem with this is an extra access to oid table each time a
+ class is deserialized and overall storage is constant. Would make it easy to
+ invalidate objects though!]
- Richer controller modes:
- Single-user mode (cache values in instance slots for fast reads, write-through)
- Prevalence mode (read/write to normal slots except on object creation or synch)
@@ -122,6 +128,31 @@
- Concurrent mode (for backends that allow multiple processes to connect, current default)
- Controller 'switches'
- NoSynch - allow transactions to be lost on failure but maintains consistency instead of performance
+ - Usage model examples
+
+0.7.1 - Elephant/BDB/SQL Production Release
+--------------------------------------------------
+ - More work on testing, examples and documentation
+ - Intent is for this to be a major, long-term supported release prior
+ to work on the new backend
+
+0.7.2 - Additional Tools
+--------------------------------------------------
+ - Add needed support (if any) for persistent graph structures & queries (Ian on a branch)
+ - Simple object query language (Ian - orthogonal, on main branch)
+ - Repository browser (Ian - orthogonal, on main branch)
+ (a simple REPL tool to see what classes are in a repository and
+ what state they're in...useful for long-lived repositories)
+
+0.8.0 - Native Backend & Datastructure Library (
+--------------------------------------------------
+ - A native lisp backend controller (Ian)
+ - Native BDB persistent hashes (easy; can do on SQL backends?)
+ - Support for cheap persistent sets (medium? can do on SQL?)
+ - Usage model examples
+
+
+
========================================================
========================================================
