Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv20086

Modified Files: TODO Log Message: Fix SBCL struct serialization; cleanup TODO after Trac conversion; remove persistant aggregate stubs

--- /project/elephant/cvsroot/elephant/TODO 2007/02/25 03:40:18 1.60 +++ /project/elephant/cvsroot/elephant/TODO 2007/02/25 09:12:47 1.61 @@ -1,7 +1,10 @@

-Last updated: February 4, 2007 +Last updated: February 25th, 2007

-Ongoing release plan notes: +Ongoing release plan notes. + +(Note: use of this file is deprecated after 0.6.1, see Trac site at + http://trac.common-lisp.net/elephant)

0.6.1 - performance, safety and portability -------------------------------------------- @@ -75,7 +78,8 @@

Feature tweaking: x Orthogonal feature addition: map-index, map-class and map-instances to avoid consing (Ian) -x Tests to validate new map interfaces on top of existing tests +x Tests to validate new map interfaces on top of existing tests (Ian) +x Added support and tests for serializing structure objects on all supported platforms (Ian)

DEVELOPMENT CHECKINS:

@@ -156,151 +160,6 @@ x Remove sleepycat name. Change sleepycat to db-bdb to reflect oracle ownership and avoid confusion for new users

-0.7.0 - Advanded work, low-hanging fruit (Summer '07) ------------------------------------------------------ - -Migrate code base to Darcs and create feature/bug tickets in TRAC - -Storage and Indexing: -- Add :inverse-reader to slot options to create a named method that indexes into objects - based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in? -- If a class inherits an indexed slot, is it also indexed for that class? This means a - proliferation of indexes, or requires user to explicitly add an index as a derived slot. - (Proposal: any subclass must have an :index specifier that is the same as the base class) - (Imp. option: one index for all instances of primary and subclasses; or index per set?) - (Perhaps slot indices should not be secondary so they can point to different class types?) -- What if we want an index to index into a range of different subclasses or objects sharing - a generic function? (roll your own?) -- Reclaim table storage on index drop? It's nice to be able to reconnect sometimes! - Perhaps an API command that allows explicit dropping of tables for a class and a policy - parameter that determines if this is the default? -- Delete persistent slot values from the slot store with remove-kv to ensure that - there's no data left lying around if you define then redefine a class and add - back a persistent slot name that you thought was deleted and it gets the old - value by default? -- Can we do automatic join cursors? -- Add lazy deserialize to map functions? -- First-cut query interface - -Performance: -- Improve SQL base-64 serializer performance? -- Implement unicode performance hacks for various lisps; validate UTF8 works everywhere -- Metering and understanding locking issues. Large transactions seem - to use a lot of locks. In general understanding how to use Berkeley DB - efficiently seems like a good thing. (From Ben) -- Add dependency information into secondary index callback functions so that - we can more easily compute which indices need to be updated to avoid the - global remove/add in order to maintain consistency (Ian) -- Track derived indices across classes - -Design: - - Move secondary index maintenance to backend; decison on how to call update fn's - Will make lisp backend cheaper due to ability to life tree manipulation ops - - Use SWIG and CFFI to better track changes in defconstant? (too expensive to be useful) - - Evaluate porting elephant to closer-to-MOP to make it easier to - support additional lisps and to seriously clean up - metaclasses.lisp and classes.lisp protocols (no love on first attempt) - - Work to integrate a proper condition system and potential restarts - for various errors (especially recoverable ones) while accessing db data - - deserialize fubar / diagnose & return value - - missing package / add package/symbol translation - - others? - (log these in Track; not part of 0.6.2?) - -Features: - - Backup function: allow users to specify a backup function to archive the database contents - and checkpoint any active functions (how to lock out other threads?) In BDB this means - running checkpoint and copying the DB files and any active log files. - - Class option MOP add-on to support declared persistent baseclass slots for standard base classes - - A wrapper around migration that emulates a stop-and-copy GC - -Documentation: - - Tutorial example rethink: update the blog tutorial using indexed - objects to create different views as well as integrating something - like logging for admin or version control purposes. - - Finish serious update and review of users manual (building on 0.6.0 update) - - A guide to dealing with transactions - - A guide to dealing with multiple open stores - - A guide to performance - - An overview of licensing issues... - -0.8.0: Native Lisp Backend (beta), Fast In-Memory Operations ------------------------------------------------------------- - -Major features: - - Native Lisp Backend - - All in common-lisp - - Page-based architecture - - Cheap copy-on-write transaction policy w/ concurrent transaction commits - in non-conflicting transactions - - Fast in-memory operations / prevalence like features - - Make storage policy decisions on per-class (or per-instance) basis - - Concurrent mode - - Current default - - For backends that allow multiple processes to connect - - Full ACID functionality - - Single-threaded objects - - Cache values in instance slots for fast reads - - Writes update slots and write to disk as normal - - Violates consistency and isolation; users must enforce single operator - - Prevalence mode - - Standard object model (user enforced ACID properties) - - Read/write to normal slots - - Backup slot values on object creation and explicit synch calls on class or instances - - In-memory slot indexing, write-through disk class indexing - - NoSynch controller switch - - Violates durability - - Offline garbage collection (via migration) - - Class schemas - - Improve synchronization, support class and instance versioning - - POBJs encoded by OID/CID and CID's are cached in working memory - This way we can issue conditions if an object is out of date so the - user can determine how/if to upgrade the reference to the current schema - - From Ben's e-mail: - We are storing persistent objects incorrectly. They should be stored only as OIDs, - and we should have a separate OID->class table. This way change-class can be - handled correctly. This also non-trivially compresses storage in the - database as we only need to store oids rather than serialized class names. - [Ian comment: only problem with this is an extra access to oid table each time a - class is deserialized and overall storage is constant. Would make it easy to - invalidate objects though!] - - Persistent variables (abstraction that allows compound lisp objects at the cost of - full serialization after each write that indirects through the API). Can this be done - with clean semantics or should we punt it? - - Support a simple object query language - -Details: - - Revisit duplicate sorting on primary key (artifact of btree index storage order) - - Usage model examples for new features - - 64-bit oids / 64-bit file sizes - -0.8.1 - Elephant BDB/SQL/Lisp Pre-production Release --------------------------------------------------- - - More work on testing, examples and documentation - - Intent is for this to be a major, long-term supported release prior - to work on the new backend (i.e. patches against this release for - bugs rather than only available in latest development tree) - - Online GC for lisp backend? - - -0.9.0 - Supporting Tools Release --------------------------------------------------- - - Richer query language and query compiler - - Repository browser - a simple REPL tool like the Slime inspector - to see what classes are in a repository and what state they're in...useful - for long-lived repositories or if you've forgotten a variable name - - Add special support (if any) for persistent graph structures & queries (ala AllegroCache) - - Support for cheap persistent sets (medium? can do on SQL?) - - Persistent aggregates for better conceptual integration with lisp? - - pcons, parray, pstruct, etc - - push/pop elements from a list defined on a persistent slot? - -1.0 - Final Production release (1st long-term version since 0.7.1) ------------------------------------------------------------------------- - - Significant work on test cases & testing framework - - Final pass of performance enhancements and review - - Invite community review and testing - ======================================================== ========================================================