Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv13701

Modified Files: TODO elephant.asd Log Message: Export btree utilities; implement efficient map operators, reimplement get-instance methods; add test of map-index; better declarations

--- /project/elephant/cvsroot/elephant/TODO 2007/02/18 10:58:58 1.54 +++ /project/elephant/cvsroot/elephant/TODO 2007/02/20 19:12:57 1.55 @@ -6,15 +6,13 @@ 0.6.1 - performance, safety and portability --------------------------------------------

-TASKS TO GET TO ALPHA: +TASKS TO GET TO BETA:

Migration: - Validate SQL migration 0.6.0->0.6.1 (Robert)

-TASKS TO GET TO BETA: - -BDB Features/Cleanup: -- Derived indices used to fail on re-connect, verify that this is fixed +Ian todo: +- Apply Robert's package translation changes

Lisp Support: - Win32 builds @@ -35,12 +33,15 @@ - Migration: Improve printing and informative messages

Test coverage: +- Make tests idempotent, clean up interface to tests - Test for optimize storage method (just add probe-file methods to get file size?) - Multi-threading stress tests? Ensure that there are conflicts and lots of serialization happening concurrently to make sure that multi-threading is in good shape (Henrik's code) - Unicode tests - Test with UTF-16 and UTF-32 strings (construct with char-code?) - Ensure that variable length UTF-8 is automatically stored as UTF-16 +- Map tests +- Class index sychronization tests

TASKS TO GET TO FINAL RELEASE:

@@ -58,7 +59,20 @@ 0.6.1 - Features COMPLETED to date ----------------------------------

-February +POST ALPHA CHECKINS: + +Major Bugs: +x Derived indices fail to re-connect after reopening a database under :class synchronization policy (Ian) + +Minor Bugs: +x Enable with-transactions to properly process forms returning multiple values (Ian) +x Fixed typos in SQL backend (Ian/Robert/Henrik) +x Fixed build bug for linux (Henrik) +x Fixed error condition while opening SQLITE3 in SQL backend (Robert) +x Fixed idempotence problem in PREPARE-BDB test + +Feature tweaking: +x Orthogonal feature addition: map-index, map-class and map-instances to avoid consing (Ian)

February 13-17th, 2007: x Allow checkpoint of BDB via db-bdb::checkpoint @@ -152,6 +166,8 @@ there's no data left lying around if you define then redefine a class and add back a persistent slot name that you thought was deleted and it gets the old value by default. +- Can we do automatic join cursors? +- Add lazy deserialize to map functions?

Performance: - Implement unicode performance hacks for various lisps; validate UTF8 works everywhere @@ -167,15 +183,12 @@ - Evaluate porting elephant to closer-to-MOP to make it easier to support additional lisps and to seriously clean up metaclasses.lisp and classes.lisp protocols (no love on first attempt) - (log these in Track) + (log these in Track; not part of 0.6.2)

Features: - Backup function: allow users to specify a backup function to archive the database contents and checkpoint any active functions (how to lock out other threads?) In BDB this means running checkpoint and copying the DB files and any active log files. - - Persistent variables (abstraction that allows compound lisp objects at the cost of - full serialization after each write that indirects through the API). Can this be done - with clean semantics or should we punt it? - Class option MOP add-on to support declared persistent baseclass slots for standard base classes - A wrapper around migration that emulates a stop-and-copy GC

@@ -191,32 +204,53 @@

0.7.0: Native Lisp Backend (beta), Fast In-Memory Operations ------------------------------------------------------------ + +Major features: + - Native Lisp Backend + - All in common-lisp + - Page-based architecture + - Cheap copy-on-write transaction policy w/ concurrent transaction commits + in non-conflicting transactions + - Fast in-memory operations / prevalence like features + - Make storage policy decisions on per-class (or per-instance) basis + - Concurrent mode + - Current default + - For backends that allow multiple processes to connect + - Full ACID functionality + - Single-threaded objects + - Cache values in instance slots for fast reads + - Writes update slots and write to disk as normal + - Violates consistency and isolation; users must enforce single operator + - Prevalence mode + - Standard object model (user enforced ACID properties) + - Read/write to normal slots + - Backup slot values on object creation and explicit synch calls on class or instances + - In-memory slot indexing, write-through disk class indexing + - NoSynch controller switch + - Violates durability + - Offline garbage collection (via migration) + - Class schemas + - Improve synchronization, support class and instance versioning + - POBJs encoded by OID/CID and CID's are cached in working memory + This way we can issue conditions if an object is out of date so the + user can determine how/if to upgrade the reference to the current schema + - From Ben's e-mail: + We are storing persistent objects incorrectly. They should be stored only as OIDs, + and we should have a separate OID->class table. This way change-class can be + handled correctly. This also non-trivially compresses storage in the + database as we only need to store oids rather than serialized class names. + [Ian comment: only problem with this is an extra access to oid table each time a + class is deserialized and overall storage is constant. Would make it easy to + invalidate objects though!] + - Persistent variables (abstraction that allows compound lisp objects at the cost of + full serialization after each write that indirects through the API). Can this be done + with clean semantics or should we punt it? + - Support a simple object query language + +Details: - Revisit duplicate sorting on primary key (artifact of btree index storage order) - - Full support for DCM or integration of DCM functionality - - Integrate prevalence-like in-memory database system for single image, - multiple-thread operation - - Richer set of policy decisions on per-class basis - - Concurrent mode (for backends that allow multiple processes to connect, current default) - - Single-user mode (cache values in instance slots for fast reads, write-through) - - Backing store mode (read/write to normal slots except on object creation or synch) - (in-memory slot indexing, on disk class) - (works for any backend) - - Backing-store mode - - Controller 'switches' - - NoSynch - allow transactions to be lost on failure but maintains consistency - instead of performance - - Upgrade overall functionality - - Solid garbage collection strategy - - 64-bit oids / 64-bit file sizes? - - class templates stored and cached - - (From Ben's e-mail) We are storing persistent objects incorrectly. They should be - stored only as OIDs, and we should have a separate OID->class table. This way - change-class can be handled correctly. This also non-trivially compresses storage - in the database as we only need to store oids rather than serialized class names. - [Ian comment: only problem with this is an extra access to oid table each time a - class is deserialized and overall storage is constant. Would make it easy to - invalidate objects though!] - - Usage model examples + - Usage model examples for new features + - 64-bit oids / 64-bit file sizes

0.7.1 - Elephant BDB/SQL/Lisp Production Release -------------------------------------------------- @@ -224,20 +258,19 @@ - Intent is for this to be a major, long-term supported release prior to work on the new backend (i.e. patches against this release for bugs rather than only available in latest development tree) + - Online GC for lisp backend?

0.8.0 - Supporting Tools Release -------------------------------------------------- - - Add special support (if any) for persistent graph structures & queries - (ala AllegroCache) - - Support for cheap persistent sets (medium? can do on SQL?) - - Native persistent hashes (easy for BDB; can do on SQL backends?) - - Persistent aggregates for better conceptual integration with lisp? - - pcons, parray, pstruct, etc - - Support a simple object query language over the database + - Richer query language and compiler - Repository browser - a simple REPL tool like the Slime inspector to see what classes are in a repository and what state they're in...useful for long-lived repositories or if you've forgotten a variable name + - Add special support (if any) for persistent graph structures & queries (ala AllegroCache) + - Support for cheap persistent sets (medium? can do on SQL?) + - Persistent aggregates for better conceptual integration with lisp? + - pcons, parray, pstruct, etc

1.0 - Final Production release (1st long-term version since 0.7.1) ------------------------------------------------------------------------ --- /project/elephant/cvsroot/elephant/elephant.asd 2007/02/18 22:45:39 1.31 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/02/20 19:12:57 1.32 @@ -118,6 +118,31 @@ input-file "-lm"))

+;;(defmethod compiler-options ((compiler (eql :gcc-cygwin)) (c elephant-c-source) &key input-file output-file) +;; (unless (and input-file output-file) +;; (error "Must specify both input and output files")) +;; (list +;; "-shared" +;; "-mno-cygwin" +;; "-mwindows" +;; "-std=c99" +;; input-file +;; "-o" output-file +;; "--export-symbols" +;; (namestring (make-pathname :defaults output-file :type "def")))) + +;;gcc -mno-cygwin -mwindows -std=c99 -c libmemutil.c +;;dlltool -z libmeutil.def --export-all-symbols -e exports.o -l libmemutil.lib libmemutil.o +;;gcc -shared -mno-cygwin -mwindows libmemutil.o exports.o -o libmemutil.dll + +;;gcc -shared -mno-cygwin -mwindows libmemutil.o exports.o -o libmemutil.dll + +;;And this is the script for libsleepycat.dll: +;; +;;gcc -mno-cygwin -mwindows -c -Wall -std=c99 -L/c/DB/Berkeley\ DB\ 4.4.20/lib/ -I/c/DB/Berkeley\ DB\ 4.4.20/include/ libsleepycat.c +;;dlltool -z libsleepycat.def --export-all-symbols -e exports.o -l libsleepycat.lib libsleepycat.o +;;gcc -shared -mno-cygwin -mwindows -L/c/DB/Berkeley\ DB\ 4.4.20/bin/ -llibdb44 libsleepycat.o exports.o -o libsleepycat.dll + (defmethod compiler-options ((compiler (eql :msvc)) (c elephant-c-source) &key input-file output-file) (error "MSVC compiler option not supported yet"))