Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv32730

Modified Files: TODO config.sexp elephant.asd Log Message: Documentation, optimizations, deadlock process, etc

--- /project/elephant/cvsroot/elephant/TODO 2007/02/12 20:36:43 1.47 +++ /project/elephant/cvsroot/elephant/TODO 2007/02/14 04:36:08 1.48 @@ -6,65 +6,72 @@ 0.6.1 - performance, safety and portability --------------------------------------------

-ALPHA RELEASE TASKS +TASKS TO GET TO ALPHA: + +Migration: +- Validate BDB migration 0.6.0->0.6.1 (Ian) +- Validate SQL migration 0.6.0->0.6.1 (Robert) +- Legacy conversions issue for SQL (and BDB?) due to package rename (both)

-Bug and feature fixes: -~ Resolve duplicate sorting guarantee in btree interface; currently supported - by BDB but not SQL and it is not tested in the regression suite (Robert to comment) + +TASKS TO GET TO BETA:

BDB Features/Cleanup: -- Trace all paths to db-put or db-delete and ensure that there is a check or a - default ensure-transaction around the primitive components - write a document - clarifying transaction design & assumptions in the backend] -- Determine how to detect deadlock conditions as an optional run-safe mode? -- Automatically run db_deadlock when opening a bdb backend? Requires path to - functions and ability to launch shell command. Closing the store stops the - sub-process. - Always support locks that timeout? Tradeoffs? - Figure out how to compact a specific btree and/or key-range using optimize-storage. Probably need to update keyword part of the API +- Perform checkpoints (prep for DCM functionality) +- Verify db_deadlock for other lisps (launch and kill background program I/F) +- Derived indices fail to re-connect, verify this

-BETA RELEASE TASKS - -Lisp support: +Lisp Support: +- 64-bit lisp verification - Win32 builds - Windows support for asdf-based library builds? Include 32-bit dll in release? - Validate OpenMCL 1.1 on Mac OS X - Validate Lispworks -- 64-bit lisp verification

Stability and Performance: +- Validate that migrate can use either O(c) or O(n/c) where c << n memory for large DBs - Review and address all NOTE comments in the code - Review SBCL string serialization performance - Improve SQL base-64 serializer performance?

-Migration: -- Validate migration 0.6.0->0.6.1 -- Validate that migrate can use either O(c) or O(n/c) where c << n memory for large DBs - -FINAL RELEASE TASKS +TASKS TO GET TO FINAL RELEASE:

Test coverage: -- Test for optimize storage method (just add probe-file methods to get file size) +- Test for optimize storage method (just add probe-file methods to get file size?) - Multi-threading stress tests? Ensure that there are conflicts and lots of serialization - happening concurrently to make sure that multi-threading is in good shape + happening concurrently to make sure that multi-threading is in good shape (Henrik's code) - Unicode tests - Test with UTF-16 and UTF-32 strings (construct with char-code?) - Ensure that variable length UTF-8 is automatically stored as UTF-16

Documentation: +- Add document section about backend interface: - Add notes about with/ensure-transaction usage (abort & commit behavior on exit) - Add notes about optimize-storage - Add notes about deadlock-detect +- Add notes about checkpoint (null in SQL?) - Add notes about new BDB 4.5 *auto-commit* behavior. Default for entire store-controller will auto create a transaction if none is active if open with :auto-commit t or will never auto-commit (regardless of operator flags) if it is not. Make sure open-store defaults to auto-commit and there is a flag to turn it off. +- More notes about transaction performance

0.6.1 - Features COMPLETED to date ----------------------------------

+February 13th, 2007: +x User choice to run db_deadlock when opening a bdb backend? Requires path to + functions and ability to launch shell command. Closing the store stops the sub-process. +x Resolve duplicate ordering issues (punting to future release; documentation fix) +x Trace all paths to db-put or db-delete and ensure that there is a check or a + default ensure-transaction around the primitive components - write a document + clarifying transaction design & assumptions in the backend + + February 9th, 2007 x Improve SQL secondary cursor performance (Robert) x Fix *dbconnection-spec* to support multiple controllers for multiple threads @@ -77,6 +84,7 @@ subtle issues in database metadata and the user of the serializer

Feburary 2nd, 2007 checkins: +x Punted duplicate sorting to 0.7.0 x Support locks in serializer for all systems x Provide support for fast and slow critical sections by lisps: buffer-streams, circularity-arrays/hashes, shared controller side-effects... (see email) @@ -178,6 +186,7 @@

0.7.0: Native Lisp Backend (beta), Fast In-Memory Operations ------------------------------------------------------------ + - Revisit duplicate sorting on primary key (artifact of btree index storage order) - Full support for DCM or integration of DCM functionality - Integrate prevalence-like in-memory database system for single image, multiple-thread operation @@ -193,7 +202,7 @@ instead of performance - Upgrade overall functionality - Solid garbage collection strategy - - 64-bit oids / 64-bit file sizes + - 64-bit oids / 64-bit file sizes? - class templates stored and cached - (From Ben's e-mail) We are storing persistent objects incorrectly. They should be stored only as OIDs, and we should have a separate OID->class table. This way --- /project/elephant/cvsroot/elephant/config.sexp 2007/02/02 23:51:58 1.5 +++ /project/elephant/cvsroot/elephant/config.sexp 2007/02/14 04:36:08 1.6 @@ -1,6 +1,7 @@ -((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/") +((:berkeley-db-include-dir . "/opt/local/BerkeleyDB.4.5/") (:berkeley-db-lib-dir . "/opt/local/lib/db45/") - (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libDB-4.5.dylib") + (:berkeley-db-lib . "/opt/local/BerkeleyDB.4.5/lib/libDB-4.5.dylib") + (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") (:pthread-lib . nil) (:clsql-lib . nil))

--- /project/elephant/cvsroot/elephant/elephant.asd 2007/02/12 20:36:44 1.28 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/02/14 04:36:08 1.29 @@ -159,7 +159,9 @@ ((:module utils :components ((:file "package") - (:file "locks"))) + (:file "convenience") + (:file "locks") + (:file "os"))) (:module memutil :components ((:elephant-c-source "libmemutil")