Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv3271

Modified Files: TODO config.sexp Log Message: Large changeset to enable thread safety; more *auto-commit* removal; sql class-root fix; new transaction model; cleaned up defaults for *store-controller*

--- /project/elephant/cvsroot/elephant/TODO 2007/02/01 04:03:26 1.39 +++ /project/elephant/cvsroot/elephant/TODO 2007/02/02 23:51:58 1.40 @@ -6,32 +6,13 @@ 0.6.1 - performance, safety and portability --------------------------------------------

-Lisp support: -- OpenMCL 1.1 on Mac OS X -- Win32 builds -- Lispworks? - Active tasks: -- Full 64-bit support (arrays, native 64-bit fixnums, etc) - - Set parameter at startup based on *features* - - Mark fixnums appropriately: 32-bit lisps can decode 64-bit fixnums as bignums (two 32-bit entities) - - propogate assumptions to bignum byte specs - - are there other fixed assumptions? - - char vs. uint8 in buffer-stream to read-out (See Marco e-mail) - -- Ensure serialization is thread-safe and reasonably efficient +- Support locks in serializer for all systems - Provide support for fast and slow critical sections by lisps: buffer-streams, circularity-arrays/hashes, shared controller side-effects... (see email) - - Resourced-byte-spec should be per-thread (or removed - ok to cons during bignum serialization) -- Think about dynamic vs. object-based store & transaction variables - - Perform error checking when mixed - - Current store specific *current-transaction* stack -- Allow elephant threads to appropriately bind dynamic variables? -- Thread safety for all global vars -- Thread safe API option for user-managed store-controller? -- Thread safe API for transactions -- Throw condition when store spec is invalid, etc -- Test with BDB 4.5? +- Trace all paths to db-put or db-delete and ensure that there is a check or a + default ensure-transaction around the primitive components - write a document + clarifying transaction design & assumptions in the backend]

BDB Features: ? Determine how to detect deadlock conditions as an optional run-safe mode? @@ -39,74 +20,72 @@ functions and ability to launch shell command. Closing the store stops the sub-process. ? Always support locks that timeout? Tradeoffs? -- Roll deprecation of *auto-commit* through code base so leaf functions stop referring to it -- Trace all paths to db-put or db-delete and ensure that there is a check or a - default with-transaction around the primitive components - write a document - clarifying transaction design & assumptions in the backend] Add asserts if - *auto-index* is false and we're not in a transaction to help users avoid lockups - in bdb? Should be able to turn off for performance but it will help catch - missing with-transaction statemetns in user code. (Both) - Figure out how to compact a specific btree and/or key-range using optimize-storage. Probably need to update keyword part of the API

-Indexing efficiency and policies: -- Add :inverse-reader to slot options to create a named method that indexes into objects - based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in? -- Reclaim table storage on index drop? It's nice to be able to reconnect sometimes! - Perhaps an API command that allows explicit dropping of tables for a class and a policy - parameter that determines if this is the default? +ALPHA RELEASE ITEMS

-Performance: -- Implement unicode performance hacks for various lisps; validate UTF8 works everywhere -- Metering and understanding locking issues. Large transactions seem - to use a lot of locks. In general understanding how to use Berkeley DB - efficiently seems like a good thing. (From Ben) -- Add dependency information into secondary index callback functions so that - we can more easily compute which indices need to be updated to avoid the - global remove/add in order to maintain consistency (Ian) +Lisp support: +- 64-bit lisp verification +- Win32 builds + - Windows support for asdf-based library builds? Include 32-bit dll in release? +- OpenMCL 1.1 on Mac OS X +- Lispworks

Stability: -- Delete persistent slot values from the slot store with remove-kv to ensure that - there's no data left lying around if you define then redefine a class and add - back a persistent slot name that you thought was deleted and it gets the old - value by default. -- Cleaner failure modes if operations are performed without repository or without - transaction or auto-commit (auto-commit solved by 4.4?) - Review and address all NOTE comments in the code -- Use SWIG and CFFI to better track changes in defconstant?

-RELEASE ISSUES +Migration: +- Validate migration 0.6.0->0.6.1 +- Validate that migrate can use either O(c) or O(n/c) where c << n memory for large DBs + +BETA RELEASE ITEMS

Test coverage: - Test for optimize storage method (just add probe-file methods to get file size) - Multi-threading stress tests? Ensure that there are conflicts and lots of serialization happening concurrently to make sure that multi-threading is in good shape - -Utilities and Build features: -- Validate migration 0.6.0->0.6.1 - - Validate that migrate can use either O(c) or O(n/c) where c << n memory -- Windows support for asdf-based library builds? Include dll? +- Unicode tests + - Test with UTF-16 and UTF-32 strings (construct with char-code?) + - Ensure that variable length UTF-8 is automatically stored as UTF-16

Documentation: -- Migrate code base to SVN and create tickets in TRAC -- Add notes about with-transaction usage (abort & commit behavior on exit) -- Add notes about fast-symbols +- Migrate code base to Darcs and create tickets in TRAC +- Add notes about with/ensure-transaction usage (abort & commit behavior on exit) - Add notes about optimize-storage -- Add notes about new BDB 4.4 *auto-commit* behavior. Default for entire store-controller, - will auto create a transaction if none is active if open with :auto-commit t or will - never auto-commit (regardless of operator flags) if it is not. Make sure open-store - defaults to auto-commit and there is a flag to turn it off. +- Add notes about deadlock-detect +- Add notes about new BDB 4.4 *auto-commit* behavior. Default for entire + store-controller will auto create a transaction if none is active if open + with :auto-commit t or will never auto-commit (regardless of operator flags) + if it is not. Make sure open-store defaults to auto-commit and there is a + flag to turn it off.

0.6.1 - Features COMPLETED to date ----------------------------------

-January 22, 2006 checkins: +Feburary 2nd, 2007 checkins: +x Check for manual & automatic transactions running concurrently +x Modify *current-transaction* to be null on default, allowing backends to choose the default format (vs. +NULL-VOID+) +x Update BDB backend to properly provide result +x Roll deprecation of *auto-commit* through code base so leaf functions stop referring to it; modify berkeley-db to not refer to auto-commit except where it's appropriate (open commands) +x Ensure serialization is thread-safe and reasonably efficient +x Resourced-byte-spec should be per-thread (or removed - ok to cons during bignum serialization) +x Allow elephant threads to appropriately bind dynamic variables? +x Thread safety for all global vars +x Thread safe API option for user-managed store-controller? +x Thread safe API for transactions +x Ported to and tested with BDB 4.5 +x Full 64-bit support (arrays, native 64-bit fixnums, etc) + x Mark fixnums appropriately: 32-bit lisps can decode 64-bit fixnums as bignums (two 32-bit entities) + x char vs. uint8 in buffer-stream to read-out (See Marco e-mail) + +January 22, 2007 checkins: x Modularize serializers for easy upgrade x MCL 1.1 unicode support; clean up other lisp support for unicode x Simplify user-specific configuration parameters using config.sexp and my-config.sexp x Ensure thread safety in buffer-stream allocation!

-January 2006 checkins; minor fixes +January 2007 checkins; minor fixes x Think through default *store-controller* vs. explicit parameter passing referencing all over the APIs (Enable explicit passing everywhere, maintain *store-controller* defaults. This makes multi-threading support simpler. Users can pass the store controller or rely on a global *store-controller*) @@ -134,17 +113,41 @@

0.6.2 - Advanded work, low-hanging fruit (Summer '07) -------------------------------------------------- - - BDB sorting - - Compare strings of different types in BDB C sorting function - - Or support Lisp sorting callback +Storage and Indexing: +- Add :inverse-reader to slot options to create a named method that indexes into objects + based on slot values. Is this a GF or defun? Do we dispatch on class name or bake it in? +- Reclaim table storage on index drop? It's nice to be able to reconnect sometimes! + Perhaps an API command that allows explicit dropping of tables for a class and a policy + parameter that determines if this is the default? +- Delete persistent slot values from the slot store with remove-kv to ensure that + there's no data left lying around if you define then redefine a class and add + back a persistent slot name that you thought was deleted and it gets the old + value by default. + +Performance: +- Implement unicode performance hacks for various lisps; validate UTF8 works everywhere +- Metering and understanding locking issues. Large transactions seem + to use a lot of locks. In general understanding how to use Berkeley DB + efficiently seems like a good thing. (From Ben) +- Add dependency information into secondary index callback functions so that + we can more easily compute which indices need to be updated to avoid the + global remove/add in order to maintain consistency (Ian) +- Improve SQL serializer performance (Robert/Ian) + +Design: + - Use SWIG and CFFI to better track changes in defconstant? + - Evaluate porting elephant to closer-to-MOP to make it easier to + support additional lisps and to seriously clean up + metaclasses.lisp and classes.lisp protocols + +Features: - Persistent variables (abstraction that allows compound lisp objects at the cost of full serialization after each write that indirects through the API). Can this be done with clean semantics or should we punt it? - Class option MOP add-on to support declared persistent baseclass slots for standard base classes - - Evaluate porting elephant to closer-to-MOP to make it easier to - support additional lisps and to seriously clean up - metaclasses.lisp and classes.lisp protocols - A wrapper around migration that emulates a stop-and-copy GC + +Documentation: - Tutorial example rethink: update the blog tutorial using indexed objects to create different views as well as integrating something like logging for admin or version control purposes. @@ -153,7 +156,6 @@ - A guide to dealing with multiple open stores - A guide to performance - An overview of licensing issues... - - Improve SQL serializer performance (Robert/Ian)

0.7.0: Fast In-Memory Database (Not backwards compatible) -------------------------------------------------- --- /project/elephant/cvsroot/elephant/config.sexp 2007/01/25 18:17:59 1.4 +++ /project/elephant/cvsroot/elephant/config.sexp 2007/02/02 23:51:58 1.5 @@ -1,10 +1,15 @@ -((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.4/") - (:berkeley-db-lib-dir . "/opt/local/lib/db44/") - (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.4/lib/libDB-4.4.dylib") +((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/") + (:berkeley-db-lib-dir . "/opt/local/lib/db45/") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libDB-4.5.dylib") (:pthread-lib . nil) (:clsql-lib . nil))

+;; Berkeley 4.5 is required, each system will have different settings for +;; these directories, use this as an indication of what each key means +;; ;; Typical pthread settings are: /lib/tls/libpthread.so.0 +;; ;; nil means that the library in question is not loaded -;; NOTE: The latest SBCL on linux no longer needs the pthread library, +;; +;; NOTE: The latest SBCL (0.9.17+) on linux no longer needs the pthread library, ;; it is statically linked against it now with the new thread support \ No newline at end of file