Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv4494

Modified Files: NOTES TODO config.sexp ele-bdb.asd elephant.asd Log Message: Checkpoint for 0.6.1 feature set - BROKEN

--- /project/elephant/cvsroot/elephant/NOTES 2006/04/26 17:53:43 1.7 +++ /project/elephant/cvsroot/elephant/NOTES 2006/12/16 19:35:09 1.8 @@ -28,7 +28,6 @@ database / serializer, specials are needed. Also specials will probably play nice with threaded lisps.

- ----------------------- CLASSES AND METACLASSES ----------------------- @@ -182,17 +181,22 @@ SERIALIZER: GENERAL -------------------

-Currently assumes a 32-bit architecture, e.g. fixnums fit in -(signed-byte 32), that there are single- and double-floats -(IEEE). Shouldn't be hard to port. +** Ian: update this + +The serializer should be lisp independant but is machine architecture dependant. +Serialization depends on endianness and the native size of fixnums (31 bit or +63 bit) so that a fixnum written on a 64-bit machine would fail on a 32-bit machine +and vice versa. These restrictions are made for the sake of performance. To move +machine architectures (i.e. x86-32 to x86-64, or PPC to x86) you'll need to dump +the DB to some format. (Migration will not work in these instances although someone +is welcome to write a serialization tool that will read foreign formats. I don't think +the time is worth it compared to other features)

No optimization for specialized arrays at the moment, other than strings (which should be wickedly fast.)

the serializer and deserializer are recursive etypecase and -conds, respectively. in the case of the serializer on -CMUCL this appears to be better than generic functions, -though i don't know why. +conds, respectively.

--------------------------- SERIALIZER: PRIMITIVE TYPES @@ -280,6 +284,23 @@

support callables, closures, structures et al.

+----------------- +Backend Protocol +----------------- + +In generalizing the elephant metaclass and serializer so it can +work with multiple backend we formalized the interface between the +lisp common functionality and the SQL/BDB specific logic. There +are five protocols backends need to support: + +- Controller setup/teardown +- Persistent slot API +- Collection API +- Transaction API +- Symbol ID serialization protocol + +** Ian TODO + --------- SLEEPYCAT --------- @@ -355,6 +376,14 @@ don't in many cases) should see better non-consing behavior hopefully.

-Waiting for Berkeley DB 4.3 to get counters (sequences.) -ETA October 2004. +There are several BDB specific functions available via the +BDB store-controller.

+1) Database compaction: when deleting large swaths of the database + it helps to compact the disk storage so we free up disk space. +2) Deadlock detection; when running multi-threaded, one lisp + thread can block another depending on how they're interleaved. + Also if we have multiple OS processes or machines talking + to the same DB we can end up with a deadlock situation. + The typical solution is to run deadlock detection in a separate + thread or launch a process to do so... \ No newline at end of file --- /project/elephant/cvsroot/elephant/TODO 2006/11/11 18:41:10 1.30 +++ /project/elephant/cvsroot/elephant/TODO 2006/12/16 19:35:09 1.31 @@ -1,5 +1,5 @@

-Last updated: November 11, 2006 +Last updated: November 21, 2006

Ongoing release plan notes:

@@ -7,19 +7,23 @@ --------------------------------------------

Bugs or Observations: -x 64-bit support (from Marco) -x Windows support for asdf-based library builds? -x MCL 1.1 unicode support; rationalize other lisp support for unicode +- Windows support for asdf-based library builds? Include dll? +- Validate migration 0.6.0->0.6.1 +- Full 64-bit support (arrays, native 64-bit fixnums, etc) + - char vs. uint8 in buffer-stream + - flexible handling of 64-bit fixnums

Stability: -- Remove build gensym warnings in sleepycat +- Remove build gensym warnings in sleepycat.lisp - Delete persistent slot values from the slot store with remove-kv to ensure that there's no data left lying around if you define then redefine a class and add back a persistent slot name that you thought was deleted and it gets the old value by default. - Cleaner failure modes if operations are performed without repository or without - transaction or auto-commit (Both) -- Review all the NOTE comments in the code + transaction or auto-commit (auto-commit solved by 4.4?) +- Review all NOTE comments in the code +- Validate that migrate can use either O(c) or O(n/c) where c << n memory +- Migrate code base to SVN and create tickets in TRAC

Store variables: - Think through default *store-controller* vs. explicit parameter passing @@ -30,15 +34,15 @@ - Throw condition when store spec is invalid, etc

Multi-threading operation: -- Make elephant threads appropriately bind dynamic variables -- Verify that operations such as indexing are thread safe -- Verify that serialization is thread safe +- Make elephant threads appropriately bind dynamic variables? +x Verify that operations such as indexing are thread safe

BDB Features: -~ Automatically run db_deadlock when opening a bdb backend? Requires path to +? Determine how to detect deadlock conditions as an optional run-safe mode? +? Automatically run db_deadlock when opening a bdb backend? Requires path to functions and ability to launch shell command. Closing the store stops the sub-process. -- Always support locks that timeout? Tradeoffs? +? Always support locks that timeout? Tradeoffs? - Roll deprecation of *auto-commit* through code base so leaf functions stop referring to it - Trace all paths to db-put or db-delete and ensure that there is a check or a default with-transaction around the primitive components - write a document @@ -73,6 +77,7 @@ Documentation: - Add notes about with-transaction usage (abort & commit behavior on exit) - Add notes about optimize-storage +- Add notes about fast-symbols - Add notes about new BDB 4.4 *auto-commit* behavior. Default for entire store-controller, will auto create a transaction if none is active if open with :auto-commit t or will never auto-commit (regardless of operator flags) if it is not. Make sure open-store @@ -80,15 +85,22 @@

0.6.1 - Features COMPLETED to date ---------------------------------- +x Improved optimization options to be more user controlled (Pierre Thierry) +x Implement backend support for symbol-table protocol +x Speed up symbol storage and reference using symbol id's +x Ensure serialization is thread-safe and reasonably efficient +x MCL 1.1 unicode support; rationalize other lisp support for unicode +x Modularize serializers for easy upgrade x New build interface; all-lisp compilation (sans win32) -x Ensure serialization is multi-threaded and efficient -x Determine how to detect deadlock conditions as an optional run-safe mode? +x Simplify user-specific configuration parameters using config.sexp and my-config.sexp +x Make sure to ensure thread safety in buffer-stream allocation! + x BDB overwrite of values makes DB grow [So far I can only find that it grows on the 2nd write, but not after that...artifact of page allocation or caching of memory pools?] x FEATURE: Investigate BDB record size; it's 2x larger than expected? [Ditto above] -x Update to support BDB 4.4 +x Update to support BDB 4.4/4.5 x Add ability from within lisp to reclaim DB space after deleting btree key-value pairs x Should we delete slot-values in the db when redefining classes, currently those values stay around - probably indefinitely unless we GC (no, we'll resolve this with a @@ -100,8 +112,9 @@ 0.6.2 - Advanded work, low-hanging fruit (Fall '06) -------------------------------------------------- - Class option MOP add-on to support declared persistent baseclass slots for standard base classes - - Port elephant to closer-to-MOP to make it easier to support additional lisps and to - seriously clean up metaclasses.lisp and classes.lisp protocols + - Evaluate porting elephant to closer-to-MOP to make it easier to + support additional lisps and to seriously clean up + metaclasses.lisp and classes.lisp protocols - A wrapper around migration that emulates a stop-and-copy GC

0.6.3 - Documentation & Tools (Winter '06) @@ -117,7 +130,7 @@

0.7.0: Fast In-Memory Database (Not backwards compatible) -------------------------------------------------- - - Integrate prevalence-like in-memory database system + - Integrate prevalence-like in-memory database system for single image, multiple-thread operation - Fast serializer port w/ upgrade strategy and prevalence like storage solution - Further improve SQL 64-bit serialization performance (if possible) - (From Ben's e-mail) We are storing persistent objects incorrectly. They should be @@ -127,12 +140,13 @@ [Ian comment: only problem with this is an extra access to oid table each time a class is deserialized and overall storage is constant. Would make it easy to invalidate objects though!] - - Richer controller modes: - - Single-user mode (cache values in instance slots for fast reads, write-through) - - Prevalence mode (read/write to normal slots except on object creation or synch) - (in-memory slot indexing, on disk class) - (works for any backend) + - Richer set of policy decisions on per-class basis - Concurrent mode (for backends that allow multiple processes to connect, current default) + - Single-user mode (cache values in instance slots for fast reads, write-through) + - Backing store mode (read/write to normal slots except on object creation or synch) + (in-memory slot indexing, on disk class) + (works for any backend) + - Backing-store mode - Controller 'switches' - NoSynch - allow transactions to be lost on failure but maintains consistency instead of performance - Usage model examples --- /project/elephant/cvsroot/elephant/config.sexp 2006/11/11 18:45:04 1.1 +++ /project/elephant/cvsroot/elephant/config.sexp 2006/12/16 19:35:09 1.2 @@ -1,7 +1,8 @@ ((:berkeley-db-root . "/usr/local/BerkeleyDB.4.4/") (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.4/lib/libDB-4.4.dylib") (:pthread-lib . nil) - (:clsql-lib . nil)) + (:clsql-lib . nil) + (:fast-symbols . t))

;; Typical pthread settings are: /lib/tls/libpthread.so.0 ;; nil means that the library in question is not loaded --- /project/elephant/cvsroot/elephant/ele-bdb.asd 2006/11/11 18:41:10 1.12 +++ /project/elephant/cvsroot/elephant/ele-bdb.asd 2006/12/16 19:35:09 1.13 @@ -45,8 +45,8 @@ (defclass bdb-c-source (elephant-c-source) ())

(defmethod compiler-options ((compiler (eql :gcc)) (c bdb-c-source) &key &allow-other-keys) - (let* ((include (merge-pathnames (get-config-option :berkeley-db-root c) "include")) - (lib (merge-pathnames (get-config-option :berkeley-db-root c) "lib"))) + (let* ((include (make-pathname :directory (get-config-option :berkeley-db-include-dir c))) + (lib (make-pathname :directory (get-config-option :berkeley-db-lib-dir c)))) (append (list (format nil "-L~A" lib) (format nil "-I~A" include)) (call-next-method) (list "-ldb")))) @@ -78,8 +78,10 @@ (:bdb-c-source "libberkeley-db") (:file "berkeley-db") (:file "bdb-controller") - (:file "bdb-transactions") - (:file "bdb-collections")) + (:file "bdb-symbol-tables") + (:file "bdb-slots") + (:file "bdb-collections") + (:file "bdb-transactions")) :serial t)))) :depends-on (:uffi :elephant))

--- /project/elephant/cvsroot/elephant/elephant.asd 2006/11/11 06:27:37 1.20 +++ /project/elephant/cvsroot/elephant/elephant.asd 2006/12/16 19:35:09 1.21 @@ -146,12 +146,16 @@ (:module elephant :components ((:file "package") - (:file "variables") + (:file "cross-platform") #+cmu (:file "cmu-mop-patches") #+openmcl (:file "openmcl-mop-patches") + (:file "variables") (:file "transactions") (:file "metaclasses") (:file "classes") + (:file "serializer1") ;; 0.6.0 db's + (:file "serializer2") ;; 0.6.1 db's + (:file "unicode2") (:file "serializer") (:file "cache") (:file "controller") @@ -162,5 +166,5 @@ (:file "backend")) :serial t :depends-on (memutil))))) - :depends-on (:uffi)) + :depends-on (:uffi :cl-base64))