elephant-cvs

elephant-cvs@common-lisp.net

858 discussions

[elephant-cvs] CVS update: elephant/src/serializer.lisp
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant/src In directory common-lisp.net:/tmp/cvs-serv7895/src Modified Files: serializer.lisp Log Message: new license, bignum bugfix, don't save class slots Date: Sun Aug 29 22:40:49 2004 Author: blee Index: elephant/src/serializer.lisp diff -u elephant/src/serializer.lisp:1.5 elephant/src/serializer.lisp:1.6 --- elephant/src/serializer.lisp:1.5 Sun Aug 29 09:54:46 2004 +++ elephant/src/serializer.lisp Sun Aug 29 22:40:49 2004 @@ -12,6 +12,10 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; <ablumberg(a)common-lisp.net> <blee(a)common-lisp.net> ;;; +;;; This program is released under the following license +;;; ("GPL"). For differenct licensing terms, contact the +;;; copyright holders. +;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software @@ -121,10 +125,9 @@ (buffer-write-byte +negative-bignum+ bs) (buffer-write-byte +positive-bignum+ bs)) (buffer-write-int needed bs) - (loop for i fixnum from 0 below word-size - ;; shouldn't this be "below"? + (loop for i fixnum from 0 to word-size for byte-spec = (int-byte-spec i) - ;; this ldb is consing! + ;; this ldb is consing on CMUCL! ;; there is an OpenMCL function which should work ;; and non-cons for the-uint of-type (unsigned-byte 32) = (ldb byte-spec num) @@ -204,7 +207,9 @@ for slot-name = (slot-definition-name sd) with ret = () do - (when (slot-boundp o slot-name) + (when (and (slot-boundp o slot-name) + (eq :instance + (slot-definition-allocation sd))) (push (slot-value o slot-name) ret) (push slot-name ret)) finally (return ret)))

1 0

[elephant-cvs] CVS update: elephant/src/metaclasses.lisp
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant/src In directory common-lisp.net:/tmp/cvs-serv7868/src Modified Files: metaclasses.lisp Log Message: new license, declare optimize Date: Sun Aug 29 22:40:06 2004 Author: blee Index: elephant/src/metaclasses.lisp diff -u elephant/src/metaclasses.lisp:1.1 elephant/src/metaclasses.lisp:1.2 --- elephant/src/metaclasses.lisp:1.1 Sun Aug 29 09:47:00 2004 +++ elephant/src/metaclasses.lisp Sun Aug 29 22:40:06 2004 @@ -12,6 +12,10 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; <ablumberg(a)common-lisp.net> <blee(a)common-lisp.net> ;;; +;;; This program is released under the following license +;;; ("GPL"). For differenct licensing terms, contact the +;;; copyright holders. +;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software @@ -137,7 +141,8 @@ (defun make-persistent-reader (name) (lambda (instance) - (declare (type persistent instance)) + (declare (optimize (speed 3)) + (type persistent instance)) (persistent-slot-reader instance name))) (defmacro persistent-slot-writer (new-value instance name) @@ -154,7 +159,8 @@ (defun make-persistent-writer (name) (lambda (new-value instance) - (declare (type persistent instance)) + (declare (optimize (speed 3)) + (type persistent instance)) (persistent-slot-writer new-value instance name))) (defgeneric handle-optimized-accessors (slot-def))

1 0

[elephant-cvs] CVS update: elephant/src/libsleepycat.c
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant/src In directory common-lisp.net:/tmp/cvs-serv7840/src Modified Files: libsleepycat.c Log Message: new license Date: Sun Aug 29 22:39:30 2004 Author: blee Index: elephant/src/libsleepycat.c diff -u elephant/src/libsleepycat.c:1.5 elephant/src/libsleepycat.c:1.6 --- elephant/src/libsleepycat.c:1.5 Sun Aug 29 09:54:13 2004 +++ elephant/src/libsleepycat.c Sun Aug 29 22:39:29 2004 @@ -12,6 +12,10 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; <ablumberg(a)common-lisp.net> <blee(a)common-lisp.net> ;;; +;;; This program is released under the following license +;;; ("GPL"). For differenct licensing terms, contact the +;;; copyright holders. +;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software

1 0

[elephant-cvs] CVS update: elephant/src/elephant.lisp
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant/src In directory common-lisp.net:/tmp/cvs-serv7815/src Modified Files: elephant.lisp Log Message: new license, non-interned exports Date: Sun Aug 29 22:38:43 2004 Author: blee Index: elephant/src/elephant.lisp diff -u elephant/src/elephant.lisp:1.5 elephant/src/elephant.lisp:1.6 --- elephant/src/elephant.lisp:1.5 Sun Aug 29 09:53:27 2004 +++ elephant/src/elephant.lisp Sun Aug 29 22:38:43 2004 @@ -12,6 +12,10 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; <ablumberg(a)common-lisp.net> <blee(a)common-lisp.net> ;;; +;;; This program is released under the following license +;;; ("GPL"). For differenct licensing terms, contact the +;;; copyright holders. +;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software @@ -39,22 +43,22 @@ (defpackage elephant (:nicknames ele :ele) (:use common-lisp sleepycat) - (:shadow with-transaction) - (:export *store-controller* *current-transaction* *auto-commit* - open-store close-store - store-controller open-controller close-controller - with-open-controller controller-path controller-environment - controller-db controller-root add-to-root get-from-root - persistent persistent-object persistent-metaclass - persistent-collection btree get-value remove-kv - db-transaction-begin db-transaction-abort db-transaction-commit - with-transaction - db-env-set-lock-detect db-env-get-lock-detect - db-transaction-id db-env-lock-id db-env-lock-id-free - db-env-lock-get db-env-lock-put with-lock - db-env-set-timeout db-env-get-timeout - db-env-set-flags db-env-get-flags - run-elephant-thread + (:shadow #:with-transaction) + (:export #:*store-controller* #:*current-transaction* #:*auto-commit* + #:open-store #:close-store + #:store-controller #:open-controller #:close-controller + #:with-open-controller #:controller-path #:controller-environment + #:controller-db #:controller-root #:add-to-root #:get-from-root + #:persistent #:persistent-object #:persistent-metaclass + #:persistent-collection #:btree #:get-value #:remove-kv + #:db-transaction-begin #:db-transaction-abort + #:db-transaction-commit #:with-transaction + #:db-env-set-lock-detect #:db-env-get-lock-detect + #:db-transaction-id #:db-env-lock-id #:db-env-lock-id-free + #:db-env-lock-get #:db-env-lock-put #:with-lock + #:db-env-set-timeout #:db-env-get-timeout + #:db-env-set-flags #:db-env-get-flags + #:run-elephant-thread ) #+cmu (:import-from :pcl

1 0

[elephant-cvs] CVS update: elephant/src/controller.lisp
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant/src In directory common-lisp.net:/tmp/cvs-serv7763/src Modified Files: controller.lisp Log Message: new license, better defaults for root methods Date: Sun Aug 29 22:37:58 2004 Author: blee Index: elephant/src/controller.lisp diff -u elephant/src/controller.lisp:1.5 elephant/src/controller.lisp:1.6 --- elephant/src/controller.lisp:1.5 Sun Aug 29 09:48:34 2004 +++ elephant/src/controller.lisp Sun Aug 29 22:37:58 2004 @@ -12,6 +12,10 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; <ablumberg(a)common-lisp.net> <blee(a)common-lisp.net> ;;; +;;; This program is released under the following license +;;; ("GPL"). For differenct licensing terms, contact the +;;; copyright holders. +;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software @@ -53,30 +57,27 @@ creation, counters, locks, the root and garbage collection, et cetera.")) -(defgeneric add-to-root (sc key value)) -(defgeneric get-from-root (sc key)) -(defgeneric remove-from-root (sc key)) (defgeneric cache-instance (sc obj)) (defgeneric get-cached-instance (sc oid class-name)) (defgeneric next-oid (sc)) (defgeneric open-controller (sc &key recover recover-fatal thread)) (defgeneric close-controller (sc)) -(defmethod add-to-root ((sc store-controller) key value) +(defun add-to-root (key value &key (store-controller *store-controller*)) "Add an arbitrary persistent thing to the root, so you can retrieve it in a later session. Keys may be arbitrary persistables as well (though note collection key semantics!) N.B. this means it (and everything it points to) won't get gc'd." - (setf (get-value key (controller-root sc)) value)) + (setf (get-value key (controller-root store-controller)) value)) -(defmethod get-from-root ((sc store-controller) key) +(defmethod get-from-root (key &key (store-controller *store-controller*)) "Get a persistent thing from the root." - (get-value key (controller-root sc))) + (get-value key (controller-root store-controller))) -(defmethod remove-from-root ((sc store-controller) key) +(defmethod remove-from-root (key &key (store-controller *store-controller*)) "Get a persistent thing from the root." - (remove-kv key (controller-root sc))) + (remove-kv key (controller-root store-controller))) (defmethod cache-instance ((sc store-controller) obj) "Register an instance of a user persistent-class with the

1 0

[elephant-cvs] CVS update: elephant/src/collections.lisp
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant/src In directory common-lisp.net:/tmp/cvs-serv7729/src Modified Files: collections.lisp Log Message: new license, should return values on setf Date: Sun Aug 29 22:36:49 2004 Author: blee Index: elephant/src/collections.lisp diff -u elephant/src/collections.lisp:1.5 elephant/src/collections.lisp:1.6 --- elephant/src/collections.lisp:1.5 Sun Aug 29 09:48:04 2004 +++ elephant/src/collections.lisp Sun Aug 29 22:36:48 2004 @@ -12,6 +12,10 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; <ablumberg(a)common-lisp.net> <blee(a)common-lisp.net> ;;; +;;; This program is released under the following license +;;; ("GPL"). For differenct licensing terms, contact the +;;; copyright holders. +;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software @@ -53,12 +57,14 @@ (defmethod get-value (key (ht btree)) (declare (optimize (speed 3) (safety 0) (space 3))) (buffer-write-int (oid ht) *key-buf*) - (let ((key-length (serialize key *key-buf*))) + (let* ((key-length (serialize key *key-buf*)) + (buf (db-get-key-buffered + (controller-db *store-controller*) + (buffer-stream-buffer *key-buf*) + key-length))) (declare (type fixnum key-length)) - (deserialize (db-get-key-buffered - (controller-db *store-controller*) - (buffer-stream-buffer *key-buf*) - key-length)))) + (if buf (values (deserialize buf) T) + (values nil nil)))) (defmethod (setf get-value) (value key (ht btree)) (declare (optimize (speed 3) (safety 0))) @@ -69,7 +75,8 @@ (buffer-stream-buffer *key-buf*) key-length (buffer-stream-buffer *out-buf*) val-length :transaction *current-transaction* - :auto-commit *auto-commit*))) + :auto-commit *auto-commit*) + value)) (defmethod remove-kv (key (ht btree) &key (transaction *current-transaction*)

1 0

[elephant-cvs] CVS update: elephant/src/classes.lisp
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant/src In directory common-lisp.net:/tmp/cvs-serv7699/src Modified Files: classes.lisp Log Message: new license Date: Sun Aug 29 22:36:18 2004 Author: blee Index: elephant/src/classes.lisp diff -u elephant/src/classes.lisp:1.4 elephant/src/classes.lisp:1.5 --- elephant/src/classes.lisp:1.4 Sun Aug 29 09:46:34 2004 +++ elephant/src/classes.lisp Sun Aug 29 22:36:18 2004 @@ -12,6 +12,10 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; <ablumberg(a)common-lisp.net> <blee(a)common-lisp.net> ;;; +;;; This program is released under the following license +;;; ("GPL"). For differenct licensing terms, contact the +;;; copyright holders. +;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software

1 0

[elephant-cvs] CVS update: elephant/elephant.asd
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv7674 Modified Files: elephant.asd Log Message: new license Date: Sun Aug 29 22:35:52 2004 Author: blee Index: elephant/elephant.asd diff -u elephant/elephant.asd:1.3 elephant/elephant.asd:1.4 --- elephant/elephant.asd:1.3 Sun Aug 29 09:45:49 2004 +++ elephant/elephant.asd Sun Aug 29 22:35:52 2004 @@ -12,6 +12,10 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; <ablumberg(a)common-lisp.net> <blee(a)common-lisp.net> ;;; +;;; This program is released under the following license +;;; ("GPL"). For differenct licensing terms, contact the +;;; copyright holders. +;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software

1 0

[elephant-cvs] CVS update: elephant/TUTORIAL
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv7649 Modified Files: TUTORIAL Log Message: updates, persistent classes, collections, threading, performance Date: Sun Aug 29 22:35:37 2004 Author: blee Index: elephant/TUTORIAL diff -u elephant/TUTORIAL:1.3 elephant/TUTORIAL:1.4 --- elephant/TUTORIAL:1.3 Sun Aug 29 09:51:02 2004 +++ elephant/TUTORIAL Sun Aug 29 22:35:37 2004 @@ -90,12 +90,12 @@ You can put something into the root object by -* (add-to-root *store-controller* "my key" "my value") +* (add-to-root "my key" "my value") => NIL and get things out via -* (get-from-root *store-controller* "my key") +* (get-from-root "my key") => "my value" => T @@ -115,14 +115,14 @@ things: numbers (except for complexes, which will be easy to support), symbols, strings, nil, characters, pathnames, conses, hash-tables, arrays, CLOS objects. Nested and -circular structures are allowed. Basically everything -except lambdas, closures, structures, packages and streams. -(These may eventually get supported too.) +circular things are allowed. You can serialize basically +anything except lambdas, closures, structures, packages and +streams. (These may eventually get supported too.) Unfortunately Berekely DB doesn't understand Lisp, so Lisp data needs to be serialized to enter the database, (e.g. converted to byte arrays), and deserialized on the way -out. This introduces some caveats: +out. This introduces some caveats (not unique to Elephant): 1) Lisp identity can't be preserved. Since this is a store which persists across invocations of Lisp, this probably @@ -130,12 +130,12 @@ * (setq foo (cons nil nil)) => (NIL) -* (add-to-root *store-controller* "my key" foo) +* (add-to-root "my key" foo) => NIL -* (add-to-root *store-controller* "my other key" foo) +* (add-to-root "my other key" foo) => NIL -* (eq (get-from-root *store-controller* "my key") - (get-from-root *store-controller* "my other key")) +* (eq (get-from-root "my key") + (get-from-root "my other key")) => NIL As a consequence, btrees have a sort of mishmash eql / @@ -145,7 +145,7 @@ * (setf (car foo) T) => T -* (get-from-root *store-controller* "my key") +* (get-from-root "my key") => (NIL) You can of course manually re-input objects. @@ -175,12 +175,12 @@ * (setq foo (make-instance 'my-persistent-class)) => #<MY-PERSISTENT-CLASS {492F4F85}> -* (add-to-root *store-controller* "foo" foo) +* (add-to-root "foo" foo) => NIL -* (add-to-root *store-controller* "bar" foo) +* (add-to-root "bar" foo) => NIL -* (eq (get-from-root *store-controller* "foo") - (get-from-root *store-controller* "bar")) +* (eq (get-from-root "foo") + (get-from-root "bar")) => T What's going on here? Persistent classes, that is, classes @@ -190,7 +190,7 @@ are stored in separate entries, keyed by OID and slot. Loading (deserializing) a persistent class -* (get-from-root *store-controller* "foo") +* (get-from-root "foo") => #<MY-PERSISTENT-CLASS {492F4F85}> instantiates the object or finds it from the cache, if it @@ -211,7 +211,7 @@ * (setf (slot1 foo) "three") => "three" -* (slot1 (get-from-root *store-controller* "bar")) +* (slot1 (get-from-root "bar")) => "three" Although it is hard to see here, serialization / @@ -219,6 +219,37 @@ than ordinary CLOS objects. Finally, they do not suffer from merge-conflicts (more on this later.) +------------------------------ +Rules about Persistent Classes +------------------------------ + +Using the persistent-metaclass metaclass declares all slots +to be persistent by default. To make a non-persistent slot +use the :transient t flag. Class slots are never persisted, +for either persistent or ordinary classes. (Is this the +right behavior?) + +Readers, writers, accessors, and slot-value-using-class are +instrumented. Because slot-value is not a generic function, +it is not guaranteed to work properly with persistent slots +-- don't use it! + +Persistent classes may inherit from other classes. slots +inherited from persistent classes remain persistent. +transient slots and slots inherited from ordinary classes +remain transient. + +Ordinary classes cannot inherit from persistent classes -- +slots need to get stored! Likewise, once a slot is declared +persistent, it cannot later be changed to a transient slot. + +Note that the database is read every time you access a slot. +In particular, if your slot value is not an immediate value, +this will cons the value. Gets are not an expensive +operation (I can do a million reads in 30 seconds), but if +you're concerned, cache values. (In the future we will +provide automatic value caching.) + ------------ Transactions ------------ @@ -258,12 +289,102 @@ If for some reason (like db error) you decide to abort, you can do so via (db-transaction-abort). -All of this is packaged up in two macros: with-transaction -and with-transaction-retry. The first starts a new -transaction, executes the body, then tries to commit the +All of this is packaged up in with-transaction. It starts a +new transaction, executes the body, then tries to commit the transaction. If anywhere along the way there is a database -error, the transaction is aborted. +error, the transaction is aborted, and it attempts to retry +(a fixed number of times) by re-executing the whole body. + +----------- +Collections +----------- + +The btrees class are to hash-tables as persistent-objects +are to ordinary objects. btrees have a hash-table-like +interface, but store their keys and values directy in a +Sleepycat btree. Btrees may be persisted simply by their +OID. Hence they have all the nice properties of persistent +objects: identity, fast serialization / deserialization, no +merge conflicts..... + +* (defvar friends-birthdays (make-instance 'btree)) +=> FRIENDS-BIRTHDAYS + +* (add-to-root "friends-birthdays" friends-birthdays) +=> #<BTREE {4951CF6D}> + +* (setf (get-value "Andrew" friends-birthdays) "12/22/1976") +=> "12/22/1976" + +* (get-value "Andrew" friends-birthdays) +=> "12/22/1976" +=> T -with-transaction-retry does the same thing, except on a -failure, after aborting it attempts to automatically retry a -few times: it re-runs the body, and again tries to commit. +Because of serialization semantics, btrees hash on a value, +not identity. This is probably ok for strings, numbers, and +persistent things, but not for ordinary aggregates. + +In the future there will be support for automatically +generating secondary indicies to search or index into btrees +with. + +--------- +Threading +--------- + +Sleepycat plays well with threads and processes. The store +controller is thread-safe by default, that is, can be shared +amongst threads. Transactions may not be shared amongst +threads except serially. One thing which is NOT thread and +process safe is recovery, which should be run when no one is +else is talking to the database environment. Consult the +Sleepycat docs for more information. + +Elephant uses some specials to hold parameters and buffers. +If you're using a natively threaded lisp, you can initialize +these specials to thread-local storage by using the +"run-elephant-thread" function, assuming your lisp creates +thread-local storage for let-bound specials. + +Persisting ordinary aggregate types suffers from something +called "merge-conflicts." Since updating one value of an +aggregate object requires the entire object to be written to +the database, in heavily threaded situations you may +overwrite changes another thread or process has committed. +This is not protected by transactions. + +Consider two processes operating on the same cons: + +-----start--read--update-car--write--commit----------------- +-start------read--update-cdr-----------------write--commit-- + +Although the first process successfully committed their +transaction, their work (writing to the car) will be erased +by the second process's transaction (which writes both the +car and cdr.) + +Persistent classes and persistent collections do not suffer +from merge-conflicts, since each slot / entry is a separate +database entry. + +----------- +Performance +----------- + +Performance is usually measured in transactions per second. +Database reads are cheap. To get more transactions +throughput, consider setting + +* (db-env-set-flags (controller-environment *store-controller*) 1 + :txn-nosync t) + +or look at other flags in the sleepycat docs. This will +greatly increase your throughput at the cost of some +durability; I get around a 100x improvement. This can be +recovered with judicious use of checkpointing and +replication, though this is currently not supported by +Elephant -- see the sleepycat docs. + +The serializer is definitely fast on fixnums, strings, and +persistent things. It is fairly fast but consing with +floats and doubles. YMMV with other values.

1 0

[elephant-cvs] CVS update: elephant/Makefile
by blee＠common-lisp.net 29 Aug '04

29 Aug '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv7618 Added Files: Makefile Log Message: first version, thank you Rafal Strzalinski Date: Sun Aug 29 22:34:10 2004 Author: blee

1 0

← Newer
1
...
78
79
80
81
82
83
84
85
86
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

elephant-cvs