September 2004 - elephant-cvs@common-lisp.net

[elephant-cvs] CVS update: elephant/src/berkeley-db.lisp
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant/src In directory common-lisp.net:/tmp/cvs-serv27751/src Modified Files: berkeley-db.lisp Log Message: docstring fix Date: Sun Sep 19 19:46:57 2004 Author: blee Index: elephant/src/berkeley-db.lisp diff -u elephant/src/berkeley-db.lisp:1.1 elephant/src/berkeley-db.lisp:1.2 --- elephant/src/berkeley-db.lisp:1.1 Thu Sep 16 06:19:57 2004 +++ elephant/src/berkeley-db.lisp Sun Sep 19 19:46:56 2004 @@ -1370,5 +1370,5 @@ (lambda (condition stream) (declare (type db-error condition) (type stream stream)) (format stream "Berkeley DB error: ~A" - (db-strerror (db-error-errno condition)))) - :documentation "Berkeley DB / Sleepycat errors.")) + (db-strerror (db-error-errno condition))))) + (:documentation "Berkeley DB / Sleepycat errors."))

1 0

[elephant-cvs] CVS update: elephant/doc/tutorial.texinfo elephant/doc/reference.texinfo elephant/doc/package-elephant.texinfo elephant/doc/notes.texinfo elephant/doc/make-ref.lisp elephant/doc/intro.texinfo elephant/doc/elephant.texinfo elephant/doc/docstrings.lisp elephant/doc/cvs2cl.pl elephant/doc/copying.texinfo
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant/doc In directory common-lisp.net:/tmp/cvs-serv26988/doc Added Files: tutorial.texinfo reference.texinfo package-elephant.texinfo notes.texinfo make-ref.lisp intro.texinfo elephant.texinfo docstrings.lisp cvs2cl.pl copying.texinfo Log Message: first version of texinfo docs, automagic changelog Date: Sun Sep 19 19:44:43 2004 Author: blee

1 0

[elephant-cvs] CVS update: Directory change: elephant/doc
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant/doc In directory common-lisp.net:/tmp/cvs-serv26905/doc Log Message: Directory /project/elephant/cvsroot/elephant/doc added to the repository Date: Sun Sep 19 19:41:51 2004 Author: blee New directory elephant/doc added

1 0

[elephant-cvs] CVS update: elephant/TODO
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv26883 Modified Files: TODO Log Message: updates Date: Sun Sep 19 19:41:43 2004 Author: blee Index: elephant/TODO diff -u elephant/TODO:1.5 elephant/TODO:1.6 --- elephant/TODO:1.5 Thu Sep 16 06:11:21 2004 +++ elephant/TODO Sun Sep 19 19:41:43 2004 @@ -8,35 +8,23 @@ tweak performance of transactions! dynamic-extent in CMUCL / SBCL. -more documentation: reference! +more documentation: texinfo NOTES. -secondary index generation, cursors: - -- secondary indices on the "lisp" side : minor / nil -performance gains and DB handle badness for DB->associate - -- create 2 DBs per sorting function: primary and secondary, -without and with duplicates. - -- in addition to the usual lexicographic sorter, create a -"lisp" version -- sorts primitive types (numbers, strings, -symbols.) use http://oss.software.ibm.com/icu/ for 16-bit -unicode. - -- equality joins have to be done on the lisp side: +equality joins have to be done on the lisp side: end-of-table is not the same as end-of-btree. GC (need cursors) Lispworks stuff (fli:replace-foreign-array...) -bignum fix: CMUCL / SBCL use %bignum-ref. OpenMCL: check -that ldb is non-consing (i think it is), look at -%ldb-fixnum-from-bignum) +bignum fix: OpenMCL: check that ldb is non-consing (i think +it is), look at %ldb-fixnum-from-bignum. profile +%bignum-ref on CMUCL / SBCL. serialize lambdas, closures, packages.....this is hard! -lispy pointer arithmetic (profile sap-alien, etc) +openmcl lispy pointer arithmetic (profile sap-alien, etc). +profile CMUCL / SBCL sap arithmetic. performance hacks: class / slot to ID @@ -53,4 +41,6 @@ incorporate requirements from ICU license -cursor-put : move the cursor after insert. \ No newline at end of file +cursor-put : move the cursor after insert. + +change :transient flag to an allocation type (fix CMUCL!) \ No newline at end of file

1 0

[elephant-cvs] CVS update: elephant/README
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv26856 Modified Files: README Log Message: updates Date: Sun Sep 19 19:40:29 2004 Author: blee Index: elephant/README diff -u elephant/README:1.3 elephant/README:1.4 --- elephant/README:1.3 Tue Aug 31 01:46:12 2004 +++ elephant/README Sun Sep 19 19:40:29 2004 @@ -40,17 +40,23 @@ http://www.common-lisp.net/project/elephant ------------------- -License + Warrenty ------------------- +----------------------------- +Copyright, License + Warrenty +----------------------------- See LICENSE. +------------ +Installation +------------ + +See INSTALL + ---------------------- -Installation and Usage +Tutorial and Reference ---------------------- -See INSTALL and TUTORIAL. +HTML docs and texinfo sources can be found in the docs/ directory. ------ Design

1 0

[elephant-cvs] CVS update: elephant/NOTES
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv26826 Modified Files: NOTES Log Message: updates Date: Sun Sep 19 19:39:59 2004 Author: blee Index: elephant/NOTES diff -u elephant/NOTES:1.4 elephant/NOTES:1.5 --- elephant/NOTES:1.4 Mon Aug 30 23:37:36 2004 +++ elephant/NOTES Sun Sep 19 19:39:59 2004 @@ -3,10 +3,10 @@ GENERAL ------- -this has been optimized for use with CMUCL. it has been -tested and somewhat optimized for allegro. SBCL and OpenMCL -are definitely also desired targets. Lispworks is a target -as well but less so: i don't have access to it. +this has been optimized for use with CMUCL / SBCL. it has +been tested and somewhat optimized for allegro. OpenMCL is +definitely also a target. Lispworks is a target as well but +less so: i don't have access to it. Theoretically one can port this to any lisp with a decent FFI and MOP. However since those are two of the less @@ -46,6 +46,10 @@ slot-boundp-using-class inside of shared-initialize, which necessitates some work. +CMUCL doesn't do non-standard allocation types correctly, so +we've created our own slot definition keyword :transient. +In the future this will change. + Andrew will add some notes here in the future. ----------- @@ -89,8 +93,25 @@ over ordinary hash-tables from the point of view of persistence. -TODO: programmatic way to create secondary indicies -(probably Lisp-level, since FFI callbacks are nasty.) +There is a separate table for BTrees. This is because we +use a hand coded C function for sorting, which understands a +little of the serialized data. It can handle numbers (up to +64-bit bignums -- they are approximated by floats) and +strings (case-insensitive for 8-bit, code-point-order for +16-bit Unicode.) It should be fast but we don't want a +performance penalty on objects. + +Secondary indices are mostly handled on the lisp side, +because of our weird table layout (see below) and to avoid +crossing FFI boundaries. Some unscientific microbenchmarks +indicated that there was no performance benefit on CMUCL / +SBCL, and only minor benefit (asymptotically nil) on +OpenMCL. They have a separate table. Actually two handles +are opened on this table: one which is plain, and one which +is associated to the primary btree table by a no-op indexing +function. Since we maintain the secondary keys ourselves, +the associated handle is good for gets / cursor traversals. +We use the unassociated handle for updates. ---------- CONTROLLER @@ -142,13 +163,15 @@ OID + Slot ID -Collections use +Collections use 2 tables, one for primaries and one for +secondaries (which supports duplicates.) They are keyed on OID + key -the root object is a btree with OID = 0. Since keys are +The root object is a btree with OID = 0. Since keys are lexicographically ordered, this will create cache locality -for items in the same persistent object / collection. +for items in the same persistent object / collection. We +use a custom C sorter for the btree tables. Other layout options: @@ -214,7 +237,7 @@ CMUCL's consing dpb/ldb arithmetic means serializing bignums conses (but they shouldn't have to!) Serializing everything else should not cons (with the exception of maybe symbols -and pathnames.) +and pathnames.) SBCL seems much better with this. Deserialization of fixnums is non-consing. floats appear to cons on CMUCL, i'm not sure if this is just because of @@ -300,15 +323,17 @@ pointer-arithmetic is bignum and therefore consing. TODO: write faster, lispier versions of the -pointer-arithmetic functions. (Definitely possible under -OpenMCL; maybe possible using SAP arithmetic under CMUCL. -Dunno about Allegro, Lispworks.) +pointer-arithmetic functions. This is done for CMUCL / +SBCL. (Definitely possible under OpenMCL. Dunno about +Allegro, Lispworks.) CMUCL et al can't do dynamic-extent buffers, so we use globals bound to specials, which should be thread-safe if properly initialized. While we provide functions talk to -the DB using strings, Elephant itself only uses foreign char -buffers. +the DB using strings, Elephant itself only uses +"buffer-streams", which are structures which have a +stream-like interface to foreign char buffers for reading / +writing C datatypes. Lispworks is much happier passing back and forth statically allocated lisp arrays. since the general string will almost

1 0

[elephant-cvs] CVS update: elephant/NEWS
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv25357 Modified Files: NEWS Log Message: updates for release Date: Sun Sep 19 19:38:26 2004 Author: blee Index: elephant/NEWS diff -u elephant/NEWS:1.4 elephant/NEWS:1.5 --- elephant/NEWS:1.4 Thu Sep 2 17:11:58 2004 +++ elephant/NEWS Sun Sep 19 19:38:25 2004 @@ -1,4 +1,20 @@ +September 19, 2004 - + +Elephant 0.2 released. This is an BETA release. + +New features: + +- Secondary indices and cursors +- PPC Darwin OpenMCL / SBCL +- Doc strings and improved documentation +- An RT-based test suite +- many bugfixes + +This release has been tested on CMUCL 19a, SBCL 0.8.14 and +Allegro 6.2 on x86 Linux and FreeBSD, and OpenMCL 0.14.2-p1 +and SBCL 0.8.14 on PPC Darwin. + September 2, 2004 - The bad news: there was a bug in 0.1 which made OID

1 0

[elephant-cvs] CVS update: elephant/LICENSE
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv24435 Modified Files: LICENSE Log Message: added ICU license bit Date: Sun Sep 19 19:37:25 2004 Author: blee Index: elephant/LICENSE diff -u elephant/LICENSE:1.2 elephant/LICENSE:1.3 --- elephant/LICENSE:1.2 Sun Aug 29 22:32:18 2004 +++ elephant/LICENSE Sun Sep 19 19:37:25 2004 @@ -7,6 +7,12 @@ ("GPL"). For differenct licensing terms, contact the copyright holders. +Portions of this program (namely the C unicode string +sorter) are derived from IBM's ICU: + +http://oss.software.ibm.com/icu/ + +whose copyright and license follows the GPL below. The GNU General Public License (GPL) Version 2, June 1991 @@ -330,4 +336,46 @@ HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. -END OF TERMS AND CONDITIONS \ No newline at end of file +END OF TERMS AND CONDITIONS + + + +ICU License - ICU 1.8.1 and later +COPYRIGHT AND PERMISSION NOTICE + +Copyright (c) 1995-2003 International Business Machines +Corporation and others All rights reserved. + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, +and/or sell copies of the Software, and to permit persons to +whom the Software is furnished to do so, provided that the +above copyright notice(s) and this permission notice appear +in all copies of the Software and that both the above +copyright notice(s) and this permission notice appear in +supporting documentation. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY +KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE +WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR +PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN NO +EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS +NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR +CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING +FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF +CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS +SOFTWARE. + +Except as contained in this notice, the name of a copyright +holder shall not be used in advertising or otherwise to +promote the sale, use or other dealings in this Software +without prior written authorization of the copyright holder. + +------------------------------------------------------------ +All trademarks and registered trademarks mentioned herein +are the property of their respective owners. +

1 0

[elephant-cvs] CVS update: elephant/INSTALL
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv24408 Modified Files: INSTALL Log Message: updates Date: Sun Sep 19 19:37:03 2004 Author: blee Index: elephant/INSTALL diff -u elephant/INSTALL:1.8 elephant/INSTALL:1.9 --- elephant/INSTALL:1.8 Thu Sep 2 17:19:17 2004 +++ elephant/INSTALL Sun Sep 19 19:37:03 2004 @@ -3,8 +3,8 @@ Requirements ------------ -CMUCL 19a, SBCL 0.8.13, OpemMCL 0.14.2, or Allegro CL 6.2. -I've tested under FreeBSD, Linux and OpenMCL / Darwin. A +CMUCL 19a, SBCL 0.8.14, OpemMCL 0.14.2, or Allegro CL 6.2. +I've tested under x86 FreeBSD, Linux and PPC Darwin. A Lispworks version will come if requested. ASDF - http://www.cliki.net/asdf @@ -55,7 +55,7 @@ This compiles src/libsleepycat.c and installs it into -/usr/local/share/common-lisp/elephant/ +/usr/local/share/common-lisp/elephant-0.2/ or where you specified. On Darwin / OS X you need to have the developer tools installed. @@ -72,9 +72,7 @@ (asdf:operate 'asdf:load-op :elephant) This will load and compile Elephant. This will also -automatically load UFFI. You may get "constant redefinition -errors", especially on SBCL. They are not issues, go ahead -and redefine the constants. +automatically load UFFI. I can't seem to make OpenMCL not intern default keyword values of my macros -- something which doesn't happen on @@ -160,3 +158,24 @@ NIL CL-USER> + +------- +Testing +------- + +Elephant uses RT for regression testing, available at: + +http://www.cliki.net/RT + +Once RT is installed, edit tests/elephant-tests.lisp to make +*testdb-path* point to somewhere appropriate. Symlink +elephant-tests.asd to your asdf systems directory, then run + +(asdf:operate 'asdf:load-op :elephant-tests) +(in-package :ele-tests) +(do-all-tests) + +this should take about 5 minutes on decent hardware. Note +that the "no-eval-initform" test fails, this is a known bug +which will get fixed in a future release. +

1 0

[elephant-cvs] CVS update: elephant/CREDITS
by blee＠common-lisp.net 19 Sep '04

19 Sep '04

Update of /project/elephant/cvsroot/elephant In directory common-lisp.net:/tmp/cvs-serv24311 Modified Files: CREDITS Log Message: updates Date: Sun Sep 19 19:36:23 2004 Author: blee Index: elephant/CREDITS diff -u elephant/CREDITS:1.2 elephant/CREDITS:1.3 --- elephant/CREDITS:1.2 Tue Aug 31 01:46:12 2004 +++ elephant/CREDITS Sun Sep 19 19:36:22 2004 @@ -6,13 +6,16 @@ Thanks to: -Sleepycat for Berkeley DB +Sleepycat for Berkeley DB, especially Ron Cohen and Michael +Cahill for answering my questions Kevin Rosenberg for UFFI, answering lots of questions and letting me patch Gary Byers for OpenMCL and answering my questions +Richard Waters (and Paul Dietz, Kevin Rosenberg) for RT + Rafal Strzalinski for the Makefile and package patch The common-lisp.net people for hosting @@ -22,6 +25,10 @@ SLIME for a better environment Erik Enge (CL-IRC) for being an unwitting guinea pig + +Dan Barlow for ASDF + +IBM for ICU Paul Foley for his berkeley-db package (which we didn't use, once we settled on UFFI)

1 0