Hello!
I've faced annoying cl-prevalence behavior which I don't know how to deal with.
For example, consider this code that I want to run within a transaction:
(defun tx-add-item (store index item) (declare (ignore store)) (setf (gethash item index) t))
Current cl-prevalence implementation invokes (reset serialization-state) before each (serialize-*) call. So as long as I insert new items into the index, each log entry becomes bigger and bigger because it needs to deeply serialize "index" parameter with all its elements, including the new ones.
The problem that the index can contains thousands elements, and each of them can be a complex object itself with maybe more index tables. In such case a single transaction can consume a lot of resource.
My proposal is to not reset serialization-state before each transaction serialize step but to keep it until full snapshot is occurred. For me it seems to work correctly, but maybe I miss something?
Alexey Voznyuk wrote:
My proposal is to not reset serialization-state before eachtransaction serialize step but to keep it until full snapshot is occurred. For me it seems to work correctly, but maybe I miss something?
Here's a patch I wrote and use since; it wasn't made for performance but for correctness in maintaining object references.
I intended to merge it in but haven't got around to write tests for it yet. Perhaps you'd like to do that?
Leslie
Leslie P. Polzer wrote:
Alexey Voznyuk wrote:
My proposal is to not reset serialization-state before each transaction serialize step but to keep it until full snapshot is occurred. For me it seems to work correctly, but maybe I miss something?
Here's a patch I wrote and use since; it wasn't made for performance but for correctness in maintaining object references.
I intended to merge it in but haven't got around to write tests for it yet. Perhaps you'd like to do that?
Leslie
Sure, no problems.
Leslie P. Polzer wrote:
Alexey Voznyuk wrote:
My proposal is to not reset serialization-state before each transaction serialize step but to keep it until full snapshot is occurred. For me it seems to work correctly, but maybe I miss something?
Here's a patch I wrote and use since; it wasn't made for performance but for correctness in maintaining object references.
I intended to merge it in but haven't got around to write tests for it yet. Perhaps you'd like to do that?
Leslie
There is something wrong with this patch.
It seems it targets the old cl-prevalence release (cvs one), where serialization.lisp is not splitted yet. But it doesn't matter, I see the following problems there: - the patch seems incomplete (I didn't find where *txn-state* is defined, there are only use cases) - I didn't get what *txn-state* is for. All that it does is doubling 'known-object-id' / 'set-known-object' functions. - There are only xml serialization related fixes, nothing for sexp.
Please check my latest 20090926 cumulative patch where I only removed serialization state clearing on each transaction. There are some tests included.
Alexey Voznyuk wrote:
There is something wrong with this patch. It seems it targets the old cl-prevalence release (cvs one), whereserialization.lisp is not splitted yet. But it doesn't matter, I see the following problems there: - the patch seems incomplete (I didn't find where *txn-state* is defined, there are only use cases)
The following hunks seem to be missing:
diff --git a/lib/src/cl-prevalence/src/package.lisp b/lib/src/cl-prevalence/src/package.lisp index 3b924e9..973e9ca 100644 --- a/lib/src/cl-prevalence/src/package.lisp +++ b/lib/src/cl-prevalence/src/package.lisp @@ -13,6 +13,7 @@ (defpackage :s-serialization (:use :cl) (:export + #:*txn-state* #:serializable-slots #:serialize-xml #:serialize-sexp #:deserialize-xml #:deserialize-sexp diff --git a/lib/src/cl-prevalence/src/prevalence.lisp b/lib/src/cl-prevalence/src/prevalence.lisp index 7d21e2a..acb51d8 100644 --- a/lib/src/cl-prevalence/src/prevalence.lisp +++ b/lib/src/cl-prevalence/src/prevalence.lisp @@ -213,7 +213,9 @@
(defmethod log-transaction ((system prevalence-system) (transaction transaction)) "Log transaction for system" - (let ((out (get-transaction-log-stream system))) + (let ((*txn-state* (make-hash-table)) + (out (get-transaction-log-stream system))) + (declare (special *txn-state*)) (funcall (get-serializer system) transaction out (get-serialization-state system)) (terpri out) (finish-output out)))
- I didn't get what *txn-state* is for. All that it does isdoubling 'known-object-id' / 'set-known-object' functions.
*txn-state* holds the transaction-specific state. The old model only has transaction states via a system-global variable that gets reset before every transaction. This new code moves this state to *txn-state* and uses the system-global variable as a true global state that does not get reset.
Does that make things clear?
- There are only xml serialization related fixes, nothing for sexp.
Sorry about that, it's missing indeed. One of the reasons it's not merged in yet...
Please check my latest 20090926 cumulative patch where I onlyremoved serialization state clearing on each transaction. There are some tests included.
I don't think this will suffice. Can your code handle cycles in the object graph?
Leslie
Leslie P. Polzer wrote:
[snip]
- I didn't get what *txn-state* is for. All that it does isdoubling 'known-object-id' / 'set-known-object' functions.
*txn-state* holds the transaction-specific state. The old model only has transaction states via a system-global variable that gets reset before every transaction. This new code moves this state to *txn-state* and uses the system-global variable as a true global state that does not get reset.
Does that make things clear?
No :) The only purpose of 'state' in serialization is to maintain the objects table already serialized, so we can reference them instead of full persistence procedure. The old cl-prevalence implementation keeps that state only during one 'transaction', so the following transactions need to deeply serialize objects, that were already been serialized during previous transactions. My purpose to maintain that objects table until full snapshot is occurred.
I don't see what *txn-state* is for. It only copies the old state behavior.
[snip]
Please check my latest 20090926 cumulative patch where I onlyremoved serialization state clearing on each transaction. There are some tests included.
I don't think this will suffice. Can your code handle cycles in the object graph?
Sure, no problems here. The references to the already known objects are not serialized.
Check test/test-complex-serialize.lisp in my recent cl-prevalence-3.patch, there are tests for complex objects structure with cycles and several indexes.
Leslie
cl-prevalence-devel@common-lisp.net
-
Alexey Voznyuk -
Leslie P. Polzer