cl-json-devel

Download

cl-json-devel@lists.common-lisp.net

July 2008

1 participants
2 discussions

[cl-json-devel] proposed: improvements to decoder customization
by Boris Smilga 24 Aug '08

24 Aug '08

[This memo started as a follow-up to the e-mail exchange between Henrik and your humble author (now recorded in the TODO file), but, on reviewing it, I thought that it would perhaps be of some interest to the broader readership of cl-json-devel. This is to explain the rather personal style of discourse, and make apologies to a reader who might —but who need not!— feel excluded.] §1. The design of the improved decoder interface laid out below came to be as I was considering an application of CL-JSON that would involve local structured storage of transmitted data in a Berkeley DB. In such a setting one would like to bypass creating Lisp data structures from JSON input, instead packing it directly to FFI C data to be processed by libdb. The question that I asked myself was how to (re)design the decoder so that this task could be dealt with gracefully. I have also checked the scheme against the several probable sets of requirements you had described in your e-mail: > Configurable: I really would want to have it optional exactly how > to decode and encode objects. You are probably right in that your > solution is a good default behaviour, but it will not be perfect in > every situation for every user. So if possible, I would like to > have the decoding and encoding configurable. So backwards > compatibility does not really mean backwards, it means > compatibility between different setups. For example, when doing > testcases or a simple json-bind the old alist setup is good, for a > more advanced setup your code is great. For a secure setup you will > probably want to have access control to what objects you are > allowed to create and so on. One solution will never be sufficient > for everyone. §2. Hence, the general principles which should be obeyed by the implementation, are: A. Separation of concerns. The implementation should comprise a fixed basic level to handle the parsing, JSON well-formedness, and flow control; and a customizable level to produce Lisp data (or perform some other JSON-driven task, as it might please the user to enjoin). You were very rightfully incensed against my intermingling parsing with looking for prototype in READ-JSON-OBJECT. It will become clear below how this can be done away with. B. Fine grain. Not only the handling of objects, but also that of other JSON types should be customizable. In particular, the handling of arrays and strings should be customizable on elemental level, i.e. the user should have a way to determine how the decoder handles elements of arrays and strings. In handling objects, customization should be available for keys as well as for key-value pairs. §3. This suggests a design similar in spirit to SAX, with a set of handlers triggered by “events”. The current implementation partly follows this scheme by providing the *JSON-OBJECT-FACTORY... callbacks, but there are more kinds of events than can be handled by those three; here's a tentatively exhaustive list: 1. An atomic constant (integer, real, or boolean). 2a. Beginning of string. 2e. A string character or escape sequence. 2z. End of string. 3a. Beginning of array. 3e. An array element. 3z. End of array. 4a. Beginning of object. 4k. An object property key. 4v. An object property value. 4z. End of object. Accordingly, we need at least as many handlers. The handlers for (1), (2e), (3e), (4k/v) shall be passed the token that triggered the event; (2a), (3a), and (4a) shall produce some fresh “accumulator” value that is then piped through successive calls to (2e), (3e), and (4k/v), respectively. E.g., reading the JSON {"f\u0151o": [1, true]} could result in the following flow of calls to handlers (I use mnemonic names of function-valued handler variables, which are certainly not definitive): READ-JSON-OBJECT 4a: *BEGINNING-OF-OBJECT-HANDLER* () produces some value (⇒) O READ-JSON-STRING 2a: *BEGINNING-OF-STRING-HANDLER* () ⇒ S 2e: *STRING-CHAR-HANDLER* (#x66, S) ⇒ S′ 2e: *STRING-CHAR-HANDLER* (#x151, S′) ⇒ S″ 2e: *STRING-CHAR-HANDLER* (#x6f, S″) ⇒ S‴ 2z: *END-OF-STRING-HANDLER* (S‴) ⇒ T 4k: *OBJECT-KEY-HANDLER* (T) ⇒ K READ-JSON-ARRAY 3a: *BEGINNING-OF-ARRAY-HANDLER* () ⇒ A 1: *INTEGER-HANDLER* ("1") ⇒ I 3e: *ARRAY-ELEMENT-HANDLER* (I, A) ⇒ A′ 1: *BOOLEAN-HANDLER* ("true") ⇒ B 3e: *ARRAY-ELEMENT-HANDLER* (B, A′) ⇒ A″ 3z: *END-OF-ARRAY-HANDLER* (A″) ⇒ V 4v: *OBJECT-VALUE-HANDLER* (K, V, O) ⇒ O′ 4z: *END-OF-OBJECT-HANDLER* (O′) ⇒ P, which is also the return value of READ-JSON-OBJECT. The nature of the values O, S, S′, S″, S‴, T, etc. is of absolutely no concern to the base level of the decoder whose only duty is to faithfully pass them around. For one, a dumb JSON syntax checker can have all handlers set to (CONSTANTLY T). §4. Now the current list semantics can be expressed straightforwardly in terms of these handlers: *BEGINNING-OF-ARRAY-HANDLER* ⇒ (lambda () (let ((list (cons nil nil))) ; First element is never used (cons list list))) ; First and last pair of the list *ARRAY-ELEMENT-HANDLER* ⇒ (lambda (elt accumulator) (destructuring-bind (head . last) accumulator (cons head (setf (cdr last) (cons elt nil))))) *END-OF-ARRAY-HANDLER* ⇒ (lambda (accumulator) (coerce (cdar accumulator) *json-array-type*)) *BEGINNING-OF-OBJECT-HANDLER* ⇒ same as *BEGINNING-OF-ARRAY-HANDLER* *OBJECT-KEY-HANDLER* ⇒ #'json-intern *OBJECT-VALUE-HANDLER* ⇒ (lambda (key value accumulator) (destructuring-bind (head . last) accumulator (cons head (setf (cdr last) (cons (cons key value) nil))))) *END-OF-OBJECT-HANDLER* ⇒ #'cdar *INTEGER-HANDLER* ⇒ #'parse-integer You can easily reconstruct the rest. §5. Our CLOS semantics is a more tricky affair. If I may allow myself to reiterate what I previously said about the handling of prototype fields: > READ-JSON-OBJECT works by reading in the key string and the colon > separator, and then recursively calling READ-JSON to consume > characters from the input and construct the object which would be > the corresponding value. With my modifications, when the key > "prototype" (or whatever the name shoud be) is encountered on the > input, the decoder is told that the value we are going to construct > shall be the prototype object. If we were not able to communicate > this beforehand, we'd have to do much post-processing of the > factory object: look up the prototype (note that, at this point, we > would not know the package and could not yet intern the keys, so > that the matching would be done on strings, degrading the > performance), convert it to some internal format (note that we > would not be able to predict accurately enough what it would have > been decoded to, as that may be influenced by user-side > configuration), remove the prototype and key from the factory, and > only then could we create the object. Put rather abstractly, to overcome this difficulty we need a means to pass certain information from outer to inner recursive calls of READ-JSON-OBJECT, and between handlers invoked on the same level. I currently see two options of implementing this: A. “Accumulator” return values of (3a/e) and (4a/k/v) handlers (what is signified by A, A′, ..., and O, O′ in the above example) are passed down as additional arguments to the recursive calls to READ-JSON-OBJECT and READ-JSON-ARRAY, and are then also passed to (3a) and (4a) handlers in these inner functions (and perhaps also to (1) and (2a)). The (4k) handler, like (4v), receives and produces an “accumulator” value rather than a representation of the key. Thus, the flow in the above example becomes: ... 4k: *OBJECT-KEY-HANDLER* (T, O) ⇒ O′ READ-JSON-ARRAY (..., O′) 3a: *BEGINNING-OF-ARRAY-HANDLER* (O′) ⇒ A ... 3z: *END-OF-ARRAY-HANDLER* (A″) ⇒ V 4v: *OBJECT-VALUE-HANDLER* (V, O′) ⇒ O″ ... This would allow for an implementation of the CLOS semantics along the following lines: *BEGINNING-OF-OBJECT-HANDLER* ⇒ (lambda (&optional (above-accumulator nil not-toplevel)) (let* ((prototype (if not-toplevel (caar above-accumulator))) (list (cons prototype nil))) (cons list list))) *OBJECT-KEY-HANDLER* ⇒ (lambda (key accumulator) (destructuring-bind (head . last) accumulator (let ((prototype (car head))) (if (and (not prototype) (string= key (symbol-name *prototype-name*))) (cons (cons t (cdr head)) last) (cons head (setf (cdr last) (cons (cons key nil) nil))))))) *OBJECT-VALUE-HANDLER* ⇒ (lambda (value accumulator) (destructuring-bind (head . last) accumulator (if (typep value 'prototype) (cons (cons value (cdr head)) last) (progn (setf (cdar last) value) accumulator)))) *END-OF-OBJECT-HANDLER* ⇒ (lambda (accumulator) (destructuring-bind ((prototype . fields) . last) accumulator (let ((*json-object-prototype* prototype)) (json-factory-make-object fields)))) B. In addition to custom handlers, the user is given the possibility to provide a list of dynamic variables which he wishes to have “JSON-structure scope”. That is, the bodies of READ-JSON-OBJECT and READ-JSON-ARRAY are wrapped in PROGVs which establish new dynamic bindings for these variables (using their outer-level bindings for respective initial values). If a handler sets a structure-scope variable, the new value is then visible to all subsequent handlers until the current READ-JSON-OBJECT or READ-JSON-ARRAY loop exits. By constrast, the handlers on the outer levels never see the value change. The flow in the example is unaltered, and the CLOS semantics is implemented thus: *JSON-STRUCTURE-SCOPE-VARIABLES* ⇒ '(*json-object-prototype*) *BEGINNING-OF-OBJECT-HANDLER* ⇒ (lambda () (let ((list (cons nil nil))) (cons list list))) *OBJECT-KEY-HANDLER* ⇒ (lambda (key) (if (and (not *json-object-prototype*) (string= key (symbol-name *prototype-name*))) (setq *json-object-prototype* t)) key) *OBJECT-VALUE-HANDLER* ⇒ (lambda (key value accumulator) (destructuring-bind (head . last) accumulator (if (typep value 'prototype) (progn (setq *json-object-prototype* value) accumulator) (cons head (setf (cdr last) (cons (cons key value) nil)))))) *END-OF-OBJECT-HANDLER* ⇒ (lambda (accumulator) (json-factory-make-object (cdar accumulator))) The choice between the options A and B seems to be one between simplicity of interface and efficiency: a lot of nested PROGVs are likely to incur some cost overhead (I do not feel myself qualified enough to predict exactly how much). On the other hand, handlers are rather more readable, as you can see in the above comparison, and the interface also stays more intuitive that way. §6. Returning to my BDB + CL-JSON application: the (2a), (3a), (4a) handlers coud be customized to call data constructor / initializer functions over the FFI, and (2z), (3z), (4z) to call the DB put method. The uppermost call to READ-JSON would be wrapped up as a transaction. That would be it. Sincerely, - B. Smilga.

3 7

[cl-json-devel] proposed patch: encoding and unencoding CLOS objects
by boris.smilga＠gmail.com 23 Jul '08

23 Jul '08

*** Rationale *** §1. A significant problem I perceive with CL-JSON in its present state is that the operations ENCODE-JSON and DECODE-JSON are not reasonably inverse. E.g.: (ENCODE-JSON-TO-STRING (DECODE-JSON-FROM-STRING "{\"foo\":null,\"bar \":null}")) => "[[\"foo\"],[\"bar\"]]" and (ENCODE-JSON-TO-STRING (DECODE-JSON-FROM-STRING "{}")) => "null" §2. This is the most painful kind of example, because the JSON object having even a single non-null field results in a proper value. Thus, the type of a JSON object emitted by an application that uses ENCODE-JSON can depend not only on the type of the source Lisp object but on its internal structure as well. This is not to say that ENCODE-JSON fusing Lisp vectors with lists and symbols and characters with strings, or DECODE-JSON fusing false with null are particularly pleasant; still, in theses cases we are able to predict typing across JSON interface. Note, however, that (ENCODE-JSON-TO-STRING (DECODE-JSON-FROM-STRING "[]")) would also yield "null". This has to be dealt with as well. §3. The root of the problem is the design decision of representing JSON objects as Lisp alists, and the solution I propose is to revert that decision and represent JSON objects as CLOS objects. This is quite feasible if we make use of meta-object protocol capabilities implemented in a more or less standardized way across most contemporary Common Lisps. In the simple form, decoding a well-formed JSON object should result in the creation of an anonymous class (i.e. an instance of STANDARD-CLASS) with such slots as found in the JSON object. Next, we create an instance of the anonymous class and populate its slots. Conversely, encoding a CLOS object Z should be done by iterating over the value of (CLASS-SLOTS (CLASS-OF Z)) and emitting a name:value pair for each slot bound in Z. §4. Attached herewith is a series of patches implementing the idea. However, I have ventured a step further by allowing JSON objects to include metadata which specify that CLOS objects are to belong (or else inherit) to non-anonymous CLOS classes. The metadata are passed as the field of the object whose name is ``prototype''. (More accurately, the name is determined by the symbol which is the value of the special variable *PROTOTYPE-NAME*: the value (FUNCALL *SYMBOL-TO-STRING-FN* *PROTOTYPE-NAME*) must be STRING= to the name of the field. I have judged the default value 'PROTOTYPE more or less developer-friendly, as the field ``prototype'' in JavaScript objects is already reserved for special purposes and so is unlikely to conflict with anyone's user-level naming schemes. Below, I invariably call this field the prototype field.) The value of the prototype field should be an object, called the prototype, in which the following fields are meaningful: "lispClass":C, "lispSuperclasses":[C1,...,Cn] and "lispPackage":P (all other fields are ignored). The following rules are employed when decoding a JSON object X to a CLOS object Z: (D1) If the prototype of X has a "lispClass":C field, Z shall be an instance of the class identified by the name C, and all fields of X which have no correspondence among the slots of the class C are discarded. (D2) If the prototype of X has a "lispSuperclasses":[C1,...,Cn] field, Z shall be an instance of an anonymous class C whose superclasses are identified by the names C1,...,Cn. The fields of X which have no correspondence among any of the slots of the classes C1,...,Cn are defined in C as direct slots. As a special case, if n=1 and all of X's fields (omitting the prototype) are defined as slots in C1 then Z shall be an instance of C1. (D3) If the prototype has both a "lispClass":C and a "lispSuperclasses":[C1,...,Cn] fields then the rule D1 applies and the latter field is ignored. (D4) If the prototype has a "lispPackage":P field, then the names of the classes in both other fields and the names of the fields in X are interned in the package specified by P instead of the default package KEYWORD. Of course, all names (including P itself) are converted from camel case before using them in Lisp. (D5) If the class of the resulting object Z provides for a slot whose name is the value of the special variable *PROTOTYPE-NAME* (JSON::PROTOTYPE by default) then that slot shall be bound to the object which internally represents the prototype of X (an instance of the class JSON::PROTOTYPE, qv. the code). NB: this slot is never created by the decoder on its own authority but always inherited from the class or superclasses specified. (D6) If the prototype has a "lispClass":"cons" field and such "lispPackage":P field that the interned class name is COMMON-LISP:CONS, the rule D1 does not apply. Instead, Z shall be an alist whose conses' cars are the names and whose conses' cdrs are the respective values of the fields in X. The field names are interned in P, and the prototype field itself is omitted. (D7) If the prototype has a "lispClass":"hashTable" field and such "lispPackage":P field that the interned class name is COMMON-LISP:HASH-TABLE, the rule D1 does not apply. Instead, Z shall be a hash table whose keys are the names and whose values are the respective values of the fields in X. The field names are interned in P, and the prototype field itself is omitted. (D8) The value of null for any of the three fields of a prototype is equivalent to the field being absent. X lacking or having a null prototype is equivalent to the prototype having all null fields. §5. Conversely, when a CLOS object Z is encoded, the encoded JSON object X shall include a prototype reconstructed from Z per following rules: (E1) If the class of Z has a name, the prototype shall have a "lispClass":C field, where C is that name (it is assumed here and below that all names are converted to camel case). (E2) If the class of Z does not have a name, the prototype shall have a "lispSuperclasses":[C1,...,Cn] field, where C1,...,Cn are the names of that class's superclasses. (E3) If Z is an alist, the prototype shall have a "lispClass":"cons" field. (E4) If Z is a hash table, the prototype shall have a "lispClass":"hashTable" field. (E5) The prototype shall have a "lispPackage":P field, where P is the name of a package such that any one of the names C or C1,...,Cn, and any one of the names of the slots bound in Z, is a symbol (either direct or inherited) in the package. (Mutatis mutandis if Z is an alist or a hash table.) If there is no such package, P shall be an unspecific package name, and the program shall signal a warning condition. §6. Below are some examples of the modified decoder / encoder. I have marked the printout with double bars, and the result with =>. This is from an SBCL session; I have also tested the examples in OpenMCL. (IN-PACKAGE JSON) => #<PACKAGE "JSON"> (DECODE-JSON-FROM-STRING "{\"foo\":null,\"bar\":null}") => #<#<STANDARD-CLASS NIL {1268AC79}> {1268D979}> (DESCRIBE *) || #<#<STANDARD-CLASS NIL {1268AC79}> {1268D979}> || is an instance of class #<STANDARD-CLASS NIL {1268AC79}>. || The following slots have :INSTANCE allocation: || BAR NIL || FOO NIL (DESCRIBE (CLASS-OF **)) || #<STANDARD-CLASS NIL {1268AC79}> is a class. It is an instance of || STANDARD-CLASS. || It has no name (the name is NIL). || The direct superclasses are: (STANDARD-OBJECT), and the direct subclasses || are: (). || The class is finalized; its class precedence list is: || (#<STANDARD-CLASS NIL {1268AC79}> STANDARD-OBJECT SB-PCL::SLOT- OBJECT T). || There are 0 methods specialized for this class. (ENCODE-JSON-TO-STRING ***) => "{\"bar\":null,\"foo\":null,\"prototype\":{\"lispClass\":null, \"lispSuperclasses\":[\"standardObject\"],\"lispPackage\":\"json\"}}" (DESCRIBE (DECODE-JSON-FROM-STRING "{}")) || #<STANDARD-OBJECT {11D01F71}> || is an instance of class #<STANDARD-CLASS STANDARD-OBJECT>. (DEFSTRUCT FOO BAR BAZ) => FOO (ENCODE-JSON-TO-STRING (MAKE-FOO :BAR 10 :BAZ 20)) => "{\"bar\":10,\"baz\":20,\"prototype\":{\"lispClass\":\"foo\", \"lispSuperclasses\":null,\"lispPackage\":\"json\"}}" (DECODE-JSON-FROM-STRING *) => #S(FOO :BAR 10 :BAZ 20) (DECODE-JSON-FROM-STRING "{\"bar\":10,\"baz\":20,\"quux\":50,\"prototype\":{\"lispClass\": \"foo\",\"lispPackage\":\"json\"}}") => #S(FOO :BAR 10 :BAZ 20) (DECODE-JSON-FROM-STRING "{\"bar\":10,\"baz\":20,\"quux\":50,\"prototype\":{\"lispClass\": \"cons\",\"lispPackage\":\"json\"}}") => ((BAR . 10) (BAZ . 20) (QUUX . 50)) (MAPHASH (LAMBDA (K V) (PRINT K) (PRINT V)) (DECODE-JSON-FROM-STRING "{\"bar\":10,\"baz\":20,\"quux\":50,\"prototype\":{\"lispClass\": \"hashTable\",\"lispPackage\":\"json\"}}")) || || BAR || 10 || BAZ || 20 || QUUX || 50 => NIL §7. The following additional names are exported from the package JSON: *PROTOTYPE-NAME* special variable (default: JSON::PROTOTYPE) A symbol which determines the name of the prototype field in a JSON object and the name of a slot in a CLOS object which may hold prototype information. As a special case, if *PROTOTYPE-NAME* is NIL the encoder does not add a prototype field when encoding an object that misses a prototype slot or key, and the decoder does not search for or treat in a special manner a prototype field in the input. The latter behaviour results in the ``simple semantics'' of §3: (LET ((*PROTOTYPE-NAME* NIL)) (DECODE-JSON-FROM-STRING "{\"bar\":10,\"baz\":20,\"prototype\":{\"lispClass\":\"foo\", \"lispPackage\":\"json\"}}")) => #<#<STANDARD-CLASS NIL {12524FA1}> {125290A9}> (DESCRIBE *) || #<#<STANDARD-CLASS NIL {12524FA1}> {125290A9}> || is an instance of class #<STANDARD-CLASS NIL {12524FA1}>. || The following slots have :INSTANCE allocation: || BAR 10 || BAZ 20 || PROTOTYPE #<#<STANDARD-CLASS NIL {12521579}> {12524219}> (DESCRIBE (SLOT-VALUE ** 'PROTOTYPE)) || #<#<STANDARD-CLASS NIL {12521579}> {12524219}> || is an instance of class #<STANDARD-CLASS NIL {12521579}>. || The following slots have :INSTANCE allocation: || LISP-CLASS "foo" || LISP-PACKAGE "json" *JSON-ARRAY-TYPE* special variable (default: VECTOR) A type specifier which determines what type to coerce JSON arrays to. The default value has been chosen to cope with the kind of problem mentioned in §2. Thus, unless *JSON-ARRAY-TYPE* is set to LIST the value of (DECODE-JSON-FROM-STRING "[]") shall be #(), yielding "[]" when re-encoded. WITH-OLD-DECODER-SEMANTICS macro Ensures backward compatibility with the old CL-JSON decoder. It can be called as (WITH-OLD-DECODER-SEMANTICS FORM*) where FORMs are an implicit PROGN. FORMs are executed in an environment where JSON objects are invariably decoded to alists, JSON arrays to lists, array keys interned in the package KEYWORD, and array fields called ``prototype'' (or whatever is specified by *PROTOTYPE-NAME*) receive no special treatment. E.g.: (WITH-OLD-DECODER-SEMANTICS (DECODE-JSON-FROM-STRING "{\"bar\":10,\"baz\":20,\"prototype\":{\"lispSuperclasses\": [\"foo\",\"bar\"],\"lispPackage\":\"json\"}}")) => ((:BAR . 10) (:BAZ . 20) (:PROTOTYPE (:LISP-SUPERCLASSES "foo" "bar") (:LISP-PACKAGE . "json"))) *** Summary of patches *** Sun Dec 23 18:10:00 MSK 2007 boris.smilga(a)gmail.com * Added CLOS / MOP infrastructure. Sun Dec 23 18:13:34 MSK 2007 boris.smilga(a)gmail.com * Added decoding of objects to CLOS objects, hash tables or alists. Sun Dec 23 18:16:10 MSK 2007 boris.smilga(a)gmail.com * Added decoding of arrays to vectors or lists. Sun Dec 23 18:17:05 MSK 2007 boris.smilga(a)gmail.com * Added WITH-OLD-DECODER-SEMANTICS. Sun Dec 23 18:17:44 MSK 2007 boris.smilga(a)gmail.com * Added encoding of CLOS objects, and prototypes for alists and hash tables.

3 3