- cxml-devel - mailman3.common-lisp.net

[cxml-devel] XPath in CXML
by Chas Emerick 13 May '07

13 May '07

Greetings, I noticed a post by David Lichteblau on c.l.l where it was mentioned that an XPath impl for CXML is on a todo list. This is something that I've had a run at, although my end point turned out to be far from the XPath standard. Rather than implementing XPath itself, I implemented a "lispy" element set addressing scheme. By lispy, I mean that rather than addressing nodesets using strings, a declarative DSL is used. Here's an example that yields the title element in an XHTML document: (xmatch :root "html" "head" "title") Here's one that yields all of the TD elements in an XHTML document that have a colspan attribute: (xmatch :root (:desc "td" (:@ "colspan"))) And here's one that yields the second TD element in each row of a table, only if it has a colspan of '1': (xmatch "tr" (:child "td" (:@value "colspan" "1"))) The first two examples are context-free, thanks to the :root directive; the third would need to be run on a table element (or list thereof). If it isn't already clear, xmatch is a macro that builds a closure that, when provided with an element, document, or list as context, returns the node-set that matches the XPath-esque definition. This makes it very similar in style and usage to cl-ppcre:create-scanner. It supports a subset of XPath predicates (such as :@, :@value, :index, and a few others) that I have needed in my application so far, but is by no means complete. I'm passing this information on, not because I have code I'm ready to contribute at the moment (although that could be arranged given some time), but because I think this approach (while not standards- compliant) is superior to any potential "direct" XPath implementation for CXML. Perhaps CXML could grow something like xmatch; in addition to it being used directly, a "proper" XPath implementation could be built on top of an xmatch-like facility. I don't want to belabor the point, but this approach is far more flexible, allows for a much richer set of predicates (and custom ones, at that), and doesn't confine the match definition to a flat string -- sexps are good here, for the same reasons why they are good elsewhere. For example, I have a couple of other macros that generate xmatch definitions themselves; XPath strings *can* be generated dynamically, but that is a Dark Path (at least by my standards). I hope I'm not suggesting anything patently obvious -- I'm functionally new to Lisp (again, this being my second tour of duty, after a long hiatus), so this may all be elementary to others. Thanks for your time, Chas Emerick Founder, Snowtide Informatics Systems Enterprise-class PDF content extraction cemerick(a)snowtide.com http://snowtide.com

1 0

[cxml-devel] Forward type reference
by Edi Weitz 27 Mar '07

27 Mar '07

Hi, this defstruct (defstruct (external-entdef (:include entdef) (:constructor make-external-entdef (extid ndata)) (:conc-name #:entdef-)) (extid (error "missing argument") :type extid) (ndata nil :type (or rod null))) in xml-parse.lisp has a forward reference to the type EXTID which is defined later in the same file. LispWorks doesn't like this (although it only warns and carries on) and I /think/ it is not portable CL, but I'm too lazy to look it up. Cheers, Edi.

1 0

[cxml-devel] Support for cxml when running in :inverse-mode
by Lars Rune Nøstdal 11 Mar '07

11 Mar '07

Hey, When running CL with the following set: (setf (readtable-case *readtable*) :invert) ..CXML fails to compile (SBCL, Linux). I've attached a patch that fixes this. I do not thing it breaks for people running with the normal readtable-mode. -- Lars Rune Nøstdal http://nostdal.org/

1 0

[cxml-devel] CXML on OpenMCL
by Sunil Mishra 21 Feb '07

21 Feb '07

Hi, What is the status of CXML on OpenMCL? I'm using the OpenMCL build from 070214. This is a pre-release version that runs on intel macs. I see the following when cxml is loaded: ;;; Checking for wide character support... no, reverting to octet strings. ;;; Building cxml with (UNSIGNED-BYTE 16) RUNES However, when I trace runes:rod=, I see: 0> Calling (RUNES:ROD= #(69 78 84 73 84 89) #(69 78 84 73 84 89)) <0 RUNES:ROD= returned T 0> Calling (RUNES:ROD= "cent" "iexcl") <0 RUNES:ROD= returned NIL 0> Calling (RUNES:ROD= "cent" "nbsp") I think this only happens when a DTD file requests the loading of an entities file. My guess is that files are being opened inconsistently in different spots in cxml. I don't understand CXML's architecture well enough to figure out this problem, perhaps one of you could help with this? Thanks, Sunil

2 3

[cxml-devel] New release tarball
by David Lichteblau 18 Feb '07

18 Feb '07

Hi, a new cxml release is available for download. Changes: * New StAX-like parser interface. * Serialization fixes (thanks to Nathan Bird, Donavon Keithley). * characters.lisp cleanup (thanks to Nathan Bird). * Namespace normalizer bugfixes. * Minor changes: clone-node on document as an extension. DOM class hierarchy reworked. New function parse-empty-document. Fixed the DOM serializer to not throw away local names. Fixed a long-standing bug in the parser for documents without a doctype. ANSI conformance fixes.

1 0

[cxml-devel] BUG: the value of UTF8-DOM::QNAME is not of type UTF8-RUNES:ROD
by Eric Marsden 30 Dec '06

30 Dec '06

Hi, Using CXML from CVS on SBCL, calling the following function on the attached file leads to an error "The value of UTF8-DOM::QNAME is #(111 102 102 105 99 101 58 99 111 108 ...), which is not of type UTF8-RUNES:ROD.". Without the ":recode nil" it parses the file. (defun foo (filename) (with-open-file (in filename :direction :input :element-type '(unsigned-byte 8)) (cxml:parse-stream in (cxml-dom:make-dom-builder) :recode nil))) -- Eric Marsden

2 1

[cxml-devel] Re: & in file parsed by CXML
by David Lichteblau 16 Sep '06

16 Sep '06

Hi Mitch, I have just checked in a fix for the bug you reported on the LispWorks mailing list. Please update to current CVS and try again. Thanks, David

2 1

[cxml-devel] CDATA doesn't preserve whitespace
by Sunil Mishra 16 Sep '06

16 Sep '06

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I've noticed the following in ACL7 on the attached document: CL-USER(115): (cxml:parse-file "C:\\graph.xml" (rune-dom:make-dom-builder)) #<RUNE-DOM::DOCUMENT @ #x212c5a82> CL-USER(116): (dom:map-document (cxml:make-namespace-normalizer (cxml:make-octet-stream-sink *standard-output*)) *) <svg xmlns="http://www.w3.org/2000/svg">
 <script type="text/css">
 
 
 
 </script>
</svg> #<MULTIVALENT stream socket connected from localhost/3813 to localhost/3817 @ #x205003d2> Now, it appears that this doesn't seem to be a problem for the javascript processor in firefox. It still seems to process lines and quotes and so on that have been thus escaped correctly. The XML 1.1 proposal in its description of CDATA states that the only markup that will be recognized by the XML processor is the CDATA end: ``Within a CDATA section, only the CDEnd string is recognized as markup, so that left angle brackets and ampersands may occur in their literal form; they need not (and cannot) be escaped using "<" and "&". CDATA sections cannot nest.'' Can cxml please correctly follow this requirement? Thanks, Sunil -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFC03QjRnqWMAEf1cRAoCLAKC0ZSIqVYeOHEimdQwdLTNsCgdrJgCdECue DAdGZCA+vOYOBDBcI5FRH8g= =5NkM -----END PGP SIGNATURE-----

2 2

[cxml-devel] Escape characters in first tag cause well-formed error
by Mitch Berkson 16 Sep '06

16 Sep '06

When I parse a file containing escape characters (& or <) in the first tag, CXML:parse-file gives the error message: "Error: Document not well-formed: entity not defined: amp". Including the characters in other tags works fine. Is this the expected behavior? Example: Case 1 (escape character in first tag): REPL> (with-open-file (s "example.xml" :direction :output :if-exists :supersede) (write-string "<test a='b&'><child/></test>" s)) --> "<test a='b&'><child/></test>" REPL> (cxml:parse-file "example.xml" (cxml-xmls:make-xmls-builder)) --> Error: Document not well-formed: entity not defined: amp Context: Line 1, column 16 in file://+ Case 2 (escape character in second tag): REPL> (with-open-file (s "example.xml" :direction :output :if-exists :supersede) (write-string "<test a='b'><child a='b&'/></test>" s)) --> (cxml:parse-file "example.xml" (cxml-xmls:make-xmls-builder)) ("test" (("a" "b")) ("child" (("a" "b&")))) Edi Weitz suggests that ENSURE-DTD may be called too late in the P/DOCUMENT function. Thanks for any help. Mitch

1 0

[cxml-devel] Latest CVS checkout is broken
by Ralf Mattes 13 Sep '06

13 Sep '06

This is on GNU/Linux Intel with SBCL: CL-USER ý (require :cxml) ; compiling file "/LISP/cxml/xml/xml-name-rune-p.lisp" (written 13 SEP 2006 05:45:04 PM): ; in: LAMBDA NIL ; (RETURN T) ; ==> ; (RETURN-FROM NIL T) ; ; caught ERROR: ; return for unknown block: NIL The function CXML::WHILE is undefined. [Condition of type UNDEFINED-FUNCTION] Restarts: 0: [RETRY] Retry performing #<ASDF:COMPILE-OP NIL {B5AAAA9}> on #<CXML-SYSTEM::CLOSURE-SOURCE-FILE "xml-name-rune-p" {AFD6801}>. 1: [ACCEPT] Continue, treating #<ASDF:COMPILE-OP NIL {B5AAAA9}> on #<CXML-SYSTEM::CLOSURE-SOURCE-FILE "xml-name-rune-p" {AFD6801}> as having been successful. 2: [ABORT-REQUEST] Abort handling SLIME request. 3: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread" {AE16669}>) Backtrace: 0: ("bogus stack frame") 1: ((LABELS CXML::LETTER-RUNE-P) 0) 2: ((LABELS CXML::NAME-RUNE-P) 0) 3: ((LABELS CXML::PREDICATE-TO-BV) #<FUNCTION (LABELS CXML::NAME-RUNE-P) {B817DA5}>) 4: (NIL) 5: (SB-INT:EVAL-IN-LEXENV (FUNCALL (COMPILE NIL (QUOTE #))) #<NULL-LEXENV>) 6: (READ-PRESERVING-WHITESPACE #<SB-SYS:FD-STREAM for "file /LISP/cxml/xml/xml-name-rune-p.lisp" {B6B6161}> NIL #<SB-SYS:FD-STREAM for "file /LISP/cxml/xml/xml-name-rune-p.lisp" {B6B6161}> T) 7: (READ-PRESERVING-WHITESPACE #<SB-SYS:FD-STREAM for "file /LISP/cxml/xml/xml-name-rune-p.lisp" {B6B6161}> NIL #<SB-SYS:FD-STREAM for "file /LISP/cxml/xml/xml-name-rune-p.lisp" {B6B6161}> NIL) 8: (READ #<SB-SYS:FD-STREAM for "file /LISP/cxml/xml/xml-name-rune-p.lisp" {B6B6161}> NIL #<SB-SYS:FD-STREAM for "file /LISP/cxml/xml/xml-name-rune-p.lisp" {B6B6161}> NIL) 9: (SB-C::READ-FOR-COMPILE-FILE #<SB-SYS:FD-STREAM for "file /LISP/cxml/xml/xml-name-rune-p.lisp" {B6B6161}> 242) 10: (SB-C::SUB-SUB-COMPILE-FILE #<SB-C::SOURCE-INFO >) 11: ((LAMBDA NIL)) 12: (SB-C::%WITH-COMPILATION-UNIT #<CLOSURE (LAMBDA NIL) {B6B5295}>) 13: (SB-C::SUB-COMPILE-FILE #<SB-C::SOURCE-INFO >) 14: (COMPILE-FILE #P"/LISP/cxml/xml/xml-name-rune-p.lisp") Hmm, file xml/xml-name-rune-p.lisp, line 110 has a 'while' -- where is this supposed to be defined? HTH Ralf Mattes

2 1

[cxml-devel] skipping the DTD decl. when `validate' is `nil'
by Sean Champ 09 Sep '06

09 Sep '06

Hello, I've noticed that the function CXML::P/DOCUMENT calls the function CXML::P/DOCTYPE-DECL whether or not CXML::P/DOCUMENT has received a non-nil VALIDATE argument. I'd thought to try and make a patch about that, myself, but I'm not sure how to address the matter. If the input stream is not being validated, the contents of the doctype decl should not matter, for anything -- need not be put, in any part, as an argument to CXML::XSTREAM-OPEN-EXTID. Yet, when the parser is validating, the text of the doctype decl. still must be 'skipped' by the parser. Looking at CXML::P/DOCTYPE-DECL, I'm not sure how to make the parser skip the text of the decl, or what it could possibly return when skipping it. I could appreciate advice on the matter. Thank you -- Sean Champ

2 4

[cxml-devel] documentation string on `sax:start-element'
by Sean Champ 09 Sep '06

09 Sep '06

Hello, I was trying to make some documentation about CXML, then in regards to the SAX API made avaialable with CXML. I noticed the following, in the documentation string for the generic function SAX:START-ELEMENT : The attributes parameter is a list (in arbitrary order) of instances of the `attribute' structure class. The for their namespace-uri and local-name properties, the same rules as for the element name apply. Additionally, namespace-declaring attributes (those whose name is "xmlns" or starts with "xmlns:") are only included if *namespace-prefixes* is non-nil. Having CXML loaded, but I was not able to find a variable named *NAMESPACE-PREFIXES*, in any package. I have noticed that there is the variable SAX:*NAMESPACE-PROCESSING*. My looking briefly at the CXML source text of a method to SAX:START-ELEMENT, it appeared that that would be the name of the variable to which the documentation string would now refer. I've made a diff for the matter. The name of the variable in that documentation string is revised, in the diff, and any end-of-line whitespace is removed from the file. XEmacs automaticaly pruned the whitespace, when I wrote the file -- using the 'nuke-trailing-whitespace' lib, here. Danke -- Sean Champ

2 1

[cxml-devel] Bugs when working with document nodes
by Sunil Mishra 30 Aug '06

30 Aug '06

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Excellent job on cxml! I have unfortunately run into a couple of problems using version 2006-01-05. I don't know if they are known issues... 1. clone-node when applied to a document doesn't work. 2. create-document does not set the dtd of a document based on the supplied document type definition. This for example inhibits get-element-by-id from working. Thanks, Sunil -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE41ErjRnqWMAEf1cRArMuAJ0TNTRzVBtq66SvBeJH9TLGY4z0lQCdHEdJ 445BRIzteFY8mPBxIQCIcW0= =bCp7 -----END PGP SIGNATURE-----

2 12

[cxml-devel] Encoding
by Edi Weitz 28 Aug '06

28 Aug '06

Some source files from CXML (e.g. xml/sax-handler.lisp) claim in the first line that they're UTF-8-encoded, but actually they aren't. LispWorks 5.0 refuses to load them because of that. It'd be nice if that could be fixed. Thanks, Edi.

2 2

[cxml-devel] Bugs in handling namespaces
by Sunil Mishra 27 Aug '06

27 Aug '06

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Given how many issues I'm posting, I should probably start contributing fixes too :-) Is there a starter document that describes the components of cxml at a high level? So, here's what I'm seeing when I try to introduce multiple namespaces into an XML document... Note that the final output states the namespaces for both svg and xlink as implicit. Further, implicit namespace declarations are not necessarily applied to attributes. CL-USER(36): (setq svg (dom:create-document 'rune-dom:implementation "http://www.w3.org/2000/svg" "svg" (dom:create-document-type 'rune-dom:implementation "svg" "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"))) #<RUNE-DOM::DOCUMENT @ #x21142992> CL-USER(37): (dom:set-attribute-ns (dom:document-element svg) "http://www.w3.org/1999/xlink" "href" "#nothing") CL-USER(38): (dom:map-document (cxml:make-namespace-normalizer (cxml:make-octet-stream-sink *standard-output*)) svg :include-doctype :canonical-notations) <svg href="#nothing" xmlns="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg"></svg> Now we see what happens when we start with a document in a string. See the attached file. Here's what happens when I parse and re-output the file: CL-USER(54): (setq svg (cxml:parse-file #p"C:\\test.xml" (rune-dom:make-dom-builder))) #<RUNE-DOM::DOCUMENT @ #x20c05482> CL-USER(55): (dom:map-document (cxml:make-namespace-normalizer (cxml:make-octet-stream-sink *standard-output*)) svg :include-doctype :canonical-notations) <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/2000/svg">
 <defs>
 <rect fill="black" height="10" id="default-node-shape" stroke="blue" width="60"></rect>
 </defs>
</svg> Thanks in advance, Sunil -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE5UqBjRnqWMAEf1cRAtzZAKCN6V3B1d/UTBfFirAZ9RcDhXy7ywCdEkzm 78n1JvO/0dLKsvkHNQuxNpA= =WCs7 -----END PGP SIGNATURE-----

2 4

[cxml-devel] inquiry and proposals - XML schema support
by Sean Champ 22 Aug '06

22 Aug '06

Hello, I would like to inquire as for whether and how there may have been proposed a means for supoprt of XML schemas in CXML. Certainly, I can direct my inquiry towards a search across the the cxml-devel list @ clnet. I had thought that a more direct inquiry would be appropriate. * Approaching an implementation I am aware as that CXML supports a representation of DTDs. I intend to become more familiar with the code by which CXML provides support for DTD representation. I consider that I will have to become more familiar with it, in order to continue about some intended projects. I have been pressing along with some projects to the Tioga project -- now having the documentation done for the first, ableit trivial release to the Tioga Auxiliary Library's tal-base system. Before releasing the documentation, I have been trying to get some things designed-out, for supporting how that documentation will be built -- across all the documentation to the project. Shortly before beginning to write this message, I had not been sure if I would be able to use CXML, immediately, in managing the system of documentatry items to the Tioga project. It had appeared that it was going to result in a bunch of DTD-hacking -- at some point, hopefully dovetailing with an application of CXML. Presently, I have recalled a matter, upon which I had wanted to raise the above inquiry. I do not mean to sound angered about SGML or XML. It is the stuff of a big system, SGML and XML, whether in integration of the both, or disjunctly. In all, XML is widely popular -- even at such as xml.house.gov for instance, let alone in ebXML, WebDAV, and furthermoe -- and SGML has been an object to quite some popularity and quite some work, in its time, as at HyTime, DSSSL, ISMID, and furthermore. Upon the prospect of having an efficient, Common Lisp programming system that supports XML -- namely, CXML -- one thing that I recall, now, that I can apprecate of it: That I may be able to stop using the generally awkward (i.e involving) DTD syntax, and not have to use the more awkard (i.e. involving, and not in CL code) XSD sytnax, so in order to represent structure in markup. We can use Lisp for it. Upon my observing how the following appears -- it was an item, put like on a restaurant napkin, within a resource-references document being written to the Tioga project:  (That may not be a valid DTD-segment, there; I'm still a bit rusty on DTD syntax. The segment would be applied as in extension onto the DocBook DTD.) It was on my lookin at that text, at which I came to recall how much I would rather use CL for it, even for the initialization of DTD information. * Constraining the Markup Definition - DTD or XSD For represenation of information apart from the markup-element names, attribute names, and content models -- e.g. representation of attribute value types more discrete than 'cdata' -- it may be done in a format compatible to a DTD, using <?processing instructions?> within the DTD markup. It may be more approached towards a more more succint representation of the information, however, if it would be appraoched as to be implemented onto XML schemas. So, I thought I would inquire, as for whether and how XML schemas may have been proposed to be supported in CXML. If there is existing work about such, and if there are proposals toward how it would be approached, I should be glad to direct my attention to it. If there is no existing work about such, I would propose that I can try to "take the matter on". * Initial propoals for a design of XSD support onto CXML ** Shared Functionality - DTDs and XSDs, as documentary schemas I would propose that the CXML DTD support code would be regarded, as for what degree of shared functionality would be applicable of the code, as between DTD support and XSD support. ** Operations in the Parser In the operations of the XSD parser, there may be applied some operations for a type-actuated value-translation mechanmism, e.g. UNMARSHAL-VALUE TYPE IN-SUBSTRATE [generic function] MARSHAL-VALUE TYPE OUT-SUBSTRATE IN-SUBSTRATE [generic function] Regarding UNMARSHAL-VALUE, in the case of an XSD parser: - the TYPE would be an object representative of a type -- a class metaobject, if not such as a CMUCL/SBCL CTYPE object (a CTYPE driven approach would be implementation-specific; I am not aware as for whether or how the code for it may be ported to other implementations. I do not know how any implementations beside CMUCL and SBCL would approach 'type handling' and 'type representation', at any level more finite than of a CLASS metaobject. Regardless of it it would be implementation-specific, It would be very convenient w.r.t type translation, to use CTYPE classes as specializers). The TYPE value would indicate the type of the object that must be initialized of the method. (Some extension onto MOP method specialization may be approached as for some optimization -- as for to ensure that the method's return-value type, and the return-value type of the effective method resulting with an end at that method, would be denoited to the compiler as being of the same type as the TYPE argument; this optimization would require that only a class-typed specialization of methods would be supported on the generic function. I would propose this optimization as a "feasible, though not directly necessary" kind of 'milestone' step, as in a project 'roadmap'.) - the IN-SUBSTRATE value, I suspect it would be such as either: (1, w.r.t a SAX approach) a stream, with the cursor positioned somewhere about the input that would have resulted in a certain SAX event (2, w.r.t a DOM approach) an instance of a class in the DOM type system; an object that would be representative of an XML elment. *** DOM or SAX? (Proposed: DOM) A directly SAX-driven approach may be appropriate, there. I consider that a DOM-driven approach may be the more appropriate. I consider that it would be easier to make a DOM-driven approach, and easier to make that approach, in paralllel to other systems that may use a DOM-driven mechanism, on the same input information. In the case of an approach directly utilizing the SAX API, the input infoset -- the in-substrate -- it would be a stream, parsed-across, once (and it would not be determinable until "late" if the infoset would be valid and well-formed). Then, the in-substrate object would still be available, as a stream (whether or not that stream would support repositioning of the stream cursor -- as a socket stream may not, and in "Linux space", would not). In the case of a DOM-driven approach, the input infoset would be parsed into a DOM representation -- ensured, then, as that it would be valid (if not well-formed ??), if it may even be represented in a DOM node-tree. To the UNMARSHAL-VALUE method, the IN-SUBSTRATE would then be a DOM object. After processing in UNMARSHAL-VALUE, the DOM representation of the initial infoset may then be disregarded, or may be retained for other uses; perhaps it may be used as to process the original DOM information, for representaton of that same information onto a CLIM pane. I would propose that an XSD parser in CXML would be addressed onto a DOM-driven approach. *** UNMARSHAL-VALUE In the case of the UNMARSHAL-VALUE operation, the TYPE value would be representative of something about the type (and element-name) of the "target DOM node". The IN-SUBSTRATE would be something similar to what would have been produced of MARSHAL-VALUE (e.g. a class, or a CTYPE). The OUT-SUBSTRATE would be the DOM node supposed to contain the generated DOM node, i.e. the DOM node that UNMARSHAL-VALUE would generate. Barring a non-local return from the method, the process calling UNMARSHAL-VALUE method would be responsible for "linking" the resulting DOM node into the containing node. This would involve a step, outside of the UNMARSHAL-VALUE method; it should result in a gretaer modularization of the code. *** MARSHAL-VALUE I would appraoch the implementaion of MARSHAL-VALUE as it being a later 'milestone'. *** The calling process What would call UNMARSHAL-VALUE, in unmarshaling of an XML schema object : another UNMARSHAL-VALUE method, specizlied on such that would represent the 'root document' of an infoset, and a TYPE representative of a "container" for the XSD information, such as an XML-SCHEMA object. What would call MARSHAL-VALUE, in the marshaling of CL information onto an XML schema object : another MARSHAL-VALUE method, with the following arguments : - IN-SUBSTRATE representing a a container of XSD information - OUT-SUBSTRATE representing either a DOM node, a stream, or a pathname; the DOM-area specialization would be the first I'd suggest to take - TYPE object being NULL; the type information for operations of methods resulting from the call should be determinable on the IN-SUBSTRATE. *** regarding the TYPE argument in MARSHAL-VALUE To retain the TYPE value as an argument to MARSHAL-VALUE, it would serve to retain some consistency if the method would be specialized onto CFFI system objects. **** Exmple onto CFFI A CL INTEGER value may be represented as a value written into a buffer of any given width at or beyond the integer-length of the value -- with allowance for the encoding of a negative value. To export an INTEGER onto a raw, malloc'd memory block, it would be necessary to specify a numeric type for the output, so in order to ensure that the value could be correctly marshaled (i.e encoded onto the external substrate). **** Example onto CXML An XML schema may include any number of type definitions, and may depend on any number of type definitions from another schema. In example: A US citizen's SSN may be represented as a nine-integer value. That nine-integer value may be represented -- typically -- in a conventional CL environment, using a (VECTOR (UNSIGNED-BYTE 4) 9). Typically, in an XML schema, that nine-integer value may be represented as it being an object of type 'SSN'. An object may be initialized so as to represent an XSD-defined type 'SSN'. Given a mechanism for it, that SSN type may be mapped, explicitly, onto the type (VECTOR (UNSIGNED-BYTE 4) 9). Such a mechanism may be implemented directly onto an XML schema, but would have to be operable without requiring modification of an XML schema. Perhaps the XSD-to-CL type-translation mechanism would be sufficiently operable, when operating in an automated manner. Some specialization might still be appropriate. ** XML-SCHEMA, slot TYPES ; class XML-SCHEMA-TYPE To represent a schema's contained body of type definitions within a single unit, it may be approached with an index initialized into an XML-SCHEMA instance, that index containing of a set of values all of the same type -- perhaps, of a class XML-SCHEMA-TYPE. Given an appropriate mechamism for it, such an index may be defined directly onto a VECTOR typed object, with approriate key functions being cached in the thing. ** Class XML-SCHEMA-TYPE an XML-SCHEMA-TYPE object would contain information representative of: 1) the schema-local representation of the value -- supportive of marshaling of the XML-SCHEMA-TYPE onto an XSD document 2) the CL-local representation of the value -- supportive of unmarshaling of an XML node in a document using the associated schema. ** Questions 1) How would a document and a schema be associated, in the CL environment? Each document is mapped to zero or one DTDs. Each *element* in a document *may* be mapped to one schema (NB: I'm not sure if that's officially of the XSD spec, but it would be feasible. It would require some sort of a conventional approach for the identification of the schema that would be intended as to be assigned to a ndoe; one could propose that an xml:schema attribute might be proposed, for it. One could use any namespace, in the development of the proposal. One should have to require that the element containin gthe foo:schema attribute would be valid on the identified schema; something would have to be done, in regards to namespaces, to ensure that the element would also be valid within the schema for the node containing the element.) (You know, I've tried to use Trac for project whiteboarding. I was thrown at the syntax used in the Trac wiki pages; it doesn't appear to support HTML markup, either, in the Trac wiki pages. I stopped short of requesting that the CLNet maintainers would consider providing a Cliki instance for each project; perhaps they would consider it to be a viable proposal, but I have not wanted to increase their workload, in any.)( I think there's an XML processing instruction, specifically applicable for mapping an XML document to an XSD schema -- similar to the DTD declaration, though using a different syntax, like an <?xfoo whatfoo="URI"?> PI. ** Possible adaptation on a ??? class -- slot SCHEM a ??? class would have to be modified so as to contain a slot SCHEMA. slot value type for the SCHEMA slot : (OR NULL XML-SCHEMA) initial form: NIL ** Possible Adaptation on a DOM Parser To handle an XMl schema on a document -- to associate an XML schema with an object representing the document, or something within the document -- A DOM parser would have towatch for that <?xfoo?> XML-schema PI. One could specialize a method about XML processing-instruction DOM objects; one could check the 'name' processing instruction, then dispatch on when name would match the name of the <?xfoo?> XML-schema PI. At that point, the XML schema would have to be already initialized -- then retrieved -- or would have to be initialized, newly, the whatfoo="URI" identified schema would be available. *** Exceptional Situation If the whatfoo="URI" identified schema would *not* be available, a condition should be signaled for it -- just a type of CONDITION. The document must still be parsed, though it could not be validated. ** Identification of XML schema objects - XML-SCHEMA, slot URI # related items : PURI; object indexing; CXML XML catalogues API An XML-SCHEMA object may be identified according to a URI. The mechanism for that identification -- for indexing an XML-SCHEMA object by its identity, and retriving an object by its identity (or triggering the initializationof a new object, then) -- that should be approached in integration with the XML catalogues mechanism. * Onto Implementation I'd like to approach this as into a prototype in the TAL codebaes to the Tioga project. I would propose to approach it as so, in order to facilitate: 1) that I would use the TAL-base system with it, of which I am familiar 2) that I would be able to implement it, without requiring any more work to the project administrators on the CXML project 3) that it would be integral with the documentary system proposed (*cough*) and being developed onto the Tioga projects Of the said documentary system, the design of it is the last hangup before I may make a first release of the TAL codebase. Without a sufficient mechanism for processing the documentation and presenting it in HTML form, it would be in a release incomplete. * Onto Conclusion At that point, I am reminded of why I had wanted to address the initial inquiry, regarding supoprt for XML schemas in CXML. I want to extend the DocBook DTD; I intend to make the extension, firstly, with Common Lisp -- then generating any DTD contents, in the end. If the code I propose to implement of it would require an application of code that I have not released, I must take the implemenation as it being, effectively, nonoperable. To hold-up a release in concern about the documentation, then upon this item, I cannot. To approach this item, it would serve to support development of the system by which the documentation would be handled. I will have to take the text, above, as it constituting a plain/text edition, as a first draft towards some items of reference documentation. To approach it into implementation, I will have to have migrated some measure more of my own code into the TAL archives -- code that I am familiar with, and which I have tested and know the intended efficiency of, besides that it may require some minor refactoring and cleanup, before the source will be checked into the project's source archive. Upon doing so, I must document each item, as in now, as while my attention is directed to it. I can glue the reference documents together, later -- will be producing one reference document per each distinct system defnition. Each system definition may be associated with a roadmap about milestones proposed for the implementation of the system, furthermore. Then, there will be a body of resource reference pages, a body of glossary entry pages, a body of bibliographic entries, and I may extract the code-item refentry pages into individual files. I should address the above proposal about an XML-schema system as it being the body a system that would be named tdoc-xml-xsd. I will have to make an index-reference pointing to the documentation that would be made about it. I can send across what will be the Arch archive identifier for it, and what information would be necessary for making sure that one can access the archives (there's a point in regards to GPG signing on the archives, reuiring that a certain a script will be available in one's Arch configuration, with some range as for how that would be approached -- side-barred with a point about something called 'agpg' and the 'quintuple-agent' system containing it. Then, there's a point in regards to available interfaces on Arch -- e.g. xtla, xetla -- and a sidebar about key management, e.g. via Seahorse, and there's the clnet keyring, such that one should want to have imported into one's GPG configuration, in order to access the archives.) That documentation should include explanation of each of those points, about how one would access and use the archives to the Tioga project -- using Arch, using some stuff in regards to GPG signing on the archives, and using probably XTLA or XETLA as an interface. Once those tools are installed and configured, then they are readily usable. I will transpose the above design proposals into some DocBook SGML. The documents can be convered to XML, with no lossage in the conversion -- no SGML-specific features being used in the documents. As being SGML, the documents may be processed without issue, using the DSSSL stylesheets for DocBook. After the material, denoted above, will be finally transposed into SGML, then I should want to put the material onto a side shelf; I should then move-in the code that I would intend to use for it, finally checkin-in that code into the TAL codebase. That would be a more viable approach than if I was to use source code from archives that I have not published, in the XSD implementation. After that TAL-area material will be imported into the TAL archives, then I may side-shelve that mateiral and return my attention to the XSD work. At some point, I will have to make a roadmap-milestone for testing the work onto the W3C's test code. I may apologize if I have not edited the above to enough of a state of "doneness". I consider that it may be to the interest of the CXML project, if I would mention the design proposed for it, openly. I regard the proposal, above, as it being material like onto a whiteboard. It is material about to be drafted into a working document. I consider that I have figured out some of how I may approach the matter. I should welcome response about the proposals. If there is work already proposed about XSD support in CXML, I hope to denote: I do not intend that this would seem as if it had run over that work. I would be glad to hear of it, in consideration towards the design of an approach for it. In regards to how the tdoc-xml-xsd system would be made most finally available, I propose to develop it in the TDoc codebase. In so doing, I will be able to develop it, using the tools that I will be using onto other Tioga projects. It could be made available from within that codebase, as a stand-alone distribution -- asdf-installable, if not debian-packaged. Upon all consideration of the licensing terms about the item -- Franz LLGPL, namely -- It may be mirrored from within that codebase, furthermore. I've an errand to follow-up about, before I will continue with the above. Good evening -- Sean Champ

1 1

[cxml-devel] DOM classes not exported?
by Sunil Mishra 21 Aug '06

21 Aug '06

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I want to be able to refer to the types present in the DOM IDL definitions in the DOM Core2 definitions. However, classes such as element and node are not present in the DOM package, let alone be exported from that package. Is this intentional? Can you recommend a reliable way of defining methods specialized on the different DOM classes? Thanks, Sunil -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE5O8gjRnqWMAEf1cRAljlAKDSt1tsXQbuEKEy87f/UzTzYx1tzQCdE02F 3QczT6XACccpkq1SReE2PJQ= =Zpxy -----END PGP SIGNATURE-----

2 4

Re: [cxml-devel] characters.lisp improvements
by David Lichteblau 20 Aug '06

20 Aug '06

Quoting Nathan Bird (nathan(a)acceleration.net) > These patches deprecate characters.lisp in favor of the > xml-name-rune-p.lisp. I pulled over #'VALID-NAME-P and #'VALID-NMTOKEN-P > that were the new functions in characters.lisp, and these functions now use > the ones provided by xml-name-rune-p.lisp. Thank you, checked in. d.

1 0

[cxml-devel] patch - xml-name-rune-p - proposing an approach w/o #.
by Sean Champ 17 Aug '06

17 Aug '06

Hello, I was compiling CXML, in SBCL 0.9.15. I noticed that the compiler kept "getting stuck", when compiling the file xml-name-rune-p. Upon looking at the file, it appeared that the 'getting stuck' behavior may have had something to do with the #. form in the file. With some simple reformatting on the forms in that file, I was able to define a set of forms effectively comparable to those under the #. form -- using the same code, essentially, though by a slightly different approach. I've used a number of DEFVAR calls to hold the predicate values, as well as a DEMACRO call defining a 'defpredicate' macro -- the lot of those calls being wrapped in a form like EVAL-WHEN :COMPILE-TOPLEVEL. The code in the file's toplevel calls the 'defpredicate' macro, to define each of the two predicate functions. The patched code still uses the thing-code-onto-bit-vector comparison. The patch for it is attached to this message. The patched code compiles, successfully, in SBCL 0.9.15. The patch affects the definitions of the functions `name-rune-p' and `name-start-p'. I've tested the redefined name-rune-p, in calling it on one ASCII alphabetic charater; it works. Absent of adverbs, -- Sean Champ

3 5

[cxml-devel] questions - utf8-string-to-rod
by Sean Champ 16 Aug '06

16 Aug '06

Hello, While building CXML in SBCL 0.9.15, I'm noticing a warning from the compiler. The warning is occurring about the following form in unparse.lisp (defun utf8-string-to-rod (str) (let* ((bytes (map '(vector (unsigned-byte 8)) #'char-code str)) (buffer (make-array (length bytes) :element-type '(unsigned-byte 16))) (n (decode-sequence :utf-8 bytes 0 (length bytes) buffer 0 0 nil)) (result (make-array n :element-type 'rune))) (map-into result #'code-rune buffer) result)) The warnning is presented as so, in the REPL: ; in: DEFUN UTF8-STRING-TO-ROD ; (MAP-INTO CXML::RESULT #'RUNES:CODE-RUNE CXML::BUFFER) ; --> DO BLOCK LET TAGBODY RETURN-FROM PROGN WHEN COND IF PROGN SETF ; ==> ; (SB-KERNEL:%SET-FILL-POINTER SB-C::RESULT SB-INT:INDEX) ; ; caught WARNING: ; Asserted type ARRAY conflicts with derived type ; (VALUES (SIMPLE-ARRAY CHARACTER (*)) &OPTIONAL). ; See also: ; The SBCL Manual, Node "Handling of Types" I can work around that that warning occurs, certainly -- just telling ASDF to treat the file as if it was compiled "all oakiedoke". I can expect that that warning will occur, every time when I will compile unparse.lisp in SBCL. It would be convenient if that would not be a concern I had thought i might be able to resolve that that warning is occurring. It appears that I cannot. I am stumped about why that warning is occurring. I hope that someone more familiar with the CXML code might know how that warning may occur. Now, maybe the issue will be moot. Upon consideration of the UTF8-STRING-TO-ROD operation -- thinking that I have understood the operations of the present form of that function, correctly -- considering it, in the context of an #+SB-UNICODE host : I would like to inquire, would the IDENTITY function not suffice? I mean, would it not suffice as the sole executor for the body of UTF8-STRING-TO-ROD -- as when the form would be compiled a host in host in which the feature :RUNE-IS-CHARACTER is defined. Please correct me, if I have misunderstood: That the function UTF8-STRING-TO-ROD, when compiled in an SB-UNICODE host, would translate a string to a string? With courtesy, -- Sean Champ

2 1

[cxml-devel] patch - cxml.asd - *.lisp on 'runes' component pathnames
by Sean Champ 16 Aug '06

16 Aug '06

Hello, I was trying to build CXML, when I encountered an error from ASDF -- something to the effect of "NIL is not of type REAL" -- that indicating a problem in regards to a component's pathname, within a system definition. Upon looking into the matter, it became apparent that the problem was in regards to the component-pathname for the 'runes' component in the cxml-runes system. The pathname-name is explicitly established for the component, with no corresponding pathname-type. Going by the ASDF release versioned 1.93, from the CCLAN repository : When ASDF tries to determine a component's pathname, then when that component has a :pathname explicitly set for it, ASDF will use that explicit pathname, merged onto the pathname for the component's containing module. When the explicit pathname has no pathname-type, then the resulting component-pathname will have no pathname-type. Without the pathname-type being set on the 'runes' pathname in the cxml-runes system, it appears it will be impossible to operate on the cxml-runes system, in some configurations. I'm using ASDF 1.93 from CCLAN and SBCL 0.9.14, here. I'm not sure how cxml has been built in other systems. On my own host, the patch attached to this message is necessary for that the cxml-runes system may be compiled by way of ASDF. In the patch, the pathname-type "lisp" is added to the respective pathnames. That aside, I'd like to voice a word of appreciation for that CXML is available. Thank you. -- Sean Champ

2 2

FW: [cxml-devel] characters.lisp improvements
by Nathan Bird 23 Jun '06

23 Jun '06

Allrighty fine, I see how it is. ;-) These patches deprecate characters.lisp in favor of the xml-name-rune-p.lisp. I pulled over #'VALID-NAME-P and #'VALID-NMTOKEN-P that were the new functions in characters.lisp, and these functions now use the ones provided by xml-name-rune-p.lisp. http://www.unwashedmeme.com/cxml/asd-remove-chars.diff (diff to cxml.asd) http://www.unwashedmeme.com/cxml/characters-merge.diff (against xml/xml-name-rune-p.lisp) And characters.lisp can be deleted now. Before doing any of the changes I made sure that #'RUNE-NAME-CHAR-P and #'NAME-RUNE-P functions returned #'EQ results for every character between 0 and +max+. I also changed the compile-time behavior of xml-name-rune-p.lisp, but it should be run-time equivalent(inlined bit-vector lookup). At compile time the code now looks a little bit more like the code in characters.lisp. The special character ranges are now vectors that are separate from the code (the binary search) that examine them, rather than ORing everything together. At compile time it will then evaluate the predicates over every possible value and save the result in a bitvector (this is how it was working before) and garbage collection should be able to reclaim the character-range vectors. This change resulted in the compilation on my machine dropping from ~8.9s to about ~.25s. Changes that might be significant: PREDICATE-TO-BV - (dotimes (i #x10000 r) + (dotimes (i +max+ r) Where +max+ = #xD800. The vector only goes up to max, so there is no point trying any characters above that right? The predicates that use the bit-vector fail for anything above max too. NAME-RUNE-P and NAME-START-RUNE-P (the public inlined functions) - (DEFINLINE NAME-RUNE-P (RUNE) - (SETF RUNE (RUNE-CODE RUNE)) - (AND (<= 0 RUNE ,+max+) - (LOCALLY (DECLARE (OPTIMIZE (SAFETY 0) (SPEED 3))) - (= 1 (SBIT ',(predicate-to-bv #'name-rune-p) - (THE FIXNUM RUNE)))))) + (DEFINLINE NAME-RUNE-P (RUNE) + (SETF RUNE (RUNE-CODE RUNE)) + (LOCALLY (DECLARE (OPTIMIZE (SAFETY 0) (SPEED 3)) + (type fixnum rune)) + (AND (<= 0 RUNE ,+max+) + (= 1 (SBIT ',(predicate-to-bv #'name-rune-p) + RUNE))))) I moved the locally declarations up by a line to include the <= check and declare the type of the rune a little bit earlier (so that <= might be able to take advantage of this). Does anyone know of problems this will cause? TEST RESULTS: Xmlconf:run-all-tests 0/1829 tests failed; 333 tests were skipped Domtest:run-all-tests 0/763 tests failed; 43 tests were skipped Timing the xmlconf tests there wasn't really any significant change in speed or memory. Profiling it showed a bit of a drop in the memory usage, but not by much. For the xmlconf tests, these functions are called a decent amount, but don't contribute all that much to the runtime. These weren't terribly exact tests, but a part of the point here was to try and clean-up and remove some of the duped code. http://www.unwashedmeme.com/cxml/characters-merge-profile.results Heh, at least compilation is faster. Whad'ya think? Nathan (My apologies if you got this twice, it looks like it didn't send the first time.) >-----Original Message----- >From: David Lichteblau [mailto:david@lichteblau.com] >Sent: Tuesday, June 13, 2006 2:02 PM >To: Nathan Bird >Cc: cxml-devel(a)common-lisp.net >Subject: Re: [cxml-devel] characters.lisp improvements > > >However, I have to admit that my characters.lisp duplicates work already >done by Gilbert. Gilbert's functions are in xml-name-rune-p.lisp and >are using inline functions accessing bitvectors. Embarrassingly, I >didn't notice the latter file before writing characters.lisp and then >just stuck a comment into the file instead of fixing it right away. >My apologies for that -- and sorry for writing that comment in german. :-( That's why we have google translate... to get bad translations of offhand comments :-)

1 0

[cxml-devel] Dom-impl improvements
by Nathan Bird 13 Jun '06

13 Jun '06

We are building up a dom in memory as part of some XHTML and XUL generation. Found time/memory being spent in append-child which we make heavy use of. Here are some changes that improve this. The children of a node are stored in a node-list (adjustable vector). Changes involved: -Don't allocate a node list until we actually HAVE children -When allocating a node-list, go ahead and make it 4 elements long (instead of 0) so we don't immediately reallocate, and again. -When vector-push-extending. Let the lisp implementation take care of increasing the size, instead of manually increasing by length 1 every time. Again this reduces constant reallocation. -Use the has-child-nodes function instead of comparing length everywhere. I attached some test results I got (kind of long), here are the summary lines: seconds | consed | calls | sec/call | name ---------------------------------------------------------- 2.092 | 32,227,920 | 708,718 | | Total ;;OLD 0.859 | 25,507,240 | 691,366 | | Total ;;NEW

2 1

[cxml-devel] characters.lisp improvements
by Nathan Bird 13 Jun '06

13 Jun '06

Doing some profiling I found a good portion of time and memory was being spent in rune-in-range-p among others. Investigating the source I found a comment in german that google translates into something like "that can be done nevertheless better". So I did. We now use a binary search instead of a linear search. Type declarations (simple-vector) helped a bit on sbcl. As well as rune-name-char-p had been calling rune-in-range-p with each of a bunch of different lists. So I combined these lists at compile time, and search through the sorted list once. Some tests I did (each one twice): ;;ORGINAL seconds | consed | calls | sec/call | name -------------------------------------------------------- 0.075 | 1,266,472 | 43,752 | 0.000002 | RUNE-IN-RANGE-P 0.010 | 804,632 | 34,574 | 0.0000003 | RUNE-NAME-CHAR-P 0.000 | 1,018,912 | 4,670 | 0.000000 | VALID-NAME-P -------------------------------------------------------- 0.085 | 3,090,016 | 82,996 | | Total seconds | consed | calls | sec/call | name -------------------------------------------------------- 0.067 | 1,222,936 | 43,752 | 0.000002 | RUNE-IN-RANGE-P 0.003 | 1,078,888 | 34,574 | 0.0000001 | RUNE-NAME-CHAR-P 0.000 | 1,036,544 | 4,670 | 0.000000 | VALID-NAME-P -------------------------------------------------------- 0.070 | 3,338,368 | 82,996 | | Total ;;NEW seconds | consed | calls | sec/call | name -------------------------------------------------------- 0.043 | 0 | 37,001 | 0.000001 | RUNE-IN-RANGE-P 0.000 | 833,704 | 34,574 | 0.000000 | RUNE-NAME-CHAR-P 0.000 | 1,056,296 | 4,670 | 0.000000 | VALID-NAME-P -------------------------------------------------------- 0.043 | 1,890,000 | 76,245 | | Total seconds | consed | calls | sec/call | name -------------------------------------------------------- 0.028 | 0 | 37,001 | 0.000001 | RUNE-IN-RANGE-P 0.000 | 808,528 | 34,574 | 0.000000 | RUNE-NAME-CHAR-P 0.000 | 1,116,776 | 4,670 | 0.000000 | VALID-NAME-P -------------------------------------------------------- 0.028 | 1,925,304 | 76,245 | | Total Nathan Bird

2 1

[cxml-devel] NIL Name Causing Error in zstream-push
by Donavon Keithley 18 May '06

18 May '06

Specifically I'm getting "The value NIL is not of type CXML::STREAM-NAME" in the lambda function passed to find-if. For reference: (defun zstream-push (new-xstream zstream) (cond ((find-if (lambda (x) (and (xstream-p x) (eql (stream-name-entity-name (xstream-name x)) (stream-name-entity-name (xstream-name new-xstream))) (eql (stream-name-entity-kind (xstream-name x)) (stream-name-entity-kind (xstream-name new-xstream))))) (zstream-input-stack zstream)) (wf-error zstream "Infinite recursion."))) (push new-xstream (zstream-input-stack zstream)) zstream) FWIW, I'm parsing an XML string that looks like this: <!DOCTYPE request [ <!ENTITY amp "&#38;"> ]> <document>&</document> The problem is that the 'x' argument -- which refers to the xstream for the entire document -- has no name. It's NIL. It was generated via the call chain PARSE-ROD > STRING->XSTREAM > MAKE-ROD-XSTREAM. string->xstream doesn't supply a name to make-rod-xstream. So, should it? Or should the lambda function be testing for a non-NIL name? I'm guessing the latter but I'd rather not guess. Donavon Keithley

2 1