- cxml-devel - mailman3.common-lisp.net

Need help with filtering elements and/or attributes
by Josef Wolf 10 Jan '22

10 Jan '22

Hello, I am new to cxml and need some assistance to get started. Thanks to https://common-lisp.net/project/cxml/quickstart.html, it is fairly easy to read existing xml and write them back (almost) identical: (defun parse-dom (file) (cxml:parse file (cxml-dom:make-dom-builder))) (defun serialize-dom (file dom new-extension) (with-open-file (out (make-pathname :defaults file :type new-extension) :direction :output :if-exists :supersede :element-type '(unsigned-byte 8)) (write-sequence #(#xef #xbb #xbf) out) ;; byte order mask (dom:map-document (cxml:make-octet-stream-sink out :encoding "utf-8") dom))) (let ((file #p"some-file.xml")) (serialize-dom file (parse-dom file) "new-xml")) with this, I can read a file and write it back. With diff -uw some-file.xml some-file.new-xml I can confirm that those files are (semantically) identical. But how do I go further? As a start, I would like to remove elements and/or attributes while writing. From the docs, I seem to understand that this can be done using sinks. But I just can't figure how to create my custom sink to do the filtering. And three das googling did not reveal any examples/howtos/tutorials. Can somebody provide an example how to create a custom sink and which methods can be specialized to do such filtering? Thanx! -- Josef Wolf jw(a)raven.inka.de

1 0

cxml-xpath depends on deleted xpath/test
by Ernesto Alfonso 05 Jul '20

05 Jul '20

I mistakenly reported this issue in plexippus-xpath instead: https://github.com/gonzojive/plexippus-xpath/issues/2 I was able to work around this by removing references to xpath/test and xpath-test from cxml-xpath. It appears that the test is no longer a separate asdf component. Ernesto

1 0

Loop while parsing
by Sven Emtell 09 Nov '18

09 Nov '18

I have noticed that cxml is looping for a certain combination of xml/dtd. It loops in the recurse function inside compile-content-model in xml-parse.lisp. I cannot fix it myself but I am hoping that somebody else can find a solution using my minimal reproducible case, which can be found at https://doremir.com/files/cxml/xml-test.zip <https://doremir.com/files/cxml/xml-test.zip>. The xml file from the case looks like this: <?xml version="1.0" encoding="UTF-8" standalone="no"?> <!DOCTYPE a PUBLIC "" ""> <a> <c/> </a> And the dtd file looks like this: <!ELEMENT b EMPTY> <!ELEMENT c EMPTY> <!ELEMENT a ((b?)*, c?)> The real world case is from MusicXML parsing and the problematic xml excerpt looks like this: <sound tempo="90"> <offset sound="yes">8</offset> </sound> And the corresponding dtd excerpt looks like this: <!ELEMENT sound ((midi-device?, midi-instrument?, play?)*, offset?)> I have verified this using version "cxml-2008-11-30" from the Closure XML Parser home page at https://common-lisp.net/project/cxml/index.html and also with the latest version from Quicklisp "cxml-20181018". By the way, why isn't the latest version downloadable from the Closure XML Parser home page? Thanks! Sven

1 0

Re: Parsing big XML files with klacks and sbcl
by Mark Janssen 30 May '18

30 May '18

On Wed, May 30, 2018 at 9:16 PM, Paul Rodriguez <paulmrodriguez(a)gmail.com> wrote: > Does your file contain namespaces? Klacks has a well-known memory leak where > namespaces are concerned. It maintains a stack of namespaces, but never > actually pops that stack. > > If that's your problem, it's a trivial fix: > > https://github.com/TBRSS/FXML/commit/7268adcd2cd40dcc407c1941327e8df4299273… I am not using namespaces, however applying the fix anyway seems to keep the unbounded memory use in check. It seems that this is indeed an issue cause by the very tight loop overwhelming the gc. As soon as I do some work in the loop the memory use is bounded.

1 0

Parsing big XML files with klacks and sbcl
by Mark Janssen 30 May '18

30 May '18

I am trying to parse a big xml file (around several GBs) and I am using klacks because of the the size. However it seems that there is some leak during parsing because the memory use continuously increase until sbcl runs out of memory. What am I missing? Regards, Mark Some info: $ sbcl --version SBCL 1.4.8 The script: (ql:quickload 'cxml) (defparameter *src* (cxml:make-source (pathname "huge.xml"))) (loop while t do (klacks:consume *src*)) A (room t) call when breaking to the debugger: 0] (gc) ; No debug variables for current frame: using EVAL instead of EVAL-IN-FRAME. NIL 0] (room t) ; No debug variables for current frame: using EVAL instead of EVAL-IN-FRAME. Dynamic space usage is: 231,363,600 bytes. Immobile space usage is: 15,866,480 bytes (116,720 bytes overhead). Read-only space usage is: 0 bytes. Static space usage is: 704 bytes. Control stack usage is: 9,648 bytes. Binding stack usage is: 2,064 bytes. Control and binding stack usage is for the current thread only. Garbage collection is currently enabled. Summary of spaces: dynamic immobile static CONS: 198,982,960 bytes, 12,436,435 objects, 100% dynamic. CODE: 13,755,264 bytes, 22,368 objects, 100% immobile, 0% dynamic. SIMPLE-VECTOR: 10,533,136 bytes, 80,217 objects, 100% dynamic. INSTANCE: 7,169,776 bytes, 126,568 objects, 2% immobile, 98% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-64: 3,423,232 bytes, 1,867 objects, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-8: 2,874,208 bytes, 39,494 objects, 100% dynamic. SIMPLE-BASE-STRING: 2,031,264 bytes, 40,187 objects, 100% dynamic. SYMBOL: 1,778,320 bytes, 37,048 objects, 0% static, 67% immobile, 33% dynamic. BIGNUM: 1,327,760 bytes, 40,630 objects, 100% dynamic. SIMPLE-CHARACTER-STRING: 1,156,736 bytes, 13,489 objects, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-32: 888,800 bytes, 24,915 objects, 100% dynamic. FDEFN: 663,360 bytes, 20,730 objects, 0% static, 100% immobile. CLOSURE: 595,344 bytes, 16,722 objects, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-16: 588,608 bytes, 4,832 objects, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-31: 243,616 bytes, 4 objects, 100% dynamic. SIMPLE-ARRAY-SIGNED-BYTE-8: 196,208 bytes, 6,131 objects, 100% dynamic. FUNCALLABLE-INSTANCE: 160,368 bytes, 4,149 objects, 44% immobile, 56% dynamic. SIMPLE-BIT-VECTOR: 44,544 bytes, 100 objects, 100% dynamic. SIMPLE-ARRAY-SIGNED-BYTE-16: 15,072 bytes, 208 objects, 100% dynamic. SIMPLE-ARRAY-SIGNED-BYTE-32: 8,096 bytes, 194 objects, 100% dynamic. VALUE-CELL: 5,584 bytes, 349 objects, 100% dynamic. SIMPLE-ARRAY-FIXNUM: 2,960 bytes, 7 objects, 100% dynamic. ARRAY-HEADER: 2,208 bytes, 28 objects, 100% dynamic. RATIO: 1,024 bytes, 32 objects, 100% dynamic. DOUBLE-FLOAT: 704 bytes, 44 objects, 100% dynamic. WEAK-POINTER: 448 bytes, 14 objects, 100% dynamic. SAP: 256 bytes, 16 objects, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-2: 96 bytes, 2 objects, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-FIXNUM: 80 bytes, 3 objects, 100% dynamic. COMPLEX-DOUBLE-FLOAT: 64 bytes, 2 objects, 100% dynamic. COMPLEX: 32 bytes, 1 object, 100% dynamic. COMPLEX-SINGLE-FLOAT: 32 bytes, 2 objects, 100% dynamic. SIMD-PACK: 32 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-NIL: 32 bytes, 2 objects, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-4: 16 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-7: 16 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-15: 16 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-UNSIGNED-BYTE-63: 16 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-SIGNED-BYTE-64: 16 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-SINGLE-FLOAT: 16 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-DOUBLE-FLOAT: 16 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-COMPLEX-SINGLE-FLOAT: 16 bytes, 1 object, 100% dynamic. SIMPLE-ARRAY-COMPLEX-DOUBLE-FLOAT: 16 bytes, 1 object, 100% dynamic. Summary total: 246,450,368 bytes, 12,916,800 objects. Top 10 dynamic instance types: COMPILED-DEBUG-FUN 1,534,912 bytes, 23,983 objects. COMPILED-DEBUG-FUN-EXTERNAL 1,468,160 bytes, 22,940 objects. COMPILED-DEBUG-INFO 986,016 bytes, 20,542 objects. DEFINITION-SOURCE-LOCATION 241,056 bytes, 7,533 objects. FAST-METHOD-CALL 239,952 bytes, 4,999 objects. SLOT-INFO 220,272 bytes, 4,589 objects. VOP-PARSE 190,624 bytes, 851 objects. VOP-INFO 183,456 bytes, 819 objects. FUN-TYPE 155,520 bytes, 1,620 objects. ARG-INFO 140,928 bytes, 1,468 objects. Other types 1,691,632 bytes, 36,184 objects. Dynamic instance total 7,052,528 bytes, 125,528 objects. Top 10 immobile instance types: LAYOUT 119,616 bytes, 1,068 objects. PACKAGE 4,224 bytes, 33 objects. Immobile instance total 123,840 bytes, 1,101 objects. Top 10 static instance types: Static instance total 0 bytes, 0 objects.

3 4

ASDF 3.3 and naming of CXML ASDF systems
by Kambiz Darabi 07 Feb '18

07 Feb '18

Hello, due to a bug in ASDF 3.3 [1], I had problems loading cxml. As you might know, ASDF 3 expects secondary systems to use a slash in their system name, e.g. foo.asd should only contain a system named 'foo' and if there are additional systems in foo.asd, they should be named 'foo/test', 'foo/api' etc. [2] The attached patch extracts the following systems to their own .asd file: - cxml-dom - cxml-xml - cxml-klacks - cxml-test I have tested the loading of the different systems and they work for me on ASDF 3. Unfortunately, I don't have a lisp implementation with ASDF 2 to test it more thoroughly. I'd be happy to modify the patch, if necessary. Cheers Kambiz Darabi -- m-creations gmbh Acker 2 55116 Mainz Germany W: http://www.m-creations.com M: darabi(a)m-creations.com [1] https://bugs.launchpad.net/asdf/+bug/1739514 [2] https://common-lisp.net/project/asdf/asdf/Components.html#Components

2 2

closure XML fix for SBCL
by Cyrus Harmon 05 Jun '17

05 Jun '17

For some reason, I get a string in SBCL which is an (simple-base string 2) and, unfortunately, that is not a subtype of '(simple-array character)). So to load my file with CXML, I have to change the type of simple-rod to: '(or (simple-array rune) simple-base-string). A patch is attached. Alternative suggestions would also be appreciated. thanks, Cyrus diff --git i/characters.lisp w/characters.lisp index 5fa1aa5..e172687 100644 --- i/characters.lisp +++ w/characters.lisp @@ -24,9 +24,9 @@ (in-package :runes) -(deftype rune () #-lispworks 'character #+lispworks 'lw:simple-char) +(deftype rune () #-(or lispworks sbcl) 'character #+lispworks 'lw:simple-char #+sbcl '(or character base-char)) (deftype rod () '(vector rune)) -(deftype simple-rod () '(simple-array rune)) +(deftype simple-rod () #-sbcl '(simple-array rune) #+sbcl '(or (simple-array rune) simple-base-string)) (definline rune (rod index) (char rod index))

1 0

bug report
by José Ronquillo 11 Feb '16

11 Feb '16

Hello CXML developers. I want to report a bug that I reached working with an ODF (Open Document Format) file, more precisely with the content.xml file that is part of it. The bug has occurred when I try to serialize a document that is just parsed, you can reproduce it with this code: (let ((document (with-open-file (stream (merge-pathnames #P"content.xml") :direction :input :element-type '(unsigned-byte 8)) (cxml:parse stream (stp:make-builder))))) (with-open-file (salida #P"1.xml" :direction :output :if-exists :overwrite) (stp:serialize document (cxml:make-character-stream-sink salida)))) The output 1.xml file is incorrect. Check the second line, columna 91841, here with some context: <table:named-expressions/></office:spreadsheet></office:body></office:document-content>le-row table:style-name="ro1"><table:table-cell table:number-columns-repeated="1024"/></table:table-row></table:table><table:named-expressions/></office:spreadsheet></office:body></office:document-content>ing" table:number-columns-spanned="1" table:number-rows-spanned="3"><text:p>DICIEMBRE </text:p></table:table-cell> The original content.xml file is attached.

2 2

[Cxml-devel] Fix for embedded DTD
by olof＠macrolet.net 09 Mar '15

09 Mar '15

Hi, I saw a question on Stack Overflow[1] Alexandre Rademaker had about handling an inlined DTD in a document and tried to solve the problem with the attached patch (also uploaded[2]). It seems to me that the output looks fine, but it would both be good if someone could confirm that this solution is okay and merge it if so. Best regards, Olof [1]: https://stackoverflow.com/questions/26738465/non-valid-output-of-broadcast-… [2]: https://gist.github.com/Ferada/5bc34d20625dff72e15d

1 0

[Cxml-devel] Patch: 0001-Fixed-potential-infinite-loop-in-ADJUST-VECTOR-EXPON.patch
by Jan Moringen 11 Feb '15

11 Feb '15

Hi, please consider the attached trivial patch which fixes a potential infinite loop in ADJUST-VECTOR-EXPONENTIALLY. Kind regards, Jan

1 0