- closure-devel - mailman3.common-lisp.net

[closure-devel] Dealing with HTML in a non-UTF-8 encoding
by Elias Mårtenson 11 Sep '12

11 Sep '12

In my application I have to deal with all sorts of garbage that poses as HTML. In this particular case, had an email that was encoded in Windows-847. The document had the following header: <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m=" http://schemas.microsoft.com/office/2004/12/omml" xmlns=" http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type *content="text/html; charset=windows-874"*><meta name=Generator content="Microsoft Word 12 (filtered medium)"> The document then contained some text in thai. When attempting to parse this document, Closure-HTML gave me the following error: Corrupted UTF-8 input (initial byte was #b10111110) It's pretty clear that this error message is expected, since the input isn't actually in UTF-8. It seems as though Closure-HTML is making an incorrect assumption here. Now, the proper way to handle this case would be to detect the* *"content" attribute in the "html" tag and reparse the text using the correct encoding. Is this something that can be easily done? I haven't looked at the necessary code changes to do this myself (yet). Regards, Elias

1 0

[closure-devel] Parsing Outlook HTML emails
by Elias Mårtenson 11 Sep '12

11 Sep '12

I am currently faced with the task of parsing HTML emails generated by Outlook. My frustrations with that thing can fill an entire email of its own, so I won't do that. Anyway, one thing it keeps doing is to create lots of non-standard tags of the form <o:p></o:p> and the likes. The problem is that when Closure-HTML parses these, they end up like this: "#BAD TAGp>". I worked around the problem by adding the following check to the function NAME-RUNE-P: (rune= char #/:). This includes the colon as a valid character in a node name, and thus will cause such nodes to be ignored in the generated output. Would it be reasonable to include this fix in an update to Closure-HTML? Regards, Elias

1 0

[closure-devel] Closure html and html5
by Gisle Sælensminde 29 Dec '11

29 Dec '11

I'm using closure-html as a a way to parse web page templates, then inserting the dynamic content and finally deserializing the page. This all works fine as long as I use html4. The problem starts once I want use html5-tags in my templates. Then closure-html will strip those tags away during parsing. I'm talking about tags like <audio>, <video> and <canvas>. I see that closure-html use the html4 dvd to make sure that the parsed html is well formed and conforming. I was thinking whether a html5 DTD could be added, but unfortunately there is no such thing, and w3c has stated that there will be none. I could of cause make a DTD that mostly would do the right thing, but that would never be 100% conforming. Have there been any thoughts about how closure-html could be extended to support html5? -Gisle

1 0

[closure-devel] Array access out of bounds in Closure HTML's sgml parser
by Keith Browne 10 Nov '11

10 Nov '11

We're using Closure HTML and Drakma to extract information from Web pages. We've run across an intermittent fault with one page in particular from YouTube. We had a little difficulty reproducing the bug at first, but we discovered that YouTube was sending us different contents each time. We ran our code in a loop and captured several hundred deliveries of the Web page in question until we got another instance that failed. I've put a copy of the HTML that trips the bug up at http://www.deepsky.com/~tuxedo/youtube-sgml-breaker.html You can see the problem by loading closure-html and drakma and evaluating this form: (chtml:parse (drakma:http-request "http://www.deepsky.com/~tuxedo/youtube-sgml-breaker.html") (chtml:make-lhtml-builder)) On SBCL 1.0.53, I'm getting this error: Index 8192 out of bounds for (SIMPLE-ARRAY CHARACTER (8192)), should be nonnegative and <8192. [Condition of type SB-INT:INVALID-ARRAY-INDEX-ERROR] The error is raised in SGML::READ-LITERAL. I only vaguely understand what's going on in that function. I note that it's raising the error when it's parsing the big block of flashvar-related stuff on line 244 of the HTML file, and if I delete or add an extra character earlier in that line, I can make the error go away. I infer that there's something happening in the character decoding at the point where it needs to grow the buffer that's making it lose, but I can't figure out just what it is. Keith Browne

1 0

[closure-devel] Use of (make-pathname ... :type :unspecific)
by Zach Beane 14 Jun '11

14 Jun '11

In src/parse/html-parser.lisp, there's this form: (make-pathname :directory '(:relative "resources") :type :unspecific) 19.2.2.2.3 says this: Whether a value of :unspecific is permitted for any component on any given file system accessible to the implementation is implementation-defined. A conforming program must never unconditionally use a :unspecific as the value of a pathname component because such a value is not guaranteed to be permissible in all implementations. CLISP signals an error on the form. What's the :type :unspecific for? It seems to work on CLISP (and all other implementations I tried) without it; is it a precaution against a *default-pathname-defaults* that apparently has a value for :type? Zach

2 5

[closure-devel] How to disable <span class="illegalstyle"> from being generated?
by Andrei Stebakov 04 Mar '11

04 Mar '11

I agree that style of the html code I sometimes get leaves much to be desired. In this case there is a <style> tag located inside <td> element. So chtml:parse injects the illegalstyle span in the code. (serialize-to-string (chtml:parse " <table> <tr><td> <style type='text/css'> .someClass { width: 26px; height:26px; } </style> some data </td></tr></table> " (stp:make-builder))) Will produce <html><head></head><body><table><tbody><tr><td> <span class=\"illegalstyle\"> .someClass { width: 26px; height:26px; } </span> some data </td></tr></tbody></table> </body></html> How do I disable this from being generated? Thank you, Andrei

2 2

[closure-devel] How to disregard namespaces
by Andrei Stebakov 04 Mar '11

04 Mar '11

Say I need to parse html that I got from some external source and for some reason there are namespaces in the text: (chtml:parse "<a href='someurl.com' somens:url='someurl.com'>text</a>" (stp:make-builder)) The parser will choke on somens namespace since it's not mapped to any url: 0: (CXML-STP:STP-ERROR "attribute with prefix but no URI")[:EXTERNAL] 1: (CXML-STP:RENAME-ATTRIBUTE #<error printing object>) 2: (CXML-STP:MAKE-ATTRIBUTE "someurl.com" "somens:url" "") 3: ((SB-PCL::FAST-METHOD SAX:START-ELEMENT (CXML-STP-IMPL::BUILDER T T T T)) ..) Is there a way to specify some global variable to turn off namespace processing? I saw *namespace-processing* variable in some other package but it doesn't seem to be relevant in this case. Thank you, Andrei

2 2

[closure-devel] asdf load
by "André Schütz" 28 Feb '11

28 Feb '11

Hello, I have a question about the loading of the asd file of closure-html into my clisp environment. I loaded the required projects closure-common and flexi-streams. These two where loaded successfully into the clisp environment. I tried loading closure-html into my clisp environment and got the following error: (asdf:operate 'asdf:load-op :closure-html) Error: ;; Datei /home/andre/.cache/common-lisp/clisp-2.44.1-unix-/home/andre/.asd-link-farm/closure-html-2010-09-20/src/parse/html-parser.fas wird geladen... *** - MAKE-PATHNAME: Unzulässiges :TYPE-Argument :UNSPECIFIC Do you have any idea? Thanks in advance Andre Schütz -- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de

1 0

[closure-devel] Dependency issue: NETLIB
by Andrew Pennebaker 19 Feb '11

19 Feb '11

Hi, I'm writing a cl-mechanize script to scrape a website. One of the dependency issues is closure-html, which depends on NETLIB. I'm using Quicklisp <http://www.quicklisp.org/> to load closure-html, but I think closure-html expects to load its own dependencies from NETLIB<http://www.netlib.org/>, which is not currently available to Quicklisp. Could closure-html have a fork which finds its dependencies with Quicklisp? Trace: >(ql:quickload "cl-mechanize") ... *** - READ from #<INPUT BUFFERED FILE-STREAM CHARACTER #P"/Users/andrew/quicklisp/dists/quicklisp/software/closure-html-20101006-git/src/parse/sgml-dtd.lisp" @410> : there is no package with name "NETLIB" The following restarts are available: TRY-RECOMPILING :R1 Try recompiling sgml-dtd RETRY :R2 Retry compiling component ("closure-html" "src" "parse" "sgml-dtd"). ACCEPT :R3 Continue, treating compiling component ("closure-html" "src" "parse" "sgml-dtd") as having been successful. ABORT :R4 Give up on "cl-mechanize" ABORT :R5 Abort main loop Specs: cl-mechanize 0.0 Quicklisp 2010121400 CLISP 2.49 MacPorts 1.9.2 Mac OS X 10.6.6 MacBook Pro 5,1 Cheers, Andrew Pennebaker

2 1

Re: [closure-devel] index error in SGML::READ-LITERAL
by Maury Davis 30 Jan '11

30 Jan '11

Apparently it's choking on a huge <input /> tag. On Sat, Jan 29, 2011 at 4:27 PM, Maury Davis <mdavis452(a)gmail.com> wrote: > Hi, > > Evaluating: > > (chtml:parse (drakma:http-request " > http://firedepartmentdirectory.com/location/County-Fire-Departments.aspx?st…") > (cxml-stp:make-builder)) > > I'm getting: > > Index 12288 out of bounds for (SIMPLE-ARRAY CHARACTER (12288)), should be > nonnegative and <12288. > [Condition of type SB-INT:INVALID-ARRAY-INDEX-ERROR] > > Restarts: > 0: [RETRY] Retry SLIME REPL evaluation request. > 1: [*ABORT] Return to SLIME's top level. > 2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "new-repl-thread" > RUNNING {C559489}>) > > Backtrace: > 0: (SGML::READ-LITERAL #<RUNES:XSTREAM NIL> #<SGML::DTD (:PUBLIC > "-//W3C//DTD HTML 4.0 Frameset//EN")> #\") > 1: (SGML::READ-ATTRIBUTE #<RUNES:XSTREAM NIL> #<SGML::DTD (:PUBLIC > "-//W3C//DTD HTML 4.0 Frameset//EN")>) > 2: (SGML::READ-NAME-AND-ATTRIBUTES #<RUNES:XSTREAM NIL> #<SGML::DTD > (:PUBLIC "-//W3C//DTD HTML 4.0 Frameset//EN")>) > 3: (SGML::READ-START-TAG #<RUNES:XSTREAM NIL> #<SGML::DTD (:PUBLIC > "-//W3C//DTD HTML 4.0 Frameset//EN")>) > 4: (SGML::READ-TAG #<RUNES:XSTREAM NIL> #<SGML::DTD (:PUBLIC "-//W3C//DTD > HTML 4.0 Frameset//EN")>) > 5: (SGML::READ-TOKEN* #<RUNES:XSTREAM NIL> #<SGML::DTD (:PUBLIC > "-//W3C//DTD HTML 4.0 Frameset//EN")>) > 6: (SGML:SGML-PARSE #<SGML::DTD (:PUBLIC "-//W3C//DTD HTML 4.0 > Frameset//EN")> #<RUNES:XSTREAM NIL>) > 7: (CLOSURE-HTML::PARSE-XSTREAM #<RUNES:XSTREAM NIL> > #<CXML-STP-IMPL::BUILDER {AC3ED09}>) > 8: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..) > > using SBCL 1.0.44. > > Have: > closure-common-20101107-git > closure-html-20101006-git > cxml-20101107-git > cxml-stp-20101107-git > > installed recently with quicklisp. > > Other web pages are parsed without error. > > Thanks, > Maury Davis >

1 0