- cl-ppcre-devel@lists.common-lisp.net - mailman3.common-lisp.net

Only strings as names for :named-registers?
by Philipp Marek 08 Apr '15

08 Apr '15

Hi, I wondered why :named-register only allows to use strings as name. I'd like to use a symbol, or, for macro-generated forms, gensyms, because that would be easier for macro-generated forms to associate with the resulting capture index. Sadly symbols don't work currently because of COPY-SEQ in METHOD CONVERT-COMPOUND-PARSE-TREE ((EQL :NAMED-REGISTER) T) Even more, the COPY-SEQ destroys EQuality, so I cannot give some string and look for it later on. What's the reason for the string-requirement? Could we allow symbols, too? Thanks for all answers. Regards, Phil

2 1

Extending regexps to other kinds of sequences
by Ron Garret 08 Apr '15

08 Apr '15

The theory of regular expressions can be applied to any kind of sequence, not just strings. This would be potentially useful for pattern-matching applications, where current approaches make it very cumbersome to say things like, “Match a list containing between three and five integers”. This sort of thing is easy to express in CL-PPCRE tree-like notation as, e.g. (:repetition (:type integer) 3 5) My question is: how hard would be it be to adapt the CL-PPCRE code to handle things like this? Is there a sequence-type-agnostic core in CL-PPCRE that could be easily re-used for this purpose, or is the assumption that regexps only apply to strings woven deeply into the code? Thanks, rg

4 4

[cl-ppcre-devel] debugging of RE matching
by Philipp Marek 01 Oct '12

01 Oct '12

Hello everybody, to make it easier to see how/where a RE does (or does not) match I've started a branch on github: https://github.com/phmarek/cl-ppcre/tree/debugging During compile-time cl-ppcre::*do-debug* decides whether some tracing output is returned by the compiled match function. Example for a successful match: > (cl-ppcre:scan-to-strings #?r(^abc(d|(e+)|fff)[xyz]+) "abcfffzzxzx") *string*: "abcfffzzxzx" SEQ [ 0 0[ ANCHOR [ 0 0[ ANCHOR STR [ 0 3[ "abc" REGISTER [ 3 3[ $1 ALTERNATION [ 3 3[ STR [ 3 3[ "" REGISTER [ 3 3[ $2 SEQ [ 3 3[ STR [ 3 3[ "" STR [ 3 6[ "fff" CHAR-CLASS [ 6 7[ [#<CLOSURE (LAMBDA (CHAR) ... REPETITION [ 7 11[ "abcfffzzxzx" #("fff" NIL) Example for a non-matching try: > (cl-ppcre:scan-to-strings #?r(^abc(d|(e+)|fff)[xyz]+) "abcffffz") *string*: "abcffffz" SEQ [ 0 0[ ANCHOR [ 0 0[ ANCHOR STR [ 0 3[ "abc" REGISTER [ 3 3[ $1 ALTERNATION [ 3 3[ STR [ 3 3[ "" REGISTER [ 3 3[ $2 SEQ [ 3 3[ STR [ 3 3[ "" STR [ 3 6[ "fff" CHAR-CLASS NO MATCH [#<CLOSURE (LAMBDA (CHAR) :IN ... NIL I think this might be useful for other people, so please provide some feedback. Is the documentations' source doc/index.html? Then I'd provide some text there, too. Regards, Phil

3 5

[cl-ppcre-devel] Using (unsigned-byte 8) instead of a (string) as TARGET-STRING
by Philipp Marek 26 Apr '12

26 Apr '12

Hello everybody, I've got a first (unoptimized) patch that allows to use (unsigned-byte 8) instead of a string as TARGET-STRING. My motivation is to use that for searching in big, binary files, which might not fit into RAM with byte => character conversion (which would be 1:4, as the binaries would have to be read as latin1 or similar). Using that patch on a ~3MB file with the string match at about 80% of the file size shows a nice speedup: only half to a third cpu time used, and much less memory usage (for the string). Details: -rw-r--r-- 1 root root 3176746 Mai 3 2010 /usr/share/doc/gcc-4.4-doc/gccint.html string, case-sensitive: 0.360022 seconds of total run time (0.360022 user, 0.000000 system) 907,556,608 processor cycles string, case-insensitive: 0.492031 seconds of total run time (0.492031 user, 0.000000 system) 1,239,853,076 processor cycles (unsigned-byte 8), case-sensitive: 0.108006 seconds of total run time (0.108006 user, 0.000000 system) 274,043,748 processor cycles (unsigned-byte 8), case-insensitive: 0.220013 seconds of total run time (0.220013 user, 0.000000 system) 553,027,836 processor cycles The small "problem" is this (one long line): $ time perl -e '$/=undef; $_=<>; print $1,$2,"\n" if /acr([o0] i)s not defined,\s+the default(\Dvalue,\s*\d+, i)s used/' < /usr/share/doc/gcc-4.4-doc/gccint.html o i value, 1, i real 0m0.016s user 0m0.004s sys 0m0.008s $ perl -v This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi ie. (this) perl5 is still ~8 times faster, including file reading etc. (what the lisp code didn't take into the measurement). (With the /i modifier it's 0.020s.) I've not yet tried to run the whole test suite against that. There are quite a few warnings (unused variable UB8-MODE etc.) - but with higher SAFETY the original CL-PPCRE gave a lot of them, too. I'd like to ask for a quick look at the patch, to get some feedback; with the many duplications I don't really like the result, but the duplicated accesses ("schar" etc.) are too deeply integrated in cl-ppcre, I couldn't easily get them out into a single macro or something like that. Cyrus, Edi, could you help me clean up the changes so that they could be taken upstream? The other file is the one I'm using for testing. Regards, Phil

2 1

[cl-ppcre-devel] sbcl unhandled exception
by Philipp Marek 19 Apr '12

19 Apr '12

Hello everyone, I'm trying to teach cl-ppcre (taken from github, updated) to use (unsigned-byte 8) data as well, to see how much faster it gets for ASCII and binary files. Attached is a diff that's my current state. It's not nice, but - it's not cleaned up yet. The problem is that with sbcl 1.0.55.0-debian (amd64) I get Unhandled memory fault at #x0. and that's not good. I'm doing this: (asdf:clear-system :cl-ppcre) (asdf:oos 'asdf:compile-op :cl-ppcre :force T) (asdf:oos 'asdf:load-source-op :cl-ppcre) then, optionally, (trace "CL-PPCRE") and then crash SBCL with (scan-to-strings "B(A+)(.)" "AAAAAABAAAAAAAAAA") Any ideas? Thanks a lot. Regards, Phil

1 0

[cl-ppcre-devel] differing results between cl-ppcre and perl -- operator precedence?
by Cyrus Harmon 13 Apr '12

13 Apr '12

In trying to track down a problem matching a cxml-rng pattern in soiree, I came across the following somewhat odd behavior in cl-pppcre. In perl, I see: (sly@barbaresco):~/projects/soiree$ echo "Ta" | perl -ne 'print if /^T(a)|(b)$/' Ta (sly@barbaresco):~/projects/soiree$ echo "Tb" | perl -ne 'print if /^T(a)|(b)$/' Tb Whereas with cl-ppcre, I see: SOIREE-ICALENDAR> (cl-ppcre:scan-to-strings "^T(a)|(b)$" "Ta") "Ta" #("a" NIL) SOIREE-ICALENDAR> (cl-ppcre:scan-to-strings "^T(a)|(b)$" "Tb") "b" #(NIL "b") Parsing the string suggests that the order of operations may be different in cl-ppcre than in perl. Is this intentional? SOIREE-ICALENDAR> (cl-ppcre:parse-string "^T(a)|(b)$") (:ALTERNATION (:SEQUENCE :START-ANCHOR #\T (:REGISTER #\a)) (:SEQUENCE (:REGISTER #\b) :END-ANCHOR)) thanks, Cyrus

2 2

[cl-ppcre-devel] omitting initial void string in split returns
by Mario Maio 13 Apr '12

13 Apr '12

I want to parse strings of space-separated integers, and split is perfect except when I have vertical aligned integers. In such cases I've got one or more spaces at beginning of string, and the resulting list has a void string as 1st element: CL-USER> (cl-ppcre:split "\\s+" " 43 76 87 33 89 ") ("" "43" "76" "87" "33" "89") Is there an option to avoid such initial void string? Thanks. Mario

3 2

[cl-ppcre-devel] scan/property behavior?
by Cyrus Harmon 08 Apr '12

08 Apr '12

I was noticing some strange behavior with cxml-rng and David Lichteblau pointed out that the problem might be due to cl-ppcre and provided this example: (let ((regex1 '(:SEQUENCE :start-anchor (:greedy-repetition 0 42 (:property digit-char-p)) #\-)) (regex2 '(:SEQUENCE :start-anchor (:property digit-char-p) #\-)) (str "8-")) (values (multiple-value-list (ppcre:scan regex1 str)) (multiple-value-list (ppcre:scan regex2 str)))) Regex1 works as expected (it matches 8-) but the second one doesn't. It's possible I don't understand (well more than possible) how :property works, but I would have expected either both or neither of these to fail. Could this be due to a bug in cl-ppcre? thanks, Cyrus

3 3

[cl-ppcre-devel] possible bug with non-word boundaries and greedy matching
by Moshe Looks 28 Mar '12

28 Mar '12

I have an odd little regular expression \B.* that seems to generate inconsistent behavior between CL-PPCRE and other regular expression engines that I've tried it with. Using CL-PPCRE: CL-USER> (cl-ppcre::scan "\\B.*" "foo bar baz") NIL This matches in Perl, PPCRE, and RE2. Am I missing something, or should CL-PPCRE be matching this as well? Oddly enough, change * to + leads CL-PPCRE to match: CL-USER> (cl-ppcre::scan "\\B.+" "foo bar baz") 1 11 #() #() It seems like + matching implies that * should as well... Thanks, Moshe

1 0

[cl-ppcre-devel] behavior of \w
by Robert Brown 12 Mar '12

12 Mar '12

Some folks I work with are using cl-ppcre. They've run into an incompatibility between cl-ppcre and the PCRE library that boils down to cl-ppcre's handling of \w. The behavior is documented in cl-ppcre's manual: CL-PPCRE uses ALPHANUMERICP to decide whether a character matches Perl's "\w", so depending on your CL implementation you might encounter differences between Perl and CL-PPCRE when matching non-ASCII characters. This reliance on ALPHANUMERICP may be a misfeature. It means that cl-ppcre behaves differently depending on the Lisp implementation it's running on. My co-workers desire compatibility between cl-ppcre on SBCL (where ALPHANUMERICP follows Unicode) and PCRE for matching Latin-1 encoded strings. They patched the cl-ppcre code to make \w match a-z, A-Z, 0-9, and underscore. Is there a better workaround for them? bob

2 2