- cl-ppcre-devel@lists.common-lisp.net - mailman3.common-lisp.net

[cl-ppcre-devel] New release 0.9.1
by Edi Weitz 29 Nov '04

29 Nov '04

Changelog: Version 0.9.1 2004-11-29 Shortcuts for REGISTER-GROUPS-BIND and DO-REGISTER-GROUPS (suggested by Alexander Kjeldaas) Download: <http://weitz.de/files/cl-ppcre.tar.gz> Cheers, Edi.

1 0

[cl-ppcre-devel] Warning from CMUCL 19a about CL-PPCRE 0.9.0 usage
by Kick Damien-DKICK1 21 Nov '04

21 Nov '04

I'm seeing some warnings from CMUCL that I don't understand. Here is the warning the simplest code I've written at the moment that produces the warning: (defun flurp (line) (let (delta) (register-groups-bind (delta-str) ('(:sequence (:register (:greedy-repetition 1 nil :digit-class))) line) (setq delta (parse-integer delta-str))))) and the warning: ; In: DEFUN FLURP ; (REGISTER-GROUPS-BIND (DELTA-STR) ('# LINE) (SETQ DELTA #)) ; --> LET MULTIPLE-VALUE-BIND MULTIPLE-VALUE-CALL FUNCTION WHEN COND IF PROGN ; --> LET* LET ; ==> ; (IF #:START-INDEX5 ; (FUNCALL #:SUBSTR-FN6 #:TARGET-STRING0 #:START-INDEX5 #) ; NIL) ; Warning: This is not a (VALUES &OPTIONAL BASE-STRING &REST T): ; NIL ; ; [Last message occurs 2 times] What follows are a few more examples, with SSMT-OUTPUT-SUMMARY being the original snippet. (in-package #:pg-user) (defun ssmt-output-summary (pname) (with-open-file (in pname :direction :input) (loop with line = nil and delta = 0 and added = 0 and changed = 0 and deleted = 0 while (setq line (read-line in nil)) when (register-groups-bind (delta-str added-str changed-str deleted-str) ('(:sequence "__UNDEF__" (:greedy-repetition 1 nil :whitespace-char-class) ;; Delta SLOC (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Skip over delta AELOC (:greedy-repetition 1 nil :digit-class) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code added (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code changed (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code deleted (:register (:greedy-repetition 1 nil :digit-class))) line) ;; Somebody doesn't like our attempting to return ;; multiple values past the REGISTER-GROUPS-BIND (setq delta (parse-integer delta-str) added (parse-integer added-str) changed (parse-integer changed-str) deleted (parse-integer deleted-str))) do (return (values delta added changed deleted)) finally (warn "Could not find summary for ~A" pname)))) (defun foo (line) (let (delta added changed deleted) (register-groups-bind (delta-str added-str changed-str deleted-str) ('(:sequence "__UNDEF__" (:greedy-repetition 1 nil :whitespace-char-class) ;; Delta SLOC (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Skip over delta AELOC (:greedy-repetition 1 nil :digit-class) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code added (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code changed (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code deleted (:register (:greedy-repetition 1 nil :digit-class))) line) (setq delta (parse-integer delta-str) added (parse-integer added-str) changed (parse-integer changed-str) deleted (parse-integer deleted-str))))) (defun bar (line) (register-groups-bind (delta-str added-str changed-str deleted-str) ('(:sequence "__UNDEF__" (:greedy-repetition 1 nil :whitespace-char-class) ;; Delta SLOC (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Skip over delta AELOC (:greedy-repetition 1 nil :digit-class) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code added (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code changed (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code deleted (:register (:greedy-repetition 1 nil :digit-class))) line) (declare (ignore delta-str added-str changed-str deleted-str)))) (defun baz (line) (register-groups-bind (delta-str added-str changed-str deleted-str) ('(:sequence "__UNDEF__" (:greedy-repetition 1 nil :whitespace-char-class) ;; Delta SLOC (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Skip over delta AELOC (:greedy-repetition 1 nil :digit-class) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code added (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code changed (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code deleted (:register (:greedy-repetition 1 nil :digit-class))) line) (declare (ignore added-str changed-str deleted-str)) delta-str)) (defun blatz (line) (register-groups-bind (delta-str added-str changed-str deleted-str) ('(:sequence "__UNDEF__" (:greedy-repetition 1 nil :whitespace-char-class) ;; Delta SLOC (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Skip over delta AELOC (:greedy-repetition 1 nil :digit-class) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code added (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code changed (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code deleted (:register (:greedy-repetition 1 nil :digit-class))) line) (values delta-str added-str changed-str deleted-str))) (defun blurp (line) (register-groups-bind (delta-str added-str changed-str deleted-str) ('(:sequence "__UNDEF__" (:greedy-repetition 1 nil :whitespace-char-class) ;; Delta SLOC (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Skip over delta AELOC (:greedy-repetition 1 nil :digit-class) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code added (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code changed (:register (:greedy-repetition 1 nil :digit-class)) (:greedy-repetition 1 nil :whitespace-char-class) ;; Code deleted (:register (:greedy-repetition 1 nil :digit-class))) line) (declare (ignore delta-str added-str changed-str deleted-str)) 69)) (defun flurp (line) (let (delta) (register-groups-bind (delta-str) ('(:sequence (:register (:greedy-repetition 1 nil :digit-class))) line) (setq delta (parse-integer delta-str))))) And the complaints during compilation. * (lisp-implementation-version) "19a" * (lisp-implementation-type) "CMU Common Lisp" * (software-version) "5.8" * (software-type) "SunOS" * (compile-file "/home/kick/tmp/scratch.lisp") ; Python version 1.1, VM version UltraSparc/Solaris 7 on 19 NOV 04 01:59:21 pm. ; Compiling: /home/kick/tmp/scratch.lisp 19 NOV 04 01:58:23 pm ; Compiling Load Time Value of (CREATE-SCANNER '(:SEQUENCE "__UNDEF__" # ...)): ; Converted SSMT-OUTPUT-SUMMARY. ; Compiling DEFUN SSMT-OUTPUT-SUMMARY: ; ; ; File: /home/kick/tmp/scratch.lisp ; In: DEFUN SSMT-OUTPUT-SUMMARY ; (REGISTER-GROUPS-BIND (DELTA-STR ADDED-STR CHANGED-STR DELETED-STR) ; ('# LINE) ; (SETQ DELTA # ADDED # ...)) ; --> LET MULTIPLE-VALUE-BIND MULTIPLE-VALUE-CALL FUNCTION WHEN COND IF PROGN ; --> LET* LET ; ==> ; (IF #:START-INDEX15 ; (FUNCALL #:SUBSTR-FN16 #:TARGET-STRING10 #:START-INDEX15 #) ; NIL) ; Warning: This is not a (VALUES &OPTIONAL BASE-STRING &REST T): ; NIL ; ; [Last message occurs 8 times] ; Compiling Load Time Value of (CREATE-SCANNER '(:SEQUENCE "__UNDEF__" # ...)): ; Converted FOO. ; Compiling DEFUN FOO: ; ; ; File: /home/kick/tmp/scratch.lisp ; In: DEFUN FOO ; (REGISTER-GROUPS-BIND (DELTA-STR ADDED-STR CHANGED-STR DELETED-STR) ; ('# LINE) ; (SETQ DELTA # ADDED # ...)) ; --> LET MULTIPLE-VALUE-BIND MULTIPLE-VALUE-CALL FUNCTION WHEN COND IF PROGN ; --> LET* LET ; ==> ; (IF #:START-INDEX5 ; (FUNCALL #:SUBSTR-FN6 #:TARGET-STRING0 #:START-INDEX5 #) ; NIL) ; Warning: This is not a (VALUES &OPTIONAL BASE-STRING &REST T): ; NIL ; ; [Last message occurs 8 times] ; Compiling Load Time Value of (CREATE-SCANNER '(:SEQUENCE "__UNDEF__" # ...)): ; Converted BAR. ; Compiling DEFUN BAR: ; Compiling Load Time Value of (CREATE-SCANNER '(:SEQUENCE "__UNDEF__" # ...)): ; Converted BAZ. ; Compiling DEFUN BAZ: ; Compiling Load Time Value of (CREATE-SCANNER '(:SEQUENCE "__UNDEF__" # ...)): ; Converted BLATZ. ; Compiling DEFUN BLATZ: ; Compiling Load Time Value of (CREATE-SCANNER '(:SEQUENCE "__UNDEF__" # ...)): ; Converted BLURP. ; Compiling DEFUN BLURP: ; Compiling Load Time Value of (CREATE-SCANNER '(:SEQUENCE #)): ; Converted FLURP. ; Compiling DEFUN FLURP: ; ; ; File: /home/kick/tmp/scratch.lisp ; In: DEFUN FLURP ; (REGISTER-GROUPS-BIND (DELTA-STR) ('# LINE) (SETQ DELTA #)) ; --> LET MULTIPLE-VALUE-BIND MULTIPLE-VALUE-CALL FUNCTION WHEN COND IF PROGN ; --> LET* LET ; ==> ; (IF #:START-INDEX5 ; (FUNCALL #:SUBSTR-FN6 #:TARGET-STRING0 #:START-INDEX5 #) ; NIL) ; Warning: This is not a (VALUES &OPTIONAL BASE-STRING &REST T): ; NIL ; ; [Last message occurs 2 times] ; Byte Compiling Top-Level Form: ; Compilation unit finished. ; 18 warnings ; /home/kick/tmp/scratch.sparcf written. ; Compilation finished in 0:00:02. #p"/home/kick/tmp/scratch.sparcf" T T * I don't understand why only SSMT-OUTPUT-SUMMARY, FOO, and FLURP are the only ones with warnings. -- Damien Kick

2 1

[cl-ppcre-devel] Buffered multi-line question
by Sébastien Saint-Sevin 14 Oct '04

14 Oct '04

Hi Edi & list, I'm doing multi-lines regex searches over big files that can't be converted to single string. So I introduced a kind of buffer that I'm using to search. Now, I need to add a constraint to scan, do-scans & others (in addition to (&key start end)) : I want to be able to specify to the engine that a scan must start before a certain index in the string (to avoid searching further results that will be cancelled later because of my buffered multi-line matching process). Logically, this :must-start-before value correspond to the first line of my buffer. If nothing starts at first line, I need to move the search one line forward, so everything that the engine would match later on in the string is wasted time. How can I do it ? Cheers, Sebastien. PS: Edi, if you are back, my previous post is still an open question ;-) (the one with FILTER...)

2 16

[cl-ppcre-devel] New version 0.9.0
by Edi Weitz 14 Oct '04

14 Oct '04

Hi! A new release is available from <http://weitz.de/files/cl-ppcre.tgz>. Here's the relevant part from the changelog: Version 0.9.0 2004-10-14 Experimental support for "filters" Bugfix for standalone regular expressions (ACCUMULATE-START-P wasn't set to NIL) Have fun, Edi.

1 0

RE: [cl-ppcre-devel] Re: Cl-ppcre usage
by Sébastien Saint-Sevin 01 Oct '04

01 Oct '04

Hi Edi & list, two points : 1) virtuals bugs 2) user hook 1) I feel really stupid : the two behaviours I mentionned in my previous mails were already corrected in recents releases. I was using 0.1.3 !!! The reason : the library was so good from its first releases that I simply forgot to upgrade ;-) 2) More interesting : I tried a few things to simply implement what I need, ie simple lookup function to validate a match. Here is what I came to: I added a (:FILTER <number> <predicate>) parse-tree S-expr. It is similar to a (:BACK-REFERENCE <number>) in that it refers to the :REGISTER defined by <number>. It is different in that it does not consume any char in the regex string, but submit the register substring to its predicate to validate or not the current register match. So to achieve what I wanted, I just have to encapsulate the parsed tree with (:SEQUENCE (:REGISTER <parsed-tree>) (:FILTER 1 MY-PREDICATE-SYMBOL)) If I still want to use (parse-string <string>) to build the main part of the tree, I just need to incf all registers within the parsed-tree by one to take into account the register I finally add to the tree. What do you think about all that ? Cheers, Sebastien. > -----Message d'origine----- > De : cl-ppcre-devel-bounces(a)common-lisp.net > [mailto:cl-ppcre-devel-bounces@common-lisp.net]De la part de Edi Weitz > Envoyé : mercredi 29 septembre 2004 01:19 > À : Sébastien Saint-Sevin > Cc : cl-ppcre-devel(a)common-lisp.net > Objet : [cl-ppcre-devel] Re: Cl-ppcre usage > > > Hi Sébastien! > > On Tue, 28 Sep 2004 16:12:21 +0200, Sébastien Saint-Sevin > <seb(a)matchix.com> wrote: > > > I've got two questions regarding cl-ppcre usage. > > > > 1) do-scans > > I need to use multi-matching within a string so I use do-scans. > > However, I've got some troubles when I use ^ to force a match at the > > begining of the string. It's like the start of the string is > > constantly reinit after each match within the string (and this is > > not what I want as you already suppose). > > > > Is there a way to do it? > > I'm not really sure what you want to achieve - maybe an example would > help. If your search pattern contains ^ then it can only match once, > right? Have you looked at the m modifier? > > > 2) user hook > > I want to combine regex and some kind of lookup tables (hash-table > > that stores string). In Michael parker's regex engine, it is > > possible to give the engine a user hook function that, when a match > > is found, check it against this user function to finally validate > > it, otherwise backtrack the matching. > > > > How can I do that with CL-PPCRE ? (where do I need to hack the code > > otherwise ? ;-)) > > I have plans to implement this in the future but my current workload > and other projects don't allow me to do that. If you provide a clean > patch which implements this functionality I'd be glad to integrate > it. (In my opinion this should only be possible in parse trees, > though, in order not to break Perl compatibility.) > > Cheers, > Edi. > > _______________________________________________ > cl-ppcre-devel site list > cl-ppcre-devel(a)common-lisp.net > http://common-lisp.net/mailman/listinfo/cl-ppcre-devel >

2 1

[cl-ppcre-devel] New version 0.8.1
by Edi Weitz 30 Sep '04

30 Sep '04

Hi! A new release is available from <http://weitz.de/files/cl-ppcre.tgz>. Here's the relevant part from the changelog: Version 0.8.1 2004-09-30 Patches for Genera 8.5 (thanks to Patrick O'Donnell) Have fun, Edi.

1 0

[cl-ppcre-devel] Porting to Genera, feature request
by Patrick O'Donnell 29 Sep '04

29 Sep '04

Hi, Edi, I just downloaded cl-ppcre to see if it would help with some parsing I'm working on. I thought I'd drop you a line just to alert you to the issues I ran into. I don't think they're worth a lot of attention, but just so you know they're there. I'm doing development on Genera, and there were several issues with compiling cl-ppcre, mostly due to Genera not quite being ANSI compliant. I finally got it to compile and ran the test. Some (21) of the tests failed. Most of the failures, I've noted are based on assumptions about character codes that do not hold for Genera (e.g. test 432 -- line termination (char-code #\Return) -> #o215, not #o12.) The others I haven't thoroughly investigated. I believe that I'll be able to use the limited features I need, and if I run into trouble, I'll let you know. The feature that I'd like to see relates to your s-expression parse tree capability. I'd like to abstract sub-parse-trees. A small change to the end of convert-aux, (otherwise (let ((translation (get parse-tree 'parse-tree-synonym))) (if translation (convert-aux translation) (signal-ppcre-syntax-error "Unknown token ~A in parse-tree" parse-tree)))) and a quick macro, (defmacro DEFINE-PARSE-TREE-SYNONYM (name parse-tree) `(setf (get ',name 'ppcre::parse-tree-synonym) ',parse-tree)) and I can do something like: (define-parse-tree-synonym A (:CHAR-CLASS (:RANGE #\a #\z) (:RANGE #\A #\Z))) (define-parse-tree-synonym X (:CHAR-CLASS (:RANGE #\a #\z) (:RANGE #\A #\Z) :DIGIT-CLASS)) (define-parse-tree-synonym N :DIGIT-CLASS) (define-parse-tree-synonym SMA-DATE (:SEQUENCE n n a a a)) (define-parse-tree-synonym AIRLINE-DESIGNATOR (:SEQUENCE x x (:GREEDY-REPETITION 0 1 a))) (define-parse-tree-synonym FLIGHT-NUMBER (:GREEDY-REPETITION 3 4 n)) (define-parse-tree-synonym OPERATIONAL-SUFFIX (:GREEDY-REPETITION 0 1 a)) (defparameter *flight-scanner* (ppcre:create-scanner '(:sequence airline-designator flight-number operational-suffix "/" sma-date))) Much more perspicuous, especially in more complex parse trees where the abstracted elements are repeated. Just a thought. - Patrick O'Donnell pao(a)ascent.com

2 7

[cl-ppcre-devel] Re: Cl-ppcre usage
by Edi Weitz 28 Sep '04

28 Sep '04

Hi Sébastien! On Tue, 28 Sep 2004 16:12:21 +0200, Sébastien Saint-Sevin <seb(a)matchix.com> wrote: > I've got two questions regarding cl-ppcre usage. > > 1) do-scans > I need to use multi-matching within a string so I use do-scans. > However, I've got some troubles when I use ^ to force a match at the > begining of the string. It's like the start of the string is > constantly reinit after each match within the string (and this is > not what I want as you already suppose). > > Is there a way to do it? I'm not really sure what you want to achieve - maybe an example would help. If your search pattern contains ^ then it can only match once, right? Have you looked at the m modifier? > 2) user hook > I want to combine regex and some kind of lookup tables (hash-table > that stores string). In Michael parker's regex engine, it is > possible to give the engine a user hook function that, when a match > is found, check it against this user function to finally validate > it, otherwise backtrack the matching. > > How can I do that with CL-PPCRE ? (where do I need to hack the code > otherwise ? ;-)) I have plans to implement this in the future but my current workload and other projects don't allow me to do that. If you provide a clean patch which implements this functionality I'd be glad to integrate it. (In my opinion this should only be possible in parse trees, though, in order not to break Perl compatibility.) Cheers, Edi.

1 0

[cl-ppcre-devel] Re: CL-PPCRE bug found in do-scans
by Edi Weitz 28 Sep '04

28 Sep '04

Hi Sébastien! On Wed, 29 Sep 2004 00:23:50 +0200, Sébastien Saint-Sevin <seb(a)matchix.com> wrote: > I think I found a bug when using do-scans : > > CL-USER 30 > (setq *scanner* (cl-ppcre:create-scanner "[0-9]-$")) > #<closure 2060674A> > > CL-USER 31 > (cl-ppcre:do-scans (s e rs re *scanner* "simple string-")) > NIL > > CL-USER 32 > (cl-ppcre:do-scans (s e rs re *scanner* "simple string 4-")) > ==> infinite loop. 100% cpu. memory growing indefinitely. > > The same regex with a simple scan is OK : > > CL-USER 34 > (cl-ppcre:scan *scanner* "simple string 4-") > 14 > 16 > #() > #() Which version of CL-PPCRE (current is 0.8.0) and which Lisp is this? I can't reproduce this problem with CMUCL 19a or AllegroCL 7.0 beta. Please use the mailing list for bug reports, thanks. Cheers, Edi.

1 0

[cl-ppcre-devel] New version 0.8.0
by Edi Weitz 16 Sep '04

16 Sep '04

Hi! A new release is available from <http://weitz.de/files/cl-ppcre.tgz>. Here's the relevant part from the changelog: Version 0.8.0 2004-09-16 Added parse tree synonyms (thanks to Patrick O'Donnell) Have fun, Edi.

1 0