GNU Info

(slib.info)Token definition

---

Next: Nud and Led Definition Prev: Ruleset Definition and Use Up: Precedence Parsing

---

Enter node , (file) or (file)node

Token definition
----------------

 - Function: tok:char-group group chars chars-proc
     The argument CHARS may be a single character, a list of
     characters, or a string.  Each character in CHARS is treated as
     though `tok:char-group' was called with that character alone.

     The argument CHARS-PROC must be a procedure of one argument, a
     list of characters.  After `tokenize' has finished accumulating
     the characters for a token, it calls CHARS-PROC with the list of
     characters.  The value returned is the token which `tokenize'
     returns.

     The argument GROUP may be an exact integer or a procedure of one
     character argument.  The following discussion concerns the
     treatment which the tokenizing routine, `tokenize', will accord to
     characters on the basis of their groups.

     When GROUP is a non-zero integer, characters whose group number is
     equal to or exactly one less than GROUP will continue to
     accumulate.  Any other character causes the accumulation to stop
     (until a new token is to be read).

     The GROUP of zero is special.  These characters are ignored when
     parsed pending a token, and stop the accumulation of token
     characters when the accumulation has already begun.  Whitespace
     characters are usually put in group 0.

     If GROUP is a procedure, then, when triggerd by the occurence of
     an initial (no accumulation) CHARS character, this procedure will
     be repeatedly called with each successive character from the input
     stream until the GROUP procedure returns a non-false value.

The following convenient constants are provided for use with
`tok:char-group'.

 - Constant: tok:decimal-digits
     Is the string `"0123456789"'.

 - Constant: tok:upper-case
     Is the string consisting of all upper-case letters
     ("ABCDEFGHIJKLMNOPQRSTUVWXYZ").

 - Constant: tok:lower-case
     Is the string consisting of all lower-case letters
     ("abcdefghijklmnopqrstuvwxyz").

 - Constant: tok:whitespaces
     Is the string consisting of all characters between 0 and 255 for
     which `char-whitespace?' returns true.