The tokenize module provides a lexical scanner for Python
source code, implemented in Python. The scanner in this module
returns comments as tokens as well, making it useful for implementing
``pretty-printers,'' including colorizers for on-screen displays.
The tokenize() function accepts two parameters: one
representing the input stream, and one providing an output mechanism
for tokenize().
The first parameter, readline, must be a callable object which
provides the same interface as the readline() method of
built-in file objects (see section 2.1.7). Each
call to the function should return one line of input as a string.
The second parameter, tokeneater, must also be a callable
object. It is called with five parameters: the token type, the
token string, a tuple (srow, scol) specifying the
row and column where the token begins in the source, a tuple
(erow, ecol) giving the ending position of the
token, and the line on which the token was found. The line passed
is the logical line; continuation lines are included.
All constants from the token module are also exported from
tokenize, as are two additional token type values that might be
passed to the tokeneater function by tokenize():
Token value used to indicate a non-terminating newline. The NEWLINE
token indicates the end of a logical line of Python code; NL tokens
are generated when a logical line of code is continued over multiple
physical lines.