-.I flex
-is a tool for generating
-.I scanners:
-programs which recognized lexical patterns in text.
-.I flex
-reads
-the given input files, or its standard input if no file names are given,
-for a description of a scanner to generate. The description is in
-the form of pairs
-of regular expressions and C code, called
-.I rules. flex
-generates as output a C source file,
-.B lex.yy.c,
-which defines a routine
-.B yylex().
-This file is compiled and linked with the
-.B \-lfl
-library to produce an executable. When the executable is run,
-it analyzes its input for occurrences
-of the regular expressions. Whenever it finds one, it executes
-the corresponding C code.
-.SH SOME SIMPLE EXAMPLES
-.PP
-First some simple examples to get the flavor of how one uses
-.I flex.
-The following
-.I flex
-input specifies a scanner which whenever it encounters the string
-"username" will replace it with the user's login name:
-.nf
-
- %%
- username printf( "%s", getlogin() );
-
-.fi
-By default, any text not matched by a
-.I flex
-scanner
-is copied to the output, so the net effect of this scanner is
-to copy its input file to its output with each occurrence
-of "username" expanded.
-In this input, there is just one rule. "username" is the
-.I pattern
-and the "printf" is the
-.I action.
-The "%%" marks the beginning of the rules.
-.PP
-Here's another simple example:
-.nf
-
- int num_lines = 0, num_chars = 0;
-
- %%
- \\n ++num_lines; ++num_chars;
- . ++num_chars;
-
- %%
- main()
- {
- yylex();
- printf( "# of lines = %d, # of chars = %d\\n",
- num_lines, num_chars );
- }
-
-.fi
-This scanner counts the number of characters and the number
-of lines in its input (it produces no output other than the
-final report on the counts). The first line
-declares two globals, "num_lines" and "num_chars", which are accessible
-both inside
-.B yylex()
-and in the
-.B main()
-routine declared after the second "%%". There are two rules, one
-which matches a newline ("\\n") and increments both the line count and
-the character count, and one which matches any character other than
-a newline (indicated by the "." regular expression).
-.PP
-A somewhat more complicated example:
-.nf
-
- /* scanner for a toy Pascal-like language */
-
- %{
- /* need this for the call to atof() below */
- #include <math.h>
- %}
-
- DIGIT [0-9]
- ID [a-z][a-z0-9]*
-
- %%
-
- {DIGIT}+ {
- printf( "An integer: %s (%d)\\n", yytext,
- atoi( yytext ) );
- }
-
- {DIGIT}+"."{DIGIT}* {
- printf( "A float: %s (%g)\\n", yytext,
- atof( yytext ) );
- }
-
- if|then|begin|end|procedure|function {
- printf( "A keyword: %s\\n", yytext );
- }
-
- {ID} printf( "An identifier: %s\\n", yytext );
-
- "+"|"-"|"*"|"/" printf( "An operator: %s\\n", yytext );
-
- "{"[^}\\n]*"}" /* eat up one-line comments */
-
- [ \\t\\n]+ /* eat up whitespace */
-
- . printf( "Unrecognized character: %s\\n", yytext );
-
- %%
-
- main( argc, argv )
- int argc;
- char **argv;
- {
- ++argv, --argc; /* skip over program name */
- if ( argc > 0 )
- yyin = fopen( argv[0], "r" );
- else
- yyin = stdin;
-
- yylex();
- }
-
-.fi
-This is the beginnings of a simple scanner for a language like
-Pascal. It identifies different types of
-.I tokens
-and reports on what it has seen.
-.PP
-The details of this example will be explained in the following
-sections.
-.SH FORMAT OF THE INPUT FILE
-The
-.I flex
-input file consists of three sections, separated by a line with just
-.B %%
-in it:
-.nf
-
- definitions
- %%
- rules
- %%
- user code
-
-.fi
-The
-.I definitions
-section contains declarations of simple
-.I name
-definitions to simplify the scanner specification, and declarations of
-.I start conditions,
-which are explained in a later section.
-.PP
-Name definitions have the form:
-.nf
-
- name definition
-
-.fi
-The "name" is a word beginning with a letter or an underscore ('_')
-followed by zero or more letters, digits, '_', or '-' (dash).
-The definition is taken to begin at the first non-white-space character
-following the name and continuing to the end of the line.
-The definition can subsequently be referred to using "{name}", which
-will expand to "(definition)". For example,
-.nf
-
- DIGIT [0-9]
- ID [a-z][a-z0-9]*
-
-.fi
-defines "DIGIT" to be a regular expression which matches a
-single digit, and
-"ID" to be a regular expression which matches a letter
-followed by zero-or-more letters-or-digits.
-A subsequent reference to
-.nf
-
- {DIGIT}+"."{DIGIT}*
-
-.fi
-is identical to
-.nf
-
- ([0-9])+"."([0-9])*
-
-.fi
-and matches one-or-more digits followed by a '.' followed
-by zero-or-more digits.
-.PP
-The
-.I rules
-section of the
-.I flex
-input contains a series of rules of the form:
-.nf
-
- pattern action
-
-.fi
-where the pattern must be unindented and the action must begin
-on the same line.
-.PP
-See below for a further description of patterns and actions.
-.PP
-Finally, the user code section is simply copied to
-.B lex.yy.c
-verbatim.
-It is used for companion routines which call or are called
-by the scanner. The presence of this section is optional;
-if it is missing, the second
-.B %%
-in the input file may be skipped, too.
-.PP
-In the definitions and rules sections, any
-.I indented
-text or text enclosed in
-.B %{
-and
-.B %}
-is copied verbatim to the output (with the %{}'s removed).
-The %{}'s must appear unindented on lines by themselves.
-.PP
-In the rules section,
-any indented or %{} text appearing before the
-first rule may be used to declare variables
-which are local to the scanning routine and (after the declarations)
-code which is to be executed whenever the scanning routine is entered.
-Other indented or %{} text in the rule section is still copied to the output,
-but its meaning is not well-defined and it may well cause compile-time
-errors (this feature is present for
-.I POSIX
-compliance; see below for other such features).
-.PP
-In the definitions section (but not in the rules section),
-an unindented comment (i.e., a line
-beginning with "/*") is also copied verbatim to the output up
-to the next "*/".
-.SH PATTERNS
-The patterns in the input are written using an extended set of regular
-expressions. These are:
-.nf
-
- x match the character 'x'
- . any character (byte) except newline
- [xyz] a "character class"; in this case, the pattern
- matches either an 'x', a 'y', or a 'z'
- [abj-oZ] a "character class" with a range in it; matches
- an 'a', a 'b', any letter from 'j' through 'o',
- or a 'Z'
- [^A-Z] a "negated character class", i.e., any character
- but those in the class. In this case, any
- character EXCEPT an uppercase letter.
- [^A-Z\\n] any character EXCEPT an uppercase letter or
- a newline
- r* zero or more r's, where r is any regular expression
- r+ one or more r's
- r? zero or one r's (that is, "an optional r")
- r{2,5} anywhere from two to five r's
- r{2,} two or more r's
- r{4} exactly 4 r's
- {name} the expansion of the "name" definition
- (see above)
- "[xyz]\\"foo"
- the literal string: [xyz]"foo
- \\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
- then the ANSI-C interpretation of \\x.
- Otherwise, a literal 'X' (used to escape
- operators such as '*')
- \\0 a NUL character (ASCII code 0)
- \\123 the character with octal value 123
- \\x2a the character with hexadecimal value 2a
- (r) match an r; parentheses are used to override
- precedence (see below)
-
-
- rs the regular expression r followed by the
- regular expression s; called "concatenation"
-
-
- r|s either an r or an s
-
-
- r/s an r but only if it is followed by an s. The
- text matched by s is included when determining
- whether this rule is the "longest match",
- but is then returned to the input before
- the action is executed. So the action only
- sees the text matched by r. This type
- of pattern is called trailing context".
- (There are some combinations of r/s that flex
- cannot match correctly; see notes in the
- Deficiencies / Bugs section below regarding
- "dangerous trailing context".)
- ^r an r, but only at the beginning of a line (i.e.,
- which just starting to scan, or right after a
- newline has been scanned).
- r$ an r, but only at the end of a line (i.e., just
- before a newline). Equivalent to "r/\\n".
-
- Note that flex's notion of "newline" is exactly
- whatever the C compiler used to compile flex
- interprets '\\n' as; in particular, on some DOS
- systems you must either filter out \\r's in the
- input yourself, or explicitly use r/\\r\\n for "r$".
-
-
- <s>r an r, but only in start condition s (see
- below for discussion of start conditions)
- <s1,s2,s3>r
- same, but in any of start conditions s1,
- s2, or s3
- <*>r an r in any start condition, even an exclusive one.
-
-
- <<EOF>> an end-of-file
- <s1,s2><<EOF>>
- an end-of-file when in start condition s1 or s2
-
-.fi
-Note that inside of a character class, all regular expression operators
-lose their special meaning except escape ('\\') and the character class
-operators, '-', ']', and, at the beginning of the class, '^'.
-.PP
-The regular expressions listed above are grouped according to
-precedence, from highest precedence at the top to lowest at the bottom.
-Those grouped together have equal precedence. For example,
-.nf
-
- foo|bar*
-
-.fi
-is the same as
-.nf
-
- (foo)|(ba(r*))
-
-.fi
-since the '*' operator has higher precedence than concatenation,
-and concatenation higher than alternation ('|'). This pattern
-therefore matches
-.I either
-the string "foo"
-.I or
-the string "ba" followed by zero-or-more r's.
-To match "foo" or zero-or-more "bar"'s, use:
-.nf
-
- foo|(bar)*
-
-.fi
-and to match zero-or-more "foo"'s-or-"bar"'s:
-.nf
-
- (foo|bar)*
-
-.fi
-.PP
-In addition to characters and ranges of characters, character classes
-can also contain character class
-.I expressions.
-These are expressions enclosed inside
-.B [:
-and
-.B :]
-delimiters (which themselves must appear between the '[' and ']' of the
-character class; other elements may occur inside the character class, too).
-The valid expressions are:
-.nf
-
- [:alnum:] [:alpha:] [:blank:]
- [:cntrl:] [:digit:] [:graph:]
- [:lower:] [:print:] [:punct:]
- [:space:] [:upper:] [:xdigit:]
-
-.fi
-These expressions all designate a set of characters equivalent to
-the corresponding standard C
-.B isXXX
-function. For example,
-.B [:alnum:]
-designates those characters for which
-.B isalnum()
-returns true - i.e., any alphabetic or numeric.
-Some systems don't provide
-.B isblank(),
-so flex defines
-.B [:blank:]
-as a blank or a tab.
-.PP
-For example, the following character classes are all equivalent:
-.nf
-
- [[:alnum:]]
- [[:alpha:][:digit:]
- [[:alpha:]0-9]
- [a-zA-Z0-9]
-
-.fi
-If your scanner is case-insensitive (the
-.B \-i
-flag), then
-.B [:upper:]
-and
-.B [:lower:]
-are equivalent to
-.B [:alpha:].
-.PP
-Some notes on patterns:
-.IP -
-A negated character class such as the example "[^A-Z]"
-above
-.I will match a newline
-unless "\\n" (or an equivalent escape sequence) is one of the
-characters explicitly present in the negated character class
-(e.g., "[^A-Z\\n]"). This is unlike how many other regular
-expression tools treat negated character classes, but unfortunately
-the inconsistency is historically entrenched.
-Matching newlines means that a pattern like [^"]* can match the entire
-input unless there's another quote in the input.
-.IP -
-A rule can have at most one instance of trailing context (the '/' operator
-or the '$' operator). The start condition, '^', and "<<EOF>>" patterns
-can only occur at the beginning of a pattern, and, as well as with '/' and '$',
-cannot be grouped inside parentheses. A '^' which does not occur at
-the beginning of a rule or a '$' which does not occur at the end of
-a rule loses its special properties and is treated as a normal character.
-.IP
-The following are illegal:
-.nf
-
- foo/bar$
- <sc1>foo<sc2>bar
-
-.fi
-Note that the first of these, can be written "foo/bar\\n".
-.IP
-The following will result in '$' or '^' being treated as a normal character:
-.nf
-
- foo|(bar$)
- foo|^bar
-
-.fi
-If what's wanted is a "foo" or a bar-followed-by-a-newline, the following
-could be used (the special '|' action is explained below):
-.nf
-
- foo |
- bar$ /* action goes here */
-
-.fi
-A similar trick will work for matching a foo or a
-bar-at-the-beginning-of-a-line.
-.SH HOW THE INPUT IS MATCHED
-When the generated scanner is run, it analyzes its input looking
-for strings which match any of its patterns. If it finds more than
-one match, it takes the one matching the most text (for trailing
-context rules, this includes the length of the trailing part, even
-though it will then be returned to the input). If it finds two
-or more matches of the same length, the
-rule listed first in the
-.I flex
-input file is chosen.
-.PP
-Once the match is determined, the text corresponding to the match
-(called the
-.I token)
-is made available in the global character pointer
-.B yytext,
-and its length in the global integer
-.B yyleng.
-The
-.I action
-corresponding to the matched pattern is then executed (a more
-detailed description of actions follows), and then the remaining
-input is scanned for another match.
-.PP
-If no match is found, then the
-.I default rule
-is executed: the next character in the input is considered matched and
-copied to the standard output. Thus, the simplest legal
-.I flex
-input is:
-.nf
-
- %%
-
-.fi
-which generates a scanner that simply copies its input (one character
-at a time) to its output.
-.PP
-Note that
-.B yytext
-can be defined in two different ways: either as a character
-.I pointer
-or as a character
-.I array.
-You can control which definition
-.I flex
-uses by including one of the special directives
-.B %pointer
-or
-.B %array
-in the first (definitions) section of your flex input. The default is
-.B %pointer,
-unless you use the
-.B -l
-lex compatibility option, in which case
-.B yytext
-will be an array.
-The advantage of using
-.B %pointer
-is substantially faster scanning and no buffer overflow when matching
-very large tokens (unless you run out of dynamic memory). The disadvantage
-is that you are restricted in how your actions can modify
-.B yytext
-(see the next section), and calls to the
-.B unput()
-function destroys the present contents of
-.B yytext,
-which can be a considerable porting headache when moving between different
-.I lex
-versions.
-.PP
-The advantage of
-.B %array
-is that you can then modify
-.B yytext
-to your heart's content, and calls to
-.B unput()
-do not destroy
-.B yytext
-(see below). Furthermore, existing
-.I lex
-programs sometimes access
-.B yytext
-externally using declarations of the form:
-.nf
- extern char yytext[];
-.fi
-This definition is erroneous when used with
-.B %pointer,
-but correct for
-.B %array.
-.PP
-.B %array
-defines
-.B yytext
-to be an array of
-.B YYLMAX
-characters, which defaults to a fairly large value. You can change
-the size by simply #define'ing
-.B YYLMAX
-to a different value in the first section of your
-.I flex
-input. As mentioned above, with
-.B %pointer
-yytext grows dynamically to accommodate large tokens. While this means your
-.B %pointer
-scanner can accommodate very large tokens (such as matching entire blocks
-of comments), bear in mind that each time the scanner must resize
-.B yytext
-it also must rescan the entire token from the beginning, so matching such
-tokens can prove slow.
-.B yytext
-presently does
-.I not
-dynamically grow if a call to
-.B unput()
-results in too much text being pushed back; instead, a run-time error results.
-.PP
-Also note that you cannot use
-.B %array
-with C++ scanner classes
-(the
-.B c++
-option; see below).
-.SH ACTIONS
-Each pattern in a rule has a corresponding action, which can be any
-arbitrary C statement. The pattern ends at the first non-escaped
-whitespace character; the remainder of the line is its action. If the
-action is empty, then when the pattern is matched the input token
-is simply discarded. For example, here is the specification for a program
-which deletes all occurrences of "zap me" from its input:
-.nf
-
- %%
- "zap me"
-
-.fi
-(It will copy all other characters in the input to the output since
-they will be matched by the default rule.)
-.PP
-Here is a program which compresses multiple blanks and tabs down to
-a single blank, and throws away whitespace found at the end of a line:
-.nf
-
- %%
- [ \\t]+ putchar( ' ' );
- [ \\t]+$ /* ignore this token */
-
-.fi
-.PP
-If the action contains a '{', then the action spans till the balancing '}'
-is found, and the action may cross multiple lines.
-.I flex
-knows about C strings and comments and won't be fooled by braces found
-within them, but also allows actions to begin with
-.B %{
-and will consider the action to be all the text up to the next
-.B %}
-(regardless of ordinary braces inside the action).
-.PP
-An action consisting solely of a vertical bar ('|') means "same as
-the action for the next rule." See below for an illustration.
-.PP
-Actions can include arbitrary C code, including
-.B return
-statements to return a value to whatever routine called
-.B yylex().
-Each time
-.B yylex()
-is called it continues processing tokens from where it last left
-off until it either reaches
-the end of the file or executes a return.
-.PP
-Actions are free to modify
-.B yytext
-except for lengthening it (adding
-characters to its end--these will overwrite later characters in the
-input stream). This however does not apply when using
-.B %array
-(see above); in that case,
-.B yytext
-may be freely modified in any way.
-.PP
-Actions are free to modify
-.B yyleng
-except they should not do so if the action also includes use of
-.B yymore()
-(see below).
-.PP
-There are a number of special directives which can be included within
-an action:
-.IP -
-.B ECHO
-copies yytext to the scanner's output.
-.IP -
-.B BEGIN
-followed by the name of a start condition places the scanner in the
-corresponding start condition (see below).
-.IP -
-.B REJECT
-directs the scanner to proceed on to the "second best" rule which matched the
-input (or a prefix of the input). The rule is chosen as described
-above in "How the Input is Matched", and
-.B yytext
-and
-.B yyleng
-set up appropriately.
-It may either be one which matched as much text
-as the originally chosen rule but came later in the
-.I flex
-input file, or one which matched less text.
-For example, the following will both count the
-words in the input and call the routine special() whenever "frob" is seen:
-.nf
-
- int word_count = 0;
- %%
-
- frob special(); REJECT;
- [^ \\t\\n]+ ++word_count;
-
-.fi
-Without the
-.B REJECT,
-any "frob"'s in the input would not be counted as words, since the
-scanner normally executes only one action per token.
-Multiple
-.B REJECT's
-are allowed, each one finding the next best choice to the currently
-active rule. For example, when the following scanner scans the token
-"abcd", it will write "abcdabcaba" to the output:
-.nf
-
- %%
- a |
- ab |
- abc |
- abcd ECHO; REJECT;
- .|\\n /* eat up any unmatched character */
-
-.fi
-(The first three rules share the fourth's action since they use
-the special '|' action.)
-.B REJECT
-is a particularly expensive feature in terms of scanner performance;
-if it is used in
-.I any
-of the scanner's actions it will slow down
-.I all
-of the scanner's matching. Furthermore,
-.B REJECT
-cannot be used with the
-.I -Cf
-or
-.I -CF
-options (see below).
-.IP
-Note also that unlike the other special actions,
-.B REJECT
-is a
-.I branch;
-code immediately following it in the action will
-.I not
-be executed.
-.IP -
-.B yymore()
-tells the scanner that the next time it matches a rule, the corresponding
-token should be
-.I appended
-onto the current value of
-.B yytext
-rather than replacing it. For example, given the input "mega-kludge"
-the following will write "mega-mega-kludge" to the output:
-.nf
-
- %%
- mega- ECHO; yymore();
- kludge ECHO;
-
-.fi
-First "mega-" is matched and echoed to the output. Then "kludge"
-is matched, but the previous "mega-" is still hanging around at the
-beginning of
-.B yytext
-so the
-.B ECHO
-for the "kludge" rule will actually write "mega-kludge".
-.PP
-Two notes regarding use of
-.B yymore().
-First,
-.B yymore()
-depends on the value of
-.I yyleng
-correctly reflecting the size of the current token, so you must not
-modify
-.I yyleng
-if you are using
-.B yymore().
-Second, the presence of
-.B yymore()
-in the scanner's action entails a minor performance penalty in the
-scanner's matching speed.
-.IP -
-.B yyless(n)
-returns all but the first
-.I n
-characters of the current token back to the input stream, where they
-will be rescanned when the scanner looks for the next match.
-.B yytext
-and
-.B yyleng
-are adjusted appropriately (e.g.,
-.B yyleng
-will now be equal to
-.I n
-). For example, on the input "foobar" the following will write out
-"foobarbar":
-.nf
-
- %%
- foobar ECHO; yyless(3);
- [a-z]+ ECHO;
-
-.fi
-An argument of 0 to
-.B yyless
-will cause the entire current input string to be scanned again. Unless you've
-changed how the scanner will subsequently process its input (using
-.B BEGIN,
-for example), this will result in an endless loop.
-.PP
-Note that
-.B yyless
-is a macro and can only be used in the flex input file, not from
-other source files.
-.IP -
-.B unput(c)
-puts the character
-.I c
-back onto the input stream. It will be the next character scanned.
-The following action will take the current token and cause it
-to be rescanned enclosed in parentheses.
-.nf
-
- {
- int i;
- /* Copy yytext because unput() trashes yytext */
- char *yycopy = strdup( yytext );
- unput( ')' );
- for ( i = yyleng - 1; i >= 0; --i )
- unput( yycopy[i] );
- unput( '(' );
- free( yycopy );
- }
-
-.fi
-Note that since each
-.B unput()
-puts the given character back at the
-.I beginning
-of the input stream, pushing back strings must be done back-to-front.
-.PP
-An important potential problem when using
-.B unput()
-is that if you are using
-.B %pointer
-(the default), a call to
-.B unput()
-.I destroys
-the contents of
-.I yytext,
-starting with its rightmost character and devouring one character to
-the left with each call. If you need the value of yytext preserved
-after a call to
-.B unput()
-(as in the above example),
-you must either first copy it elsewhere, or build your scanner using
-.B %array
-instead (see How The Input Is Matched).
-.PP
-Finally, note that you cannot put back
-.B EOF
-to attempt to mark the input stream with an end-of-file.
-.IP -
-.B input()
-reads the next character from the input stream. For example,
-the following is one way to eat up C comments:
-.nf
-
- %%
- "/*" {
- register int c;
-
- for ( ; ; )
- {
- while ( (c = input()) != '*' &&
- c != EOF )
- ; /* eat up text of comment */
-
- if ( c == '*' )
- {
- while ( (c = input()) == '*' )
- ;
- if ( c == '/' )
- break; /* found the end */
- }
-
- if ( c == EOF )
- {
- error( "EOF in comment" );
- break;
- }
- }
- }
-
-.fi
-(Note that if the scanner is compiled using
-.B C++,
-then
-.B input()
-is instead referred to as
-.B yyinput(),
-in order to avoid a name clash with the
-.B C++
-stream by the name of
-.I input.)
-.IP -
-.B YY_FLUSH_BUFFER
-flushes the scanner's internal buffer
-so that the next time the scanner attempts to match a token, it will
-first refill the buffer using
-.B YY_INPUT
-(see The Generated Scanner, below). This action is a special case
-of the more general
-.B yy_flush_buffer()
-function, described below in the section Multiple Input Buffers.
-.IP -
-.B yyterminate()
-can be used in lieu of a return statement in an action. It terminates
-the scanner and returns a 0 to the scanner's caller, indicating "all done".
-By default,
-.B yyterminate()
-is also called when an end-of-file is encountered. It is a macro and
-may be redefined.
-.SH THE GENERATED SCANNER
-The output of
-.I flex
-is the file
-.B lex.yy.c,
-which contains the scanning routine
-.B yylex(),
-a number of tables used by it for matching tokens, and a number
-of auxiliary routines and macros. By default,
-.B yylex()
-is declared as follows:
-.nf
-
- int yylex()
- {
- ... various definitions and the actions in here ...
- }
-
-.fi
-(If your environment supports function prototypes, then it will
-be "int yylex( void )".) This definition may be changed by defining
-the "YY_DECL" macro. For example, you could use:
-.nf
-
- #define YY_DECL float lexscan( a, b ) float a, b;
-
-.fi
-to give the scanning routine the name
-.I lexscan,
-returning a float, and taking two floats as arguments. Note that
-if you give arguments to the scanning routine using a
-K&R-style/non-prototyped function declaration, you must terminate
-the definition with a semi-colon (;).
-.PP
-Whenever
-.B yylex()
-is called, it scans tokens from the global input file
-.I yyin
-(which defaults to stdin). It continues until it either reaches
-an end-of-file (at which point it returns the value 0) or
-one of its actions executes a
-.I return
-statement.
-.PP
-If the scanner reaches an end-of-file, subsequent calls are undefined
-unless either
-.I yyin
-is pointed at a new input file (in which case scanning continues from
-that file), or
-.B yyrestart()
-is called.
-.B yyrestart()
-takes one argument, a
-.B FILE *
-pointer (which can be nil, if you've set up
-.B YY_INPUT
-to scan from a source other than
-.I yyin),
-and initializes
-.I yyin
-for scanning from that file. Essentially there is no difference between
-just assigning
-.I yyin
-to a new input file or using
-.B yyrestart()
-to do so; the latter is available for compatibility with previous versions
-of
-.I flex,
-and because it can be used to switch input files in the middle of scanning.
-It can also be used to throw away the current input buffer, by calling
-it with an argument of
-.I yyin;
-but better is to use
-.B YY_FLUSH_BUFFER
-(see above).
-Note that
-.B yyrestart()
-does
-.I not
-reset the start condition to
-.B INITIAL
-(see Start Conditions, below).
-.PP
-If
-.B yylex()
-stops scanning due to executing a
-.I return
-statement in one of the actions, the scanner may then be called again and it
-will resume scanning where it left off.
-.PP
-By default (and for purposes of efficiency), the scanner uses
-block-reads rather than simple
-.I getc()
-calls to read characters from
-.I yyin.
-The nature of how it gets its input can be controlled by defining the
-.B YY_INPUT
-macro.
-YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its
-action is to place up to
-.I max_size
-characters in the character array
-.I buf
-and return in the integer variable
-.I result
-either the
-number of characters read or the constant YY_NULL (0 on Unix systems)
-to indicate EOF. The default YY_INPUT reads from the
-global file-pointer "yyin".
-.PP
-A sample definition of YY_INPUT (in the definitions
-section of the input file):
-.nf
-
- %{
- #define YY_INPUT(buf,result,max_size) \\
- { \\
- int c = getchar(); \\
- result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\
- }
- %}
-
-.fi
-This definition will change the input processing to occur
-one character at a time.
-.PP
-When the scanner receives an end-of-file indication from YY_INPUT,
-it then checks the
-.B yywrap()
-function. If
-.B yywrap()
-returns false (zero), then it is assumed that the
-function has gone ahead and set up
-.I yyin
-to point to another input file, and scanning continues. If it returns
-true (non-zero), then the scanner terminates, returning 0 to its
-caller. Note that in either case, the start condition remains unchanged;
-it does
-.I not
-revert to
-.B INITIAL.
-.PP
-If you do not supply your own version of
-.B yywrap(),
-then you must either use
-.B %option noyywrap
-(in which case the scanner behaves as though
-.B yywrap()
-returned 1), or you must link with
-.B \-lfl
-to obtain the default version of the routine, which always returns 1.
-.PP
-Three routines are available for scanning from in-memory buffers rather
-than files:
-.B yy_scan_string(), yy_scan_bytes(),
-and
-.B yy_scan_buffer().
-See the discussion of them below in the section Multiple Input Buffers.
-.PP
-The scanner writes its
-.B ECHO
-output to the
-.I yyout
-global (default, stdout), which may be redefined by the user simply
-by assigning it to some other
-.B FILE
-pointer.
-.SH START CONDITIONS
-.I flex
-provides a mechanism for conditionally activating rules. Any rule
-whose pattern is prefixed with "<sc>" will only be active when
-the scanner is in the start condition named "sc". For example,
-.nf
-
- <STRING>[^"]* { /* eat up the string body ... */
- ...
- }
-
-.fi
-will be active only when the scanner is in the "STRING" start
-condition, and
-.nf
-
- <INITIAL,STRING,QUOTE>\\. { /* handle an escape ... */
- ...
- }
-
-.fi
-will be active only when the current start condition is
-either "INITIAL", "STRING", or "QUOTE".
-.PP
-Start conditions
-are declared in the definitions (first) section of the input
-using unindented lines beginning with either
-.B %s
-or
-.B %x
-followed by a list of names.
-The former declares
-.I inclusive
-start conditions, the latter
-.I exclusive
-start conditions. A start condition is activated using the
-.B BEGIN
-action. Until the next
-.B BEGIN
-action is executed, rules with the given start
-condition will be active and
-rules with other start conditions will be inactive.
-If the start condition is
-.I inclusive,
-then rules with no start conditions at all will also be active.
-If it is
-.I exclusive,
-then
-.I only
-rules qualified with the start condition will be active.
-A set of rules contingent on the same exclusive start condition
-describe a scanner which is independent of any of the other rules in the
-.I flex
-input. Because of this,
-exclusive start conditions make it easy to specify "mini-scanners"
-which scan portions of the input that are syntactically different
-from the rest (e.g., comments).
-.PP
-If the distinction between inclusive and exclusive start conditions
-is still a little vague, here's a simple example illustrating the
-connection between the two. The set of rules:
-.nf
-
- %s example
- %%
-
- <example>foo do_something();
-
- bar something_else();
-
-.fi
-is equivalent to
-.nf
-
- %x example
- %%
-
- <example>foo do_something();
-
- <INITIAL,example>bar something_else();
-
-.fi
-Without the
-.B <INITIAL,example>
-qualifier, the
-.I bar
-pattern in the second example wouldn't be active (i.e., couldn't match)
-when in start condition
-.B example.
-If we just used
-.B <example>
-to qualify
-.I bar,
-though, then it would only be active in
-.B example
-and not in
-.B INITIAL,
-while in the first example it's active in both, because in the first
-example the
-.B example
-startion condition is an
-.I inclusive
-.B (%s)
-start condition.
-.PP
-Also note that the special start-condition specifier
-.B <*>
-matches every start condition. Thus, the above example could also
-have been written;
-.nf
-
- %x example
- %%
-
- <example>foo do_something();
-
- <*>bar something_else();
-
-.fi
-.PP
-The default rule (to
-.B ECHO
-any unmatched character) remains active in start conditions. It
-is equivalent to:
-.nf
-
- <*>.|\\n ECHO;
-
-.fi
-.PP
-.B BEGIN(0)
-returns to the original state where only the rules with
-no start conditions are active. This state can also be
-referred to as the start-condition "INITIAL", so
-.B BEGIN(INITIAL)
-is equivalent to
-.B BEGIN(0).
-(The parentheses around the start condition name are not required but
-are considered good style.)
-.PP
-.B BEGIN
-actions can also be given as indented code at the beginning
-of the rules section. For example, the following will cause
-the scanner to enter the "SPECIAL" start condition whenever
-.B yylex()
-is called and the global variable
-.I enter_special
-is true:
-.nf
-
- int enter_special;
-
- %x SPECIAL
- %%
- if ( enter_special )
- BEGIN(SPECIAL);
-
- <SPECIAL>blahblahblah
- ...more rules follow...
-
-.fi
-.PP
-To illustrate the uses of start conditions,
-here is a scanner which provides two different interpretations
-of a string like "123.456". By default it will treat it as
-three tokens, the integer "123", a dot ('.'), and the integer "456".
-But if the string is preceded earlier in the line by the string
-"expect-floats"
-it will treat it as a single token, the floating-point number
-123.456:
-.nf
-
- %{
- #include <math.h>
- %}
- %s expect
-
- %%
- expect-floats BEGIN(expect);
-
- <expect>[0-9]+"."[0-9]+ {
- printf( "found a float, = %f\\n",
- atof( yytext ) );
- }
- <expect>\\n {
- /* that's the end of the line, so
- * we need another "expect-number"
- * before we'll recognize any more
- * numbers
- */
- BEGIN(INITIAL);
- }
-
- [0-9]+ {
- printf( "found an integer, = %d\\n",
- atoi( yytext ) );
- }
-
- "." printf( "found a dot\\n" );
-
-.fi
-Here is a scanner which recognizes (and discards) C comments while
-maintaining a count of the current input line.
-.nf
-
- %x comment
- %%
- int line_num = 1;
-
- "/*" BEGIN(comment);
-
- <comment>[^*\\n]* /* eat anything that's not a '*' */
- <comment>"*"+[^*/\\n]* /* eat up '*'s not followed by '/'s */
- <comment>\\n ++line_num;
- <comment>"*"+"/" BEGIN(INITIAL);
-
-.fi
-This scanner goes to a bit of trouble to match as much
-text as possible with each rule. In general, when attempting to write
-a high-speed scanner try to match as much possible in each rule, as
-it's a big win.
-.PP
-Note that start-conditions names are really integer values and
-can be stored as such. Thus, the above could be extended in the
-following fashion:
-.nf
-
- %x comment foo
- %%
- int line_num = 1;
- int comment_caller;
-
- "/*" {
- comment_caller = INITIAL;
- BEGIN(comment);
- }
-
- ...
-
- <foo>"/*" {
- comment_caller = foo;
- BEGIN(comment);
- }
-
- <comment>[^*\\n]* /* eat anything that's not a '*' */
- <comment>"*"+[^*/\\n]* /* eat up '*'s not followed by '/'s */
- <comment>\\n ++line_num;
- <comment>"*"+"/" BEGIN(comment_caller);
-
-.fi
-Furthermore, you can access the current start condition using
-the integer-valued
-.B YY_START
-macro. For example, the above assignments to
-.I comment_caller
-could instead be written
-.nf
-
- comment_caller = YY_START;
-
-.fi
-Flex provides
-.B YYSTATE
-as an alias for
-.B YY_START
-(since that is what's used by AT&T
-.I lex).
-.PP
-Note that start conditions do not have their own name-space; %s's and %x's
-declare names in the same fashion as #define's.
-.PP
-Finally, here's an example of how to match C-style quoted strings using
-exclusive start conditions, including expanded escape sequences (but
-not including checking for a string that's too long):
-.nf
-
- %x str
-
- %%
- char string_buf[MAX_STR_CONST];
- char *string_buf_ptr;
-
-
- \\" string_buf_ptr = string_buf; BEGIN(str);
-
- <str>\\" { /* saw closing quote - all done */
- BEGIN(INITIAL);
- *string_buf_ptr = '\\0';
- /* return string constant token type and
- * value to parser
- */
- }
-
- <str>\\n {
- /* error - unterminated string constant */
- /* generate error message */
- }
-
- <str>\\\\[0-7]{1,3} {
- /* octal escape sequence */
- int result;
-
- (void) sscanf( yytext + 1, "%o", &result );
-
- if ( result > 0xff )
- /* error, constant is out-of-bounds */
-
- *string_buf_ptr++ = result;
- }
-
- <str>\\\\[0-9]+ {
- /* generate error - bad escape sequence; something
- * like '\\48' or '\\0777777'
- */
- }
-
- <str>\\\\n *string_buf_ptr++ = '\\n';
- <str>\\\\t *string_buf_ptr++ = '\\t';
- <str>\\\\r *string_buf_ptr++ = '\\r';
- <str>\\\\b *string_buf_ptr++ = '\\b';
- <str>\\\\f *string_buf_ptr++ = '\\f';
-
- <str>\\\\(.|\\n) *string_buf_ptr++ = yytext[1];
-
- <str>[^\\\\\\n\\"]+ {
- char *yptr = yytext;
-
- while ( *yptr )
- *string_buf_ptr++ = *yptr++;
- }
-
-.fi
-.PP
-Often, such as in some of the examples above, you wind up writing a
-whole bunch of rules all preceded by the same start condition(s). Flex
-makes this a little easier and cleaner by introducing a notion of
-start condition
-.I scope.
-A start condition scope is begun with:
-.nf
-
- <SCs>{
-
-.fi
-where
-.I SCs
-is a list of one or more start conditions. Inside the start condition
-scope, every rule automatically has the prefix
-.I <SCs>
-applied to it, until a
-.I '}'
-which matches the initial
-.I '{'.
-So, for example,
-.nf
-
- <ESC>{
- "\\\\n" return '\\n';
- "\\\\r" return '\\r';
- "\\\\f" return '\\f';
- "\\\\0" return '\\0';
- }
-
-.fi
-is equivalent to:
-.nf
-
- <ESC>"\\\\n" return '\\n';
- <ESC>"\\\\r" return '\\r';
- <ESC>"\\\\f" return '\\f';
- <ESC>"\\\\0" return '\\0';
-
-.fi
-Start condition scopes may be nested.
-.PP
-Three routines are available for manipulating stacks of start conditions: