view release on metacpan or  search on metacpan

lib/CodeGen/Cpppp/CParser.pm  view on Meta::CPAN

Whitespace is ignored (not returned as a token) except for whitespace contained
in a 'directive' token.  The body of a directive needs further tokenized.

Each token is an arrayref of the form:

  [ $type, $value, $offset, $length, $error=undef ]
  
  $type:   'directive', 'comment', 'string', 'char', 'real', 'integer',
           'keyword', 'ident', 'unknown', or any punctuation character
  
  $value:  for constants, this is the decoded string or numeric value
           for directives and comments, it is the body text
           for punctuation, it is a copy of $type
           for unknown, it is the exact character that didn't parse
  
  $src_pos: the character offset within the source $string
  
  $src_len: the number of characters occupied in the source $string
  
  $error: if the token is invalid in some way, but still undisputedly that
          type of token (e.g. unclosed string or unclosed comment) it will be