Parser Tokens

PHP exposes parser token constants for tools that inspect source code. Token streams are useful for lightweight analysis, while real refactoring tools should normally use a full parser that understands syntax structure.

Use This Reference When

• Building a source inspection script.
• Understanding output from token_get_all().
• Choosing whether a task needs tokens or an abstract syntax tree.

Inspect Tokens

<?php

$tokens = token_get_all('<?php echo "hi";');
foreach ($tokens as $token) {
    echo is_array($token) ? token_name($token[0]) : $token;
    echo PHP_EOL;
}

// Prints token names and punctuation.

Tokens retain lexical information such as whitespace and comments. They do not by themselves provide the richer relationships a parser library builds.