Class: Tokenizer
- Package: saf.Parser
- Author: John Luxford <lux@simian.ca>
- Copyright: Copyright (C) 2001-2003, Simian Systems Inc.
- License: http://www.sitellite.org/index/license Simian Open Software License
- Version: 0.6, 2003-01-20, $Id: Tokenizer.php,v 1.3 2007/10/06 00:06:30 lux Exp $
- Access: public
Tokenizer provides an saf.Parser-like wrapper around the PHP tokenizer
extension that can be extended for token analysis and parser writing.
One of its uses in Sitellite is parsing PHP code for I18n text by the
saf.I18n.Builder package.
Note: Requires the PHP tokenizer extension, which is built into PHP by
default as of 4.3.0.
Usage Example
<?php
class ClassFinder extends Tokenizer {
var $classes = array ();
var $next = false;
function ClassFinder () {
$this->addCallback ('_class', T_CLASS);
$this->addCallback ('_string', T_STRING);
}
function find ($code) {
if ($this->parse ($code)) {
return $this->classes;
} else {
return false;
}
}
function _class ($token, $data) {
$this->next = true;
}
function _string ($token, $data) {
if ($this->next) {
$this->classes[] = $token;
$this->next = false;
}
}
}
$finder = new ClassFinder ();
$res = $finder->find ('<?php
class Foo {
var $foo = "Hello";
}
class Bar {
var $bar = "World";
}
');
if (is_array ($res)) {
foreach ($res as $class) {
echo $class . "<br />\n";
}
}
?>
Return to Top
Properties
$output
Contains the output of parse().
$buffers = array ()
The array buffer.
$original
Contains the original data sent to parse().
$struct
Contains the array of parsed elements, aka tokens.
$tokens
Stores the tokens from the previous call to parse()
$regex
Contains the output of makeRegex() on the current token list.
$switches = 's'
Contains a list of switches to the preg_split() and
preg_match() regular expression evaluations Default is 's',
for dot-all mode. Note that these are PCRE (Perl-Compatible
Regular Expression) expressions, not ereg() calls. For more
info about switches, check out the PHP documentation at
http://www.php.net/manual/en/pcre.pattern.modifiers.php
$code
Stores the code that is passed to the previous call to
parse().
$error
Contains the message if an error occurs.
Return to Top
Methods
normalize ($tokens)
- Access: public
- Return: array
Fixes some quirks and exceptions to the structure of
the output of the PHP token_get_all() function.
parse ($code)
- Access: public
- Return: boolean
Parses the specified $code and calls the callback
method assigned to each token type.
addCallback ($function, $token)
Adds a callback $function for the specified $token.
The $token is the number of the token, which may be expressed
by passing the appropriate T_* tokenizer constant.
_default ($token, $data)
- Access: public
- Return: boolean
This is the default token handler. It handles all
tokens that haven't been assigned custom callback functions.
Return to Top