 |
Index for
Section 3 |
|
 |
Alphabetical
listing for H |
|
 |
Bottom of
page |
|
HTML::TokeParser(3)
NAME
HTML::TokeParser - Alternative HTML::Parser interface
SYNOPSIS
require HTML::TokeParser;
$p = HTML::TokeParser->new("index.html") ||
die "Can't open: $!";
while (my $token = $p->get_token) {
#...
}
DESCRIPTION
The "HTML::TokeParser" is an alternative interface to the "HTML::Parser"
class. It is an "HTML::PullParser" subclass with a predeclared set of
token types. If you wish the tokens to be reported differently you
probably want to use the "HTML::PullParser" directly.
The following methods are available:
$p = HTML::TokeParser->new( $filename );
$p = HTML::TokeParser->new( $filehandle );
$p = HTML::TokeParser->new( \$document );
The object constructor argument is either a file name, a file handle
object, or the complete document to be parsed.
If the argument is a plain scalar, then it is taken as the name of a
file to be opened and parsed. If the file can't be opened for reading,
then the constructor will return "undef" and $! will tell you why it
failed.
If the argument is a reference to a plain scalar, then this scalar is
taken to be the literal document to parse. The value of this scalar
should not be changed before all tokens have been extracted.
Otherwise the argument is taken to be some object that the
"HTML::TokeParser" can read() from when it needs more data. Typically
it will be a filehandle of some kind. The stream will be read() until
EOF, but not closed.
Note that the parsing result will likely not be valid if raw undecoded
UTF-8 is used as a source. When parsing UTF-8 encoded files turn on
UTF-8 decoding:
open(my $fh, "<:utf8", "index.html") || die "Can't open 'index.html': $!";
my $p = HTML::TokeParser->new( $fh );
# ...
If a $filename is passed to the constructor the file will be opened in
raw mode and the parsing result will only be valid if its content is
Latin-1 or pure ASCII.
If parsing from an UTF-8 encoded string buffer decode it first:
utf8::decode($document);
my $p = HTML::TokeParser->new( \$document );
# ...
$p->get_token
This method will return the next token found in the HTML document, or
"undef" at the end of the document. The token is returned as an array
reference. The first element of the array will be a string denoting
the type of this token: "S" for start tag, "E" for end tag, "T" for
text, "C" for comment, "D" for declaration, and "PI" for process
instructions. The rest of the token array depend on the type like
this:
["S", $tag, $attr, $attrseq, $text]
["E", $tag, $text]
["T", $text, $is_data]
["C", $text]
["D", $text]
["PI", $token0, $text]
where $attr is a hash reference, $attrseq is an array reference and the
rest are plain scalars. The "Argspec" in HTML::Parser explains the
details.
$p->unget_token( @tokens )
If you find you have read too many tokens you can push them back, so
that they are returned the next time $p->get_token is called.
$p->get_tag
$p->get_tag( @tags )
This method returns the next start or end tag (skipping any other
tokens), or "undef" if there are no more tags in the document. If one
or more arguments are given, then we skip tokens until one of the
specified tag types is found. For example:
$p->get_tag("font", "/font");
will find the next start or end tag for a font-element.
The tag information is returned as an array reference in the same form
as for $p->get_token above, but the type code (first element) is
missing. A start tag will be returned like this:
[$tag, $attr, $attrseq, $text]
The tagname of end tags are prefixed with "/", i.e. end tag is returned
like this:
["/$tag", $text]
$p->get_text
$p->get_text( @endtags )
This method returns all text found at the current position. It will
return a zero length string if the next token is not text. Any entities
will be converted to their corresponding character.
If one or more arguments are given, then we return all text occurring
before the first of the specified tags found. For example:
$p->get_text("p", "br");
will return the text up to either a paragraph of linebreak element.
The text might span tags that should be textified. This is controlled
by the $p->{textify} attribute, which is a hash that defines how
certain tags can be treated as text. If the name of a start tag
matches a key in this hash then this tag is converted to text. The
hash value is used to specify which tag attribute to obtain the text
from. If this tag attribute is missing, then the upper case name of
the tag enclosed in brackets is returned, e.g. "[IMG]". The hash value
can also be a subroutine reference. In this case the routine is called
with the start tag token content as its argument and the return value
is treated as the text.
The default $p->{textify} value is:
{img => "alt", applet => "alt"}
This means that <IMG> and <APPLET> tags are treated as text, and that
the text to substitute can be found in the ALT attribute.
$p->get_trimmed_text
$p->get_trimmed_text( @endtags )
Same as $p->get_text above, but will collapse any sequences of white
space to a single space character. Leading and trailing white space is
removed.
$p->get_phrase
This will return all text found at the current position ignoring any
phrasal-level tags. Text is extracted until the first non phrasal-
level tag. Textification of tags is the same as for get_text(). This
method will collapse white space in the same way as get_trimmed_text()
does.
The definition of <i>phrasal-level tags</i> is obtained from the
HTML::Tagset module.
EXAMPLES
This example extracts all links from a document. It will print one line
for each link, containing the URL and the textual description between the
<A>...</A> tags:
use HTML::TokeParser;
$p = HTML::TokeParser->new(shift||"index.html");
while (my $token = $p->get_tag("a")) {
my $url = $token->[1]{href} || "-";
my $text = $p->get_trimmed_text("/a");
print "$url\t$text\n";
}
This example extract the <TITLE> from the document:
use HTML::TokeParser;
$p = HTML::TokeParser->new(shift||"index.html");
if ($p->get_tag("title")) {
my $title = $p->get_trimmed_text;
print "Title: $title\n";
}
SEE ALSO
HTML::PullParser, HTML::Parser
COPYRIGHT
Copyright 1998-2001 Gisle Aas.
This library is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.
|
Index for
Section 3 |
|
|
Alphabetical
listing for H |
|
 |
Top of
page |
|