WWW-Meta-XML-Browser
view release on metacpan or search on metacpan
=head2 WRITING A SESSION DESCRIPTION FILE
The most important part when working with C<WWW::Meta::XML::Browser> is to write a session description file. Such a file describes which http requests are made and how the results of the requests are handled.
The session description file is a simple XML file. The root element is E<lt>www-meta-xml-browserE<gt> and the DTD can be found at L<http://www.boksa.de/pub/xml/dtd/www-meta-xml-browser_v0.08.dtd>, which leads us to the following construct:
<?xml version="1.0" ?>
<!DOCTYPE www-meta-xml-browser SYSTEM "http://www.boksa.de/pub/xml/dtd/www-meta-xml-browser_v0.08.dtd">
<www-meta-xml-browser>
<!-- ... -->
</www-meta-xml-browser>
The optional meta-element can be specified as a child of the root element. The element acts as a container for different information regarding the handling of the request elements.
=head3 META-PERL INFORMATION
The perl element is a child of the meta element and can contain perl related information. The perl element can have one of the child elements described below.
=head4 ELEMENT: callback; ATTRIBUTES: name
The callback element is used to define an anonymous subroutine which can later be used as a callback. The name under which the callback can be accessed is specified by the required name attribut. The form of the callback (parameters, return value) de...
<callback name="some-callback"><![CDATA[
sub {
my ($result) = @_;
return $result;
}
]]></callback>
=head3 REQUEST DEFINITIONS
A session description file must contain at least one request.
=head4 DEFINING A REQUEST WITHOUT CONTENT
Under the root element we will add some elements for the requests we want to make. A very complete request will look like the following:
<request url="http://www.google.de/" method="get" stylesheet="google-index.xsl" result-callback="some-callback">
</request>
The only attribute of the request-element that is required is url, all other attributes can be left out.
If method is left out the default method get will be used.
If stylesheet is left out, the raw html will be transformed to a valid XML document which will than be stored as the result of that request.
The result-callback gives the user the possibility to change the raw html before it will be transformed to a XML document by calling the specified callback. This callback can be an element of the callbacks hash specified when the instance is created ...
=head4 DEFINING A REQUEST WITH CONTENT
The request-element has an optional child element, which can be used to specify the content of a request. The element is called content and is used as a child of the request element as follows (remember that & has to be written as & in XML):
<request url="http://www.google.de/search" method="get">
<content>q=42&ie=ISO-8859-1&hl=de&meta=</content>
</request>
This example shows that the content will be sent using the specified method (get in this case) to the url of the request (http://www.google.de/search).
=head4 EMBEDDED REQUESTS
Embedded request can be used to fetch pages from a result page. They can be created in the XSL stylesheet to dynamically parse a tree of pages.
As soon as a www-meta-xml-browser-request-element is created in the XSL stylesheet it is processed like a normal request-element and the result is inserted.
If the result consists of multiple pages the container-attribute has to be specified and is used as the new root for the merged (optionally transformed) pages.
=head3 REPLACEMENT EXPRESSIONS IN A SESSION DESCRIPTION FILE
There are some cases in which static urls and a static content don't fit the requirements of what has to be done.
For this case WWW::Meta::XML::Browser has an easy way to use arguments passed to the instance during creation or values from a previous result.
To access arguments passed to the instance during creation the following simple syntax is used:
#{args:key}
The word key has to be replaced with the key of the hash containing the arguments. This will lead to the replacement of C<#{args:key}> with the appropriate value from the hash.
Accessing previous results basically goes the same way, some example show, that it even offers more possibilities:
#{0:0:/foo}
#{4:1-3:/foo/too}
#{1::/foo/@argument}
#escape{0:0:/foo}
#escape{4:1-3:/foo/too}
#escape{1::/foo/@argument}
The first three example and the last three examples have only one difference, which is the word escape. This command simply tells the module to url-escape the value that is returned by that later part of the expression.
Let's look at these expressions in detail:
The first part (the number before the first colon) specifies the index (starting with 0) of the request which we want to access. This index can be mapped directly to the session description file.
The second part (between the first and the second colon) specifies the subrequest results (more about subrequests later) that will be looked at. 0 in the first example specifies the first subrequest. 1-3 in the second example specifies the subrequest...
The last part (after the second colon) specifies an XPath-Expression, which is looked up in each of the subrequest results and a list of all values which match the Expression is generated.
This list is taken and each value of the list will replace the whole replacement expression, and for each replacement one http request is made.
Naturally if the url or the content contains more than one replacement expression all possible combinations are requested (which actually is the product of the different numbers of matching XPath-Expressions).
These different http requests make up the subrequests which are stored and can be accessed, when needed. Please not that subrequests can be merged into a singele subrequest result using L<merge_subrequests()>.
=head2 CREATING A NEW BROWSER OBJECT
To create a new browser object the L<new()>-method is called, with an optional hash containing options.
$browser = WWW::Meta::XML::Browser->new(%options);
The following options are possible:
args => \%args
C<\%args> is the pointer to a hash which values can be accessed from the session description file by their keys. The syntax to access the hash values from the session file is C<#{args:key}>, where key is a key from the hash.
debug => 1
When the debug option is set, the module produces a lot of debug output about execution times.
For example C<$browser-E<gt>get_request_result(0, 2)> returns the result of the third request generated from the first request node in the session description file.
=head1 EXPORT
None by default.
=cut
require Exporter;
our @ISA = qw(Exporter);
# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.
# This allows declaration use WWW::Meta::XML::Browser ':all';
# If you do not need this, moving things directly into @EXPORT or @EXPORT_OK
# will save memory.
our %EXPORT_TAGS = ( 'all' => [ qw(
) ] );
our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } );
our @EXPORT = qw(
);
our $VERSION = '0.08';
use Digest::MD5;
use HTTP::Cookies;
use HTTP::Request;
use LWP::UserAgent;
use Time::HiRes;
use URI::Escape;
use XML::LibXML;
use XML::LibXSLT;
my $ROOT_XPATH = '/www-meta-xml-browser';
my $META_XPATH = $ROOT_XPATH.'/meta';
my $PERL_META_XPATH = $META_XPATH.'/perl';
my $CALLBACK_XPATH = $PERL_META_XPATH.'/callback';
my $REQUEST_XPATH = $ROOT_XPATH.'/request';
my $AUTHORIZATION_XPATH = './authorization';
my $CONTENT_XPATH = './content';
my $XPATH_REGEXP = '\#(escape)*\{(\d+?):(.*?):(.+?)\}';
my $ARGS_REGEXP = '\#(escape)*\{args:(.*?)\}';
my $URL_ATTRIBUTE = 'url';
my $METHOD_ATTRIBUTE = 'method';
my $RESULT_CALLBACK_ATTRIBUTE = 'result-callback';
my $STYLESHEET_ATTRIBUTE = 'stylesheet';
my $CALLBACK_NAME_ATTRIBUTE = 'name';
my $EMBEDDED_REQUEST_CONTAINER_ATTRIBUTE = 'container';
my $XML_VERSION = '1.0';
my $USER_AGENT = "WWW::Meta::XML::Browser ".$VERSION;
my $TIMEOUT = 30;
=head1 METHODS
The following methods are available:
=over 4
=cut
=item $browser = WWW::Meta::XML::Browser->new(%options);
This class method contructs a new C<WWW::Meta::XML::Browser> object and returns a reference to it.
The hash C<%options> can be used to control the behaviour of the module and to provide some data to it as well. At the moment the following Key/Value pairs are supported:
KEY: VALUE: DESCRIPTION:
--------------- ----------- -------------
args \%args a pointer to a hash of arguments which can be used in
requests
debug 0/1 a boolean true or boolean false value can be passed to
the module to control weather debugging information are
printed or not
debug_callback \&debug a pointer to a debug-callback
result_doc_callback \&result a pointer to a result-doc-callback
callbacks \%callbacks a pointer to a hash of subroutines which can be used as
callbacks in different situations
=cut
sub new {
my $type = shift;
my (%cnf) = @_;
my $this = {};
bless $this, $type;
$this->{debug_callback} = \&_debug;
$this->{result_doc_callback} = \&_result;
$this->{callbacks} = {};
$this->{args} = $cnf{'args'} if $cnf{'args'};
$this->{debug} = 1 if $cnf{'debug'};
$this->{debug_callback} = $cnf{'debug_callback'} if $cnf{'debug_callback'};
$this->{result_doc_callback} = $cnf{'result_doc_callback'} if $cnf{'result_doc_callback'};
$this->{callbacks} = $cnf{'callbacks'} if $cnf{'callbacks'};
$this->{request_nodes} = ();
$this->{request_results} = ();
$this->{ua} = LWP::UserAgent->new(cookie_jar => HTTP::Cookies->new(), requests_redirectable => ['GET', 'POST', 'HEAD']);
$this->{ua}->agent($USER_AGENT);
&{$this->{debug_callback}}(1, "process_result() called") if $this->{debug};
# the result doc is undef by default and will not change if the request was not successfull
my $doc = undef;
my $t0 = [Time::HiRes::gettimeofday()] if $this->{debug};
# create a parser for the result
my $parser = XML::LibXML->new();
$parser->recover(1);
# parse the html and generate the result doc
$doc = $parser->parse_html_string($result);
&{$this->{debug_callback}}(2, "time to parse html: ".Time::HiRes::tv_interval($t0)) if $this->{debug};
# if a stylesheet has been specified use it to transform the result doc
if ($stylesheet) {
my $t0 = [Time::HiRes::gettimeofday()] if $this->{debug};
my $style_doc = $parser->parse_file($stylesheet);
my $xslt = XML::LibXSLT->new();
my $stylesheet = $xslt->parse_stylesheet($style_doc);
# overwrite the old result doc with the new result doc
$doc = $stylesheet->transform($doc);
&{$this->{debug_callback}}(2, "time to transform result: ".Time::HiRes::tv_interval($t0)) if $this->{debug};
}
# processing embedded requests after having applied the stylesheet if it has been specified
my $doc_string = $doc->toString();
my $contains_embedded_request = 0;
if ($doc_string =~ /(<www-meta-xml-browser-request.*?\/>)/gis) {
$doc_string =~ s/(<www-meta-xml-browser-request.*?\/>)/$this->process_embedded_request($parser->parse_string($1)->getDocumentElement())/egis;
$contains_embedded_request = 1;
}
if ($doc_string =~ /(<www-meta-xml-browser-request.*?>.+?<\/www-meta-xml-browser-request>)/gis) {
$doc_string =~ s/(<www-meta-xml-browser-request.*?>.+?<\/www-meta-xml-browser-request>)/$this->process_embedded_request($parser->parse_string($1)->getDocumentElement())/egis;
$contains_embedded_request = 1;
}
if ($contains_embedded_request) {
$doc = $parser->parse_string($doc_string);
}
return &{$this->{result_doc_callback}}($doc);
}
=item $xml_string = process_embedded_request($embedded_request_node);
Processes an embedded request node, by processing it as a normal node (using L<process_request_node()>).
If the embedded request node returns only one XML document it is transformed to a string and returned.
If the embedded request node returns more than one XML documents they are merged unded the name specified by the C<$EMBEDDED_REQUEST_CONTAINER_ATTRIBUTE>-attribute of the embedded requst node.
=cut
sub process_embedded_request {
my $this = shift;
my ($er_node) = @_;
my $subrequest_result = $this->process_request_node($er_node);
if (scalar(@{$subrequest_result}) > 1) {
my $doc = $this->merge_xml_array($subrequest_result, $er_node->getAttribute($EMBEDDED_REQUEST_CONTAINER_ATTRIBUTE));
return $doc->documentElement()->toString();
}
else {
return ${$subrequest_result}[0]->documentElement()->toString();
}
}
=item $result = get_request_result($request_index, $subrequest_index);
Returns the request-result specified by C<$request_index> and C<$subrequest_index>.
=cut
sub get_request_result {
my $this = shift;
my ($request_index, $subrequest_index) = @_;
return ${$this->{request_results}}[$request_index][$subrequest_index];
}
=item print_all_request_results();
Iterates over all the request results and prints them.
=cut
sub print_all_request_results {
my $this = shift;
my @requests = @{$this->{request_results}};
my $r = 0;
foreach my $request (@requests) {
my @subrequests = @{$request};
print "-------------------- REQUEST (".($r++ + 1)."/".($#requests + 1).") --------------------\n";
my $s = 0;
foreach (@subrequests) {
print "-------------------- SUBREQUEST (".($s++ + 1)."/".($#subrequests + 1).")--------------------\n";
$this->print_request_result($_);
}
}
}
=item print_request_result($result);
Prints the specified request result.
=cut
sub print_request_result {
my $this = shift;
( run in 3.245 seconds using v1.01-cache-2.11-cpan-0b58ddf2af1 )