view release on metacpan or  search on metacpan

lib/CrawlerCommons/RobotRulesParser.pm  view on Meta::CPAN

URL string that's parsed in a URI object to provide scheme, authority, and path
for sitemap directive values.  If the directive's value begins with a '/', it
overrides the path value provided by this URL context string.

=item * C<$content>

The text content of the robots.txt file to be parsed.

=item * C<$content_type>

The content-type of the robots.txt content to be parsed.  Assumes text/plain by
default.  If type is text/html, the parser will attempt to strip-out html tags
from the content.

=item * C<$robot_name>

A string signifying for which user-agent(s) the rules should be extracted.

=back

=cut