view release on metacpan or  search on metacpan

lib/HTML/FormatExternal.pm  view on Meta::CPAN

  }
}

sub _run_version {
  my ($self_or_class, $command_aref, @ipc_options) = @_;
  ### _run_version() ...
  ###  $command_aref
  ### @ipc_options

  if (! @ipc_options) {
    @ipc_options = ('2>', File::Spec->devnull);
  }

  my $version;  # left undef if any exec/slurp problem
  eval { IPC::Run::run($command_aref,
                       '<', File::Spec->devnull,
                       '>', \$version,
                       @ipc_options) };

  # strip blank lines at end of lynx, maybe others
  if (defined $version) { $version =~ s/\n+$/\n/s; }
  return $version;
}

# return a File::Temp filehandle object
sub _tempfile {
  require File::Temp;
  my $fh = File::Temp->new (TEMPLATE => 'HTML-FormatExternal-XXXXXX',
                            SUFFIX => '.html',
                            TMPDIR => 1);
  binmode($fh) or die 'Oops, cannot set binmode() on temp file';

  ### tempfile: $fh->filename
  #  $fh->unlink_on_destroy(0);  # to preserve for debugging ...

  return $fh;
}

sub _output_wide {
  my ($options, $input_wide) = @_;
  if (! defined $options->{'output_wide'}
      || $options->{'output_wide'} eq 'as_input') {
    $options->{'output_wide'} = $input_wide;
  }
}

# $str is HTML or some initial bytes.
# Return a new string with <base> at the start.
# 
sub _base_prefix {
  my ($options, $str, $input_wide) = @_;
  my $base = delete $options->{'base'};
  ### _base_prefix: $base

  $base = "$base";           # stringize possible URI object
  $base = _entitize($base);  # probably shouldn't be any non-ascii in a url
  $base = "<base href=\"$base\">\n";

  my $pos = 0;
  unless ($input_wide) {
    # encode $base in the input_charset, and possibly after a BOM.
    #
    # Lynx recognises a BOM, if it doesn't have other -assume_charset.  It
    # recognises it only at the start of the file, so must insert <base>
    # after it here to preserve that feature of Lynx.
    #
    # If input_charset is utf-32 or utf-16 then it seems reasonable to step
    # over any BOM.  But Lynx for some reason doesn't like a BOM together
    # with utf-32 or utf-16 specified.  Dunno if that's a bug or a feature
    # on its part.

    my $input_charset = $options->{'input_charset'};
    if (! defined $input_charset || lc($input_charset) eq 'utf-32') {
      if ($str =~ /^\000\000\376\377/) {
        $input_charset = 'utf-32be';
        $pos = 4;
      } elsif ($str =~ /^\377\376\000\000/) {
        $input_charset = 'utf-32le';
        $pos = 4;
      }
    }
    if (! defined $input_charset || lc($input_charset) eq 'utf-16') {
      if ($str =~ /^\376\377/) {
        $input_charset = 'utf-16be';
        $pos = 4;
      } elsif ($str =~ /^\377\376/) {
        $input_charset = 'utf-16le';
        $pos = 2;
      }
    }
    if (defined $input_charset) {
      # encode() errors out if unknown charset, and doesn't exist for older
      # Perl, in which case leave $base as ascii.  May not be right, but
      # ought to work with the various ASCII superset encodings.
      eval {
        require Encode;
        $base = Encode::encode ($input_charset, $base);
      };
    }
  }
  substr($str, $pos,0, $base);  # insert $base at $pos
  return $str;
}

# return $str with non-ascii replaced by &#123; entities
sub _entitize {
  my ($str) = @_;
  $str =~ s{([^\x20-\x7E])}{'&#'.ord($1).';'}eg;
  ### $str
  return $str;
}

1;
__END__

=for stopwords HTML-FormatExternal formatter formatters charset charsets TreeBuilder ie latin-1 config Elinks absolutized tty Ryde filename recognise BOM UTF entitized unrepresentable untaint superset onwards overstriking

=head1 NAME

HTML::FormatExternal - HTML to text formatting using external programs

=head1 DESCRIPTION

This is a collection of formatter modules which turn HTML into plain text by
dumping it through the respective external programs.

    HTML::FormatText::Elinks
    HTML::FormatText::Html2text
    HTML::FormatText::Links
    HTML::FormatText::Lynx
    HTML::FormatText::Netrik
    HTML::FormatText::Vilistextum
    HTML::FormatText::W3m
    HTML::FormatText::Zen

The module interfaces are compatible with C<HTML::Formatter> modules such as
C<HTML::FormatText>, but the external programs do all the work.

Common formatting options are used where possible, such as C<leftmargin> and
C<rightmargin>.  So just by switching the class you can use a different
program (or the plain C<HTML::FormatText>) according to personal preference,
or strengths and weaknesses, or what you've got.

There's nothing particularly difficult about piping through these programs,
but a unified interface hides details like how to set margins and how to
force input or output charsets.

=head1 FUNCTIONS

Each of the classes above provide the following functions.  The C<XXX> in
the class names here is a placeholder for any of C<Elinks>, C<Lynx>, etc as
above.

See F<examples/demo.pl> in the HTML-FormatExternal sources for a complete
sample program.

=head2 Formatter Compatible Functions

=over 4

=item C<< $text = HTML::FormatText::XXX->format_file ($filename, key=>value,...) >>

=item C<< $text = HTML::FormatText::XXX->format_string ($html_string, key=>value,...) >>

Run the formatter program over a file or string with the given options and
return the formatted result as a string.  See L</OPTIONS> below for possible
key/value options.  For example,

    $text = HTML::FormatText::Lynx->format_file ('/my/file.html');

    $text = HTML::FormatText::W3m->format_string
      ('<html><body> <p> Hello world! </p </body></html>');

C<format_file()> ensures any C<$filename> is interpreted as a filename (by
escaping as necessary against however the programs interpret command line
arguments).

lib/HTML/FormatExternal.pm  view on Meta::CPAN

using the options in C<$formatter>, and return the result as a string.

A TreeBuilder argument (ie. a C<HTML::Element>) is accepted for
compatibility with C<HTML::Formatter>.  The tree is simply turned into a
string with C<< $tree->as_HTML >> to pass to the program, so if you've got a
string already then give that instead of a tree.

C<HTML::Element> itself has a C<format()> method (see
L<HTML::Element/format>) which runs a given C<$formatter>.
A C<HTML::FormatExternal> object can be used for C<$formatter>.

    $text = $tree->format($formatter);

    # which dispatches to
    $text = $formatter->format($tree);

=back

=head2 Extra Functions

The following are extra methods not available in the plain
C<HTML::FormatText>.

=over 4

=item C<< HTML::FormatText::XXX->program_version () >>

=item C<< HTML::FormatText::XXX->program_full_version () >>

=item C<< $formatter->program_version () >>

=item C<< $formatter->program_full_version () >>

Return the version number of the formatter program as reported by its
C<--version> or similar option.  If the formatter program is not available
then return C<undef>.

C<program_version()> is the bare version number, perhaps with "beta" or
similar indication.  C<program_full_version()> is the entire version output,
which may include build options, copyright notice, etc.

    $str = HTML::FormatText::Lynx->program_version();
    # eg. "2.8.7dev.10"

    $str = HTML::FormatText::W3m->program_full_version();
    # eg. "w3m version w3m/0.5.2, options lang=en,m17n,image,..."

The version number of the respective Perl module itself is available in the
usual way (see L<UNIVERSAL/VERSION>).

    $modulever = HTML::FormatText::Netrik->VERSION;
    $modulever = $formatter->VERSION

=back

=head1 CHARSETS

File or byte string input is by default interpreted by the programs in their
usual ways.  This should mean HTML Latin-1 but user configurations might
override that and some programs recognise a C<< <meta> >> charset
declaration or a Unicode BOM.  The C<input_charset> option below can force
the input charset.

Perl wide-character input string is encoded and passed to the program in
whatever way it best understands.  Usually this is UTF-8 but in some cases
it is entitized instead.  The C<input_charset> option can force the input
charset to use if for some reason UTF-8 is not best.

The output string is either bytes or wide chars.  By default output is the
same as input, so wide char string input gives wide output and byte input
string or file input gives byte output.  The C<output_wide> option can force
the output type (and is the way to get wide chars back from
C<format_file()>).

Byte output is whatever the program produces.  Its default might be the
locale charset or other user configuration which suits direct display to the
user's terminal.  The C<output_charset> option can force the output to be
certain or to be ready for further processing.

Wide char output is done by choosing the best output charset the program can
do and decoding its output.  Usually this means UTF-8 but some of the
programs may only have less.  The C<output_charset> option can force the
charset used and decoded.  If it's something less than UTF-8 then some
programs might for example give ASCII art approximations of otherwise
unrepresentable characters.

Byte input is usual for HTML downloaded from a HTTP server or from a MIME
email and the headers have the C<input_charset> which applies.  Byte output
is good to go straight out to a tty or back to more MIME etc.  The input and
output charsets could differ if a server gives something other than what you
want for final output.

Wide chars are most convenient for crunching text within Perl.  The default
wide input giving wide output is designed to be transparent for this.

For reference, if a C<HTML::Element> tree contains wide char strings then
its usual C<as_HTML()> method, which is used by C<format()> above, produces
wide char HTML so the formatters here give wide char text.  Actually
C<as_HTML()> produces all ASCII because its default behaviour is to entitize
anything "unsafe", but it's still a wide char string so the formatted output
text is wide.

=head1 OPTIONS

The following options can be given to the constructor or to the formatting
methods.  The defaults are whatever the respective programs do.  The
programs generally read their config files when dumping so the defaults and
formatting details may follow the user's personal preferences.  Usually this
is a good thing.

=over 4

=item C<< leftmargin => INTEGER >>

=item C<< rightmargin => INTEGER >>

The column numbers for the left and right hand ends of the text.
C<leftmargin> 0 means no padding on the left.  C<rightmargin> is the text
width, so for instance 60 would mean the longest line is 60 characters
(inclusive of any C<leftmargin>).  These options are compatible with
C<HTML::FormatText>.