view release on metacpan or  search on metacpan

lib/App/FetchwareX/HTMLPageSync.pm  view on Meta::CPAN

It receives a list of all of the download HTML::Elements that
C<html_treebuilder_callback> returned true on. It is called only once, and
should return a list of string download links for download later by
HTMLPageSync.
EOA
            keep_destination_directory => <<EOA,
keep_destination_directory is a boolean true or false configuration option that
when true prevents HTMLPageSync from deleting your destination_directory when
you run fetchware uninstall.
EOA
        }
    );

    extension_name(__PACKAGE__);

    opening_message(<<EOM);
HTMLPageSync's new command is not as sophistocated as Fetchware's. Unless you
only want to download images, you will have to get your hands dirty, and code up
some custom Perl callbacks to customize HTMLPageSync's behavior. However, it
will ask you quite nicely the basic options, so if those are all you need, then
this command will successfully generate a HTMLPageSync Fetchwarefile for you.

After it lets you choose the easy options of page_name, html_page_url,
and destination_directory, it will give you an opportunity to modify the
user_agent string HTMLPageSync uses to avoid betting banned or having your
scraping stick out like a sore thumb in the target Web server's logs. Then,
you'll be asked about the advanced options. If you want them it will add generic
ones to the Fetchwarefile that you can then fill in later on when HTMLPageSync
asks you if you want to edit the generated Fetchwarefile manually.  Finally,
after your Fetchwarefile is generated HTMLPageSync will ask you if you would
like to install your generated Fetchwarefile to test it out.
EOM

    # Ask the user for the basic configuration options.
    $page_name = fetchwarefile_name(page_name => $page_name);
    vmsg "Determined your page_name option to be [$page_name]";

    $fetchwarefile->config_options(page_name => $page_name);
    vmsg "Appended page_name [$page_name] configuration option to Fetchwarefile";

    my $html_page_url = get_html_page_url($term);
    vmsg "Asked user for html_page_url [$html_page_url] from user.";

    $fetchwarefile->config_options(html_page_url => $html_page_url);
    vmsg "Appended html_page_url [$html_page_url] configuration option to Fetchwarefile";

    my $destination_directory = get_destination_directory($term);
    vmsg "Asked user for destination_directory [$destination_directory] from user.";

    $fetchwarefile->config_options(destination_directory => $destination_directory);
    vmsg <<EOM;
Appended destination_directory [$destination_directory] configuration option to
your Fetchwarefile";
EOM

    # Asks and sets the keep_destination_directory configuratio option if the
    # user wants to set it.
    ask_about_keep_destination_directory($term, $fetchwarefile);

    vmsg 'Prompting for other options that may be needed.';
    my $other_options_hashref = prompt_for_other_options($term,
        user_agent => {
            prompt => <<EOP,
What user_agent configuration option would you like? 
EOP
            print_me => <<EOP
user_agent, if specified, will be passed to HTML::Tiny, the Perl HTTP library
Fetchware uses, where the library will lie to the Web server you are Web
scraping from to hopefully prevent the Web sever from banning you, or updating
the page you want to scrap to use too much Javascript, which would prevent the
simple parser HTMLPageSync uses from working on the specified html_page_url.
EOP
        },
        html_treebuilder_callback => {
            prompt => <<EOP,
What html_treebuilder_callback configuration option would you like? 
EOP
            print_me => <<EOP,
html_treebuilder_callback allows you to specify a perl CODEREF that HTMLPageSync
will execute instead of its default callback that just looks for images.

It receives one parameter, which is an HTML::Element at the first C<a>,
anchor/link tag.

It must [return 'True';] to indicate that that link should be included in the
list of download links, or return false, [return undef], to indicate that that
link should not be included in the list of download links.

Because Term::UI's imput is limited to just one line, please just press enter,
and a dummy value will go into your Fetchwarefile, where you can then replace
that dummy value with a proper Perl callback next, when Fetchware gives you the
option to edit your Fetchwarefile manually.
EOP
            default => 'sub { my $h = shift; die "Dummy placeholder fill me in."; }',
        },
        download_links_callback => {
            prompt => <<EOP,
What download_links_callback configuration option would you like? 
EOP
            print_me => <<EOP,
download_links_callback specifies an optional callback that will allow you to do
post processing of the list of downloaded urls. This is needed, because the
results of the html_treebuilder_callback are still HTML::Element objects that
need to be converted to just string download urls. That is what the default
C<download_links_callback> does.

It receives a list of all of the download HTML::Elements that
C<html_treebuilder_callback> returned true on. It is called only once, and
should return a list of string download links for download later by
HTMLPageSync.

Because Term::UI's imput is limited to just one line, please just press enter,
and a dummy value will go into your Fetchwarefile, where you can then replace
that dummy value with a proper Perl callback next, when Fetchware gives you the
option to edit your Fetchwarefile manually.
EOP
            default => 'sub { my @download_urls = @_; die "Dummy placeholder fill me in."; }',
        },
    );
    vmsg 'User entered the following options.';
    vmsg Dumper($other_options_hashref);

    # Append all other options to the Fetchwarefile.
    $fetchwarefile->config_options(%$other_options_hashref);
    vmsg 'Appended all other options listed above to Fetchwarefile.';

    my $edited_fetchwarefile = edit_manually($term, $fetchwarefile);
    vmsg <<EOM;
Asked user if they would like to edit their generated Fetchwarefile manually.
EOM
    # Generate Fetchwarefile.
    # If edit_manually() did not modify the Fetchwarefile, then generate it.
    if (blessed($edited_fetchwarefile)
        and
    $edited_fetchwarefile->isa('App::Fetchware::Fetchwarefile')) {
        $fetchwarefile = $fetchwarefile->generate(); 
    # If edit_manually() modified the Fetchwarefile, then do not generate it,
    # and replace the Fetchwarefile object with the new string that represents
    # the user's edited Fetchwarefile.
    } else {
        $fetchwarefile = $edited_fetchwarefile;
    }

    # Whatever variables the new() API subroutine returns are written via a pipe
    # back to the parent, and then the parent reads the variables back, and
    # makes then available to new_install(), back in the parent, as arguments.
    return $page_name, $fetchwarefile;
}



sub get_html_page_url {
    my $term = shift;


    # prompt for lookup_url.
    my $html_page_url = $term->get_reply(
        print_me => <<EOP,
Fetchware's heart and soul is its html_page_url. This is the configuration option
that tells fetchware where to check if any new links have been added to the
specified Web page that match your criteria for download.

How to determine your application's html_page_url:
    1. Simply specify the URL that of the Web page that has the images that you
    would like to have Fetchware download for you.
EOP
        prompt => q{What is your Web page's html_page_url? },
        allow => qr!(ftp|http|file)://!);

    return $html_page_url;
}



sub get_destination_directory {
    my $term = shift;

    # prompt for lookup_url.
    my $destination_directory = $term->get_reply(
        print_me => <<EOP,
destination_directory is the directory on your computer where you want the files
that you configure HTMLPageSync to parse to be copied to.
EOP
        prompt => q{What is your destination_directory? });

    return $destination_directory;
}



sub ask_about_keep_destination_directory {
    my ($term, $fetchwarefile) = @_;

    if (
        $term->ask_yn(
        print_me => <<EOP,
By default, HTMLPageSync deletes your destination_directory when you uninstall
that destination_directory's assocated Fetchware package or Fetchwarefile. This
is done, because your deleting the Fetchware package, so it makes sense to
delete that package's associated data.

If you wish to keep your destination_directory after you uninstall this
HTMLPageSync Fetchware package, then answer N below.
EOP
        prompt => 'Is deleting your destination_directory on uninstall OK? ',
        default => 'y',
        )
    ) {
        vmsg <<EOM;
User wants [keep_destination_directory 'True';] added to their Fetchwarefile.
EOM

        $fetchwarefile->config_options(keep_destination_directory => 'True');
        vmsg <<EOM;
Appended [keep_destination_directory 'True';] to user's Fetchwarefile.
EOM
    }
}





sub check_syntax {

    # Use check_config_options() to run config() a bunch of times to check the
    # already parsed Fetchwarefile.
    return check_config_options(
        Mandatory => [ 'page_name', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a page_name configuration
option. Please add one, and try again.
EOM
        Mandatory => [ 'html_page_url', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a html_page_url configuration
option. Please add one, and try again.
EOM
        Mandatory => [ 'destination_directory', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a destination_directory
configuration option. Please add one, and try again.
EOM
    );
}





###BUGALERT### lookup() returns all files each time it is run; therefore, it
#breaks the way Fetchware is supposed to work! lookup() is supposed to return
#"the latest version." And in HTMLPageSync's case, it should not include files
#already downloaded, because it should only return "new files" by comparing the
#"availabe list of files" to the "already downloaded one."
sub lookup {
    msg
    "Looking up download urls using html_page_url [@{[config('html_page_url')]}]";
    ###BUGALERT### Create a user changeable version of lookup_check_args??(), so
    #that App::Fetchware 'subclasses' can use it.
    # Download the url the user specified.
    my $filename = do {
        if (defined config('user_agent')) {
            download_http_url(config('html_page_url'),
                user_agent =>  config('user_agent'));
        } else {
            download_http_url(config('html_page_url'));
        }