App-Fetchware

 view release on metacpan or  search on metacpan

lib/App/Fetchware.pm  view on Meta::CPAN

            lookup_method => <<EOA,
lookup_method specifies how fetchware determines what version of your program
to install. The default is the 'timestamp' algorithm, and then to try the
'versionstring' algorithm if 'timestamp' fails. lookup_method specifies which
one you would like to use. Only the strings 'timestamp' and 'versionstring'
are allowed options.
EOA
            gpg_keys_url => <<EOA,
gpg_keys_url specifies the url that fetchware will use to download the author's
KEYS file that it uses for gpg verification.
EOA
            gpg_sig_url => <<EOA,
gpg_sig_url specifies the url that fetchware uses to download digital
signatures of this program. They're files that usually end .asc.
EOA
            sha1_url => <<EOA,
sha1_url specfies the url that fetchware uses to download sha1sum files of
this program. This url should be the program's main download site instead of a
mirror, because a hacked mirror could alter the sha1sum on that mirror.
EOA
            md5_url => <<EOA,
md5_url specfies the url that fetchware uses to download md5sum files of
this program. This url should be the program's main download site instead of a
mirror, because a hacked mirror could alter the md5sum on that mirror.
EOA
            verify_method => <<EOA,
verify_method specifes a specific method that fetchware should use to verify
your program. This method can be 'gpg', 'sha1', or 'md5'.
EOA
            no_install => <<EOA,
no_install specifies that this software should not be installed. Instead, the
install step is skipped, and fetchware prints to STDOUT where it downloaded,
verified, and built your program. no_install must be a true or false value.
EOA
            verify_failure_ok => <<EOA,
verify_failure_ok specifies that fetchware should not stop installing your
software and terminate with an error message if fetchware fails to verify your
software. You should never set this to true. Doing so could cause fetchware to
install software that may have been compromised, or had malware inserted into
it. Never use this option unless the author or maintainer of this program does
not gpg sign or checksum his software.
EOA
            user_keyring => <<EOA,
users_keyring if enabled causes fetchware to use the user's own gpg keyring
instead of fetchware's own keyring.
EOA
            mirror => <<EOA
The mirror configuration option provides fetchware with alternate servers to
try to download this program from. This option is used when the server
specified in the url options in this file is unavailable or times out.
EOA
        }
    );
    ###INSANEFEATUREENHANCEMENT### Prompt for name of program, and do a fuzzy 
    #search on CPAN for that program under
    #App::Fetchware::FetchwarefileX::UpCasedProgName. Consider using the meta
    #CPAN API. And if it exists ask user if they wanna use that one instead of
    #autogening one.
    #
    #Perhaps create a 'fetchwarefile' command to download and look at
    #fetchwarefiles from CPAN, and then install them, and/or perhaps upload
    #them pausing to ask for the user's PAUSE credentials!!!!!!!!!


    extension_name(__PACKAGE__);


    my $opening_message = <<EOM;
Fetchware's new command is reasonably sophisticated, and is smart enough to
determine based on the lookup_url you provide if it can autogenerate a
Fetchwarefile for you. If Fetchware cannot, then it will ask you more
questions regarding the information it requires to be able to build a
installable fetchware package for you. After that, fetchware will ask you if
you would like to edit the Fetchwarefile, fetchware has created for you in an
editor. If you say yes, fetchware will open a editor for you, but if you say
no, fetchware will skip the custom editing. Next, fetchware will create a test
Fetchwarefile for you, and ask you if you would like to test it by trying to
install it now. If you say yes, fetchware will install it, and if you say no,
then fetchware will print the location of the Fetchwarefile it created for
you to later use to install your application.
EOM

    opening_message($opening_message);

    # Ask user for name of program unless the user provided one at command
    # line such as fetchware new <programname>.
    $program_name = fetchwarefile_name(program => $program_name);
    vmsg "Determined name of your program to be [$program_name]";

    $fetchwarefile->config_options(program => $program_name);
    vmsg "Appended program [$program_name] configuration option to Fetchwarefile";

    my $lookup_url = get_lookup_url($term);
    vmsg "Asked user for lookup_url [$lookup_url] from user.";

    $fetchwarefile->config_options(lookup_url => $lookup_url);
    vmsg "Appended lookup_url [$lookup_url] configuration option to Fetchwarefile";

    vmsg "Downloaded lookup_url [$lookup_url]";
    my $filename_listing = download_lookup_url($term, $lookup_url);
    vmsg "Downloaded lookup_url's directory listing";
    vmsg Dumper($filename_listing);

    my $mirrors_hashref = get_mirrors($term, $filename_listing);
    vmsg "Added mirrors to your Fetchwarefile.";
    vmsg Dumper($mirrors_hashref);

    my $verify_hashref = get_verification($term, $filename_listing, $lookup_url);
    vmsg "Added verification settings to Fetchwarefile.";
    vmsg Dumper($verify_hashref);

    my $filter_hashref = get_filter_option($term, $filename_listing);
    vmsg "Added [$filter_hashref->{filter}] filter setting to Fetchwarefile.";

    $fetchwarefile->config_options(
        %$mirrors_hashref,
        %$verify_hashref,
        %$filter_hashref
    );

    ###BUGALERT### Ask to parrallelize make with make_options???

lib/App/Fetchware.pm  view on Meta::CPAN

            # if they're defined, which is done below.
        }
    }
    
    
    # Only try sha1 and md5 if gpg failed.
    unless ($verify_configed_flag == 1) {
        if (exists $available_verify_methods{sha1}
                and defined $available_verify_methods{sha1}
                and $available_verify_methods{sha1} > 0
        ) {
            msg <<EOM;
SHA1 checksums found. Using SHA1 verification.
EOM
            $options{verify_method} = 'sha1';
        } elsif (exists $available_verify_methods{md5}
                and defined $available_verify_methods{md5}
                and $available_verify_methods{md5} > 0
        ) {
            msg <<EOM;
MD5 checksums found. Using MD5 verification.
EOM
            $options{verify_method} = 'md5';
        } else {
            # Print a huge long nasty warning even include links to news stories
            # of mirrors actually getting hacked and serving malware, which
            # would be detected and prevented with proper verification enabled.

            # Ask user if they would like to continue installing fetchware even if
            # verification fails, and then enable the verify_failure_ok option.
            if (
                $term->ask_yn(prompt => <<EOP,
Would you like fetchware to ignore the fact that it is unable to verify the
authenticity of any downloads it makes? Are you ok with possibly downloading
viruses, worms, rootkits, or any other malware, and installing it possibly even
as root? 
EOP
                    default => 'n',
                    print_me => <<EOP,
Automatic verification of your fetchware package has failed! Fetchware is
capable of ignoring the error, and installing software packages anyway using its
verify_failure_ok configuration option. However, installing software packages
without verifying that they have not been tampered with could allow hackers to
potentially install malware onto your computer. Don't think this is *not*
possible or do you think its extremely unlikely? Well, it's actually
surprisingly common:
    1.  http://arstechnica.com/security/2012/09/questions-abound-as-malicious-phpmyadmin-backdoor-found-on-sourceforge-site/
    Discusses how a mirror for sourceforge was hacked, and the phpMyAdmin
    software package on that mirror was modified to spread malware.
    2.  http://www.geek.com/news/major-open-source-code-repository-hacked-for-months-says-fsf-551344/
    Discusses how FSF's gnu.org ftp download site was hacked.
    3.  http://arstechnica.com/security/2012/11/malicious-code-added-to-open-source-piwik-following-website-compromise/
    Discusses how Piwiki's wordpress software was hacked, and downloads of
    Piwiki had malicious code inserted into them.
    4. http://www.theregister.co.uk/2011/03/21/php_server_hacked/
    Discusses how php's wiki.php.org server was hacked yielding credentials to
    php's source code repository.
Download mirrors *do* get hacked. Do not make the mistake, and think that it is
not possible. It is possible, and it does happen, so please properly configure
your Fetchwarefile to enable fetchware to verify that the downloaded software is
the same what the author uploaded.
EOP
                )
            ) {
                # If the user is ok with not properly verifying downloads, then
                # ignore the failure, and install anyway.
                $options{verify_failure_ok} = 'On';
            } else {
                # Otherwise, throw an exception.
                die <<EOD;
fetchware: Fetchware *must* be able to verify any software packages that it
downloads. The Fetchwarefile that you were creating could not do this, because
you failed to specify how fetchware can verify its downloads. Please rerun
fetchware new again, and this time be sure to specify a gpg_keys_url, specify
user_keyring to use your own gpg keyring, or answer yes to the question
regarding adding verify_failure_ok to your Fetchwarefile to make failing
verificaton acceptable to fetchware.
EOD
            }
        }
    }

    return \%options;
}



sub get_filter_option {
    my $term = shift;
    # $filename_listing is an array of [$filename, $timestamp] arrays.
    my $filename_listing = shift;
    msg <<EOS;
Analyzing the lookup_url you provided to determine if fetchware can use it to
successfully determine when new versions of your software are released.
EOS

    my $filter;
    if (grep {$_->[0] =~ /^(CURRENT|LATEST)[_-]IS[_-].+/} @$filename_listing) {
        # There is only one version in the lookup_url directory listing, so
        # I do not need a filter option.
        msg <<EOS;
* The lookup_url you gave fetchware includes a CURRENT_IS or a LATEST_IS file
that tells fetchware and regular users what the latest version is. Because of
this we can be reasonable sure that a filter option is not needed, so I'll skip
asking for one. You can provide one later if you need to provide one, when
fetchware prompts you for any custom options you may want to use.
EOS
    } else {
        # There is a CURRENT_IS_<ver_num> or LATEST_IS_<ver_num> file that tells
        # you what the latest version is.
###BUGALERT### Why is this line in both sections of the if statement??? Inside
#this else block means that a CURRENT_IS or LATEST-IS was *not* found??? Fix
#this!!!!!!
        msg <<EOS;
* The directory listing of your lookup_url has a CURRENT_IS_<ver_num> or
LATEST_IS_<ver_num> file that specifies the latest version, which means that
your program's corresponding Fetchwarefile does not need a filter option. If you
still would like to provide one, you can do so later on, when fetchware allows
you to define any additional configuration options.
EOS
        my $what_a_filter_is = <<EOA;

lib/App/Fetchware.pm  view on Meta::CPAN

And then you add a space, and then whatever value you want it to have in quotes.

    program 'Apache';

And don't forget the semicolon C<;> on then end. The semicolon is required

You can use comments as needed to help document you Fetchwarefile like so:

    # Fetchwarefile for Apache.
    program 'Apache';

=over

=item B<1. Name it>

Use your text editor to create a file with a C<.Fetchwarefile> file extension.
Use of this convention is not required, but it makes it obvious what type of
file it is. Then, just copy and paste the example text below, and replace
C<[program]> with what you choose the name of your proram to be.
C<program> is simply a configuration option that simply names your
Fetchwarefile. It is not actually used for anything other than to name your
Fetchwarefile to document what program or behavior this Fetchwarefile manages.

    use App::Fetchware;

    # [program] - explain what [program] does.
    program '[program]';

Fetchwarefiles are actually small, well structured, Perl programs that can
contain arbitrary perl code to customize fetchware's behavior, or, in most
cases, simply specify a number of fetchware or a fetchware extension's
configuration options. Below is my filled in example App::Fetchware
fetchwarefile.

    use App::Fetchware;

    # apache 2.2 - Web Server.
    program 'apache-2.2';

Notice the C<use App::Fetchware;> line at the top. That line is
absolutely critical for this Fetchwarefile to work properly, because it is what
allows fetchware to use Perl's own syntax as a nice easy to use syntax for
Fetchwarefiles. If you do not use the matching C<use App::Fetchware;> line,
then fetchware will spit out crazy errors from Perl's own compiler listing all
of the syntax errors you have. If you ever receive that error, just ensure you
have the correct C<use App::Fetchware;> line at the top of your
Fetchwarefile.

=item B<2. Determine your lookup_url>

At the heart of App::Fetchware is its C<lookup_url>, which is
the URL to the FTP or HTTP mirror you want App::Fetchware to use to obtain a
directory listing to see if a new version of your program is available for
download. To figure this out just use your browser to find the program you
want fetchware to manage for you's Web site. Skip over the download link, and
instead look for the gpg, sha1, or md5 verify links, and copy and paste one of
those between the single quotes above in the lookup_url. Then delete the file
portion--from right to left until you reach a C</>. This is necessary, because
fetchware uses the lookup_url as a basis to download your the gpg, sha1, or md5
digital signatures or checksums to ensure that the packages fetchware downloads
and installs are exactly the same as the ones the author uploads.

    lookup_url '';

And then after you copy the url.

    lookup_url 'http://www.apache.org/dist/httpd/';

=item B<3. Determine your filter configuration option>

The C<filter> option specifies a L<perl regex|perlretut> that is matched against
the list of the files in the directory you specify in your C<lookup_url>. This
sorts through the directory to pick out which program or even which version of
the same program you want this Fetchwarefile to manage.

This is needed, because some programs such as L<Apache|http://httpd.apache.org>
have multiple versions available at the same time, so you would need to specify
which version of apache you want to download. So, you'd specify
C<filter 'httpd-2.2';> to make fetchware download and manage Apache 2.2, or you
could specify C<filter 'httpd-2.4';> to specify the newer 2.4 series version.

This option also exists to allow fetchware to pick out our program if you
specify a mirror directory that has more than one program in it. If you do this,
then fetchware can use C<filter> to pick out the program you want to download
and install from the crowd.

Just write your perl regex in between the single quotes C<'> below. You don't
need to master regular expressions to specify this option. Just specify the name
of the program and/or the main part of the version number. B<Do not> specify the
entire version number or fetchware will never update your program properly.

    filter '';

And then after you type in the text pattern.

    filter 'httpd-2.2';

=item B<4. Add mandatory verification settings>

Verification of software downloads is mandatory, because fetchware, in order to
install the software that is downloaded, must execute the build and installation
scripts on your computer sometimes even as the root administrator! Therefore,
fetchware will refuse to build and install any software package that cannot be
verified. This limitation can be bypassed by setting the C<verify_failure_ok>
configuration option to true, but this is B<not> recommended.

Instead, if standard verification fails, please set up one or more of the
configuration options below that may allow verification to succeed if the author
has his download site set up differently then fetchware expects.

=over

=item gpg_keys_url - Should list a URL to a file most likely named C<KEYS> that
contains versions of the author's gpg verification keys that is suitable to be
imported into gpg using C<gpg --import [name of file]>. An example would be:

    gpg_keys_url 'http://www.apache.org/dist/httpd/KEYS';

=item users_keyring - Tells fetchware to use the user who calls fetchware's gpg
keyring instead of fetchware's own keyring. This is handy for when you want to
install a program, but the author has no easily accessible C<KEYS> file, but the

lib/App/Fetchware.pm  view on Meta::CPAN

the section L<MANUALLY CREATING A App::Fetchware FETCHWAREFILE>. All of them are
detailed below.

=head2 program 'Program Name';

C<program> simply gives this Fetchwarefile a name. It is availabe to fetchware
after parsing your Fetchwarefile, and is used to name your Fetchwarefile when
using C<fetchware new>. It is required just like C<lookup_url>, C<mirror>,
perhaps C<filter>, and some method to verify downloads are.

=head2 filter 'perl regex here';

Specifies a Perl regular expression that fetchware uses when it determines what
the latest version of a program is. It simply compares each file in the
directory listing specified in your C<lookup_url> to this regular expression,
and only matching files are allowed to pass through to the next part of
fetchware that looks for source code archives to download.

See L<perlretut> for details on how to use and create Perl regular expressions;
however, actual regex know how is not really needed just paste verbatim text
between the single quotes C<'>. For example, C<filter 'httpd-2.2';> will cause
fetchware to only download Apache 2.2 instead of the version for Windows or
whatever is in the weird httpd-deps-* package.

=head2 temp_dir '/tmp';

C<temp_dir> tells fetchware where to store fetchware's temporary working
directory that it uses to download, verify, unarchive, build, and install your
software. By default it uses your system temp directory, which is whatever
directory L<File::Temp's> tempdir() decides to use, which is whatever
L<File::Spec>'s tmpdir() decides to use.

=head2 fetchware_db_path '~/.fetchwaredb';

C<fetchware_db_path> tells fetchware to use a different directory other
than its default directory to store the installed fetchware package for the
particular fetchware package that this option is specified in your
Fetchwarefile. Fetchware's default is C</var/log/fetchware> on Unix when run as
root, and something like C</home/[username]/.local/share/Perl/dist/fetchware/>
when run nonroot.

This option is B<not> recommended unless you only want to change it for just one
fetchware package, because fetchware also consults the
C<FETCHWARE_DATABASE_PATH> environment variable that you should set in your
shell startup files if you want to change this globally for all of your
fetchware packages. For sh/bash like shells use:

    export FETCHWARE_DATABASE_PATH='/your/path/here'

=head2 user 'nobody';

Tells fetchware what user it should drop privileges to. The default is
C<nobody>, but you can specify a different username with this configuration
option if you would like to.

Dropping privileges allows fetchware to avoid downloading files and executing
anything inside the downloaded archive as root. Except of course the commands
needed to install the software, which will still need root to able to write
to system directories. This improves security, because the downloaded software
won't have sytem privileges until after it is verified, providing that what you
downloaded is exactly what the author uploaded.

Note this only works for unix like systems, and is not used on Windows and
other non-unix systems.

Also note, that if you are running fetchware on Unix even if you do not specify
the C<user> configuration option to configure what user you will drop privileges
to, fetchware will still drop privileges using the ubiquitous C<nobody> user.
If you do B<not> want to drop privileges, then you must use the C<stay_root>
configuration option as described below.

=head2 stay_root 'On';

Tells fetchware to B<not> drop privileges. Dropping privileges when run as root
is fetchware's default behavior. It improves security, and allows fetchware to
avoid exposing the root account by downloading files as root.

Do B<not> use this feature unless you are absolutely sure you need it.

=over

=item SECURITY NOTICE

stay_root, when turned on, causes fetchware to not drop privileges when
fetchware looks up, downloads, verifies, and builds your program. Instead,
fetchware will stay root through the entire build cycle, which needlessly
exposes the root account when downloading files from the internet. These files
may come from trusted mirrors, but mirrors can, and do get cracked:

L<http://www.itworld.com/security/322169/piwik-software-installer-rigged-back-door-following-website-compromise?page=0,0>

L<http://www.networkworld.com/news/2012/092612-compromised-sourceforge-mirror-distributes-backdoored-262815.html>

L<http://www.csoonline.com/article/685037/wordpress-warns-server-admins-of-trojans>

L<http://www.computerworld.com/s/article/9233822/Hackers_break_into_two_FreeBSD_Project_servers_using_stolen_SSH_keys>

=back

=head2 lookup_url 'ftp://somedomain.com/some/path

This configuration option specifies a url of a FTP or HTTP or local (file://)
directory listing that fetchware can download, and use to determine what actual
file to download and perhaps also what version of that program to download if
more than one version is available as some mirrors delete old versions and only
keep the latest one.

This url is used for:

=over

=item 1. To determine what the actual download url is for the latest version of this program

=item 2. As the base url to also download a cryptographic signature (ends in .asc) or a SHA-1 or MD5 signature to verify the contents match what the SHA-1 or MD5 checksum is.

=back

You can use the C<mirror> configuration option to specify additional mirrors.
However, those mirrors will only be used to download the large software
archives. Only the lookup_url will be used to download directory listings to
check for new versions, and to download digital signatures or checksums to

lib/App/Fetchware.pm  view on Meta::CPAN

=item LIMITAITON

C<user_keyring> when set to true requires that the user that fetchware is
running under have a real gpg keyring with keys that have been imported into it.
This is not the case B<unless> the C<user> option has been specified with a user
account with a proper home directory and gpg keyring for gpg to use. Because of
this limitation if you need to specify C<user_keyring> be sure to also specify
the C<user> option to specify a I<real> user account instead of the default fake
one C<nobody>.

Typically you would import the keys into your own user accounts gpg keyring, and
then you would specify your own username with the C<user> option to tell
fetchware to drop privs to your own user account to have access to your own gpg
keys.

=back

=head2 gpg_sig_url 'mirror.com/some/path';

Specifies an alternate url to use to download the cryptographic signature that
goes with your program. This is usually a file with the same name as the
download url with a C<.asc> file extension added on. Fetchware will also append
the extensions C<sig> and C<sign> if C<.asc> is not found, because some pgp
programs and authors use these extensions too.

=head2 sha1_url 'mastermirror.com/some/path';

Specifies an alternate url to download the SHA-1 checksum. This checksum is used
to verify the integrity of the archive that fetchware downloads.

You B<must> specify the master mirror site, which is your programs main mirror
site, because if you download it from a mirror, its possible that both the
archive and the checksum could have been tampered with.

=head2 md5_url 'mastermirror.com/some/path';

Specifies an alternate url to download the MD5 checksum. This checksum is used
to verify the integrity of the archive that fetchware downloads.

You B<must> specify the master mirror site, which is your programs main mirror
site, because if you download it from a mirror, its possible that both the
archive and the checksum could have been tampered with.

=head2 user_agent 'Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0'

Specifies what C<user_agent> you would like fetchware to pretend to be when
downloading files using the HTTP protocol. Some sites annoying prevent some
user agents from working while allowing others. This allows you to pretend to
be a real browser such as Firefox if you need to.

=head2 verify_method 'gpg';

Chooses a method to verify your program. The default is to try C<gpg>, then
C<sha1>, and finally C<md5>, and if all three fail, then the default is to exit
fetchware with an error message, because it is insecure to install archives that
cannot be verified. The availabel options are:

=over

=item gpg - Uses the gpg program to cryptographically verify that the program you downloaded is exactly the same as its author uploaded it.

=item sha1 - Uses the SHA-1 hash function to verify the integrity of the download. This is much less secure than gpg.

=item md5 - Uses the MD5 hash function to verify the integrity of the download. This is much less secure than gpg.

=back

=head2 verify_failure_ok 'True';

Fetchware's default regarding failing to verify your downloaded Archive with
gpg, sha1, or md5 is to exit with an error message, because installing software
that cannot be cryptographically verified should never be done.

=over

=item SECURITY NOTICE

However, if the author of a program you want to use fetchware to manage for you
does not offer a gpg, sha1, or md5 file to verify its integrity, then you can
use this option to force Fetchware to install this program anyway. However, do
not enable this option lightly. Please scour the program's mirrors and homepage
to see which C<gpg_keys_url>, C<gpg_sig_url>, C<sha1_url>, C<md5_url>, or
C<user_keyring> you can use to ensure that your archive is verified before it is
compiled and installed. Even mirrors from sites large and small get hacked
regularly:

L<http://www.itworld.com/security/322169/piwik-software-installer-rigged-back-door-following-website-compromise?page=0,0>

L<http://www.networkworld.com/news/2012/092612-compromised-sourceforge-mirror-distributes-backdoored-262815.html>

L<http://www.csoonline.com/article/685037/wordpress-warns-server-admins-of-trojans>

L<http://www.computerworld.com/s/article/9233822/Hackers_break_into_two_FreeBSD_Project_servers_using_stolen_SSH_keys>

So, Please give searching for a C<gpg_keys_url>, C<gpg_sig_url>, C<sha1_url>,
C<md5_url>, or C<user_keyring> for your program another try before simply
enabling this option.

=back

=over

=item NOTICE

C<verify_failure_ok> is a boolean configuration option, which just means its
values are limited to either true or false. True values are C<'True'>, C<'On'>,
C<1>, and false values are C<'False'>, C<'Off'>, and C<0>. All other values are
syntax errors.

=back

=head2 prefix '/opt/';

Controls the AutoTools C<./configuration --prefix=...> option, which allows you
to change the base directory that most software (software that uses AutoTools)
uses as the base directory for when they install themselves.

For example, most programs copy binaries to C<prefix/bin>, documentation to
C<prefix/docs>, manpages to C<prefix/man>, and so on.

lib/App/Fetchware.pm  view on Meta::CPAN

C<make uninstall>. Many source code distributions do not provide a C<uninstall>
make target, so they can not easily be uninstalled by fetchware without such
support. In these cases, you could look into
L<paco|http://paco.sourceforge.net/>, L<src2pkg|http://www.src2pkg.net/>, or
L<fpm|https://github.com/jordansissel/fpm>. These all aid turning a source code
distribution into your operating system's package format, or somehow magically
monitoring C<make install> to track what files are installed where, and then
using this information to be able to uninstall them.

=head2 no_install 'On';

This boolean, see below, configuration option determines if fetchware should
install your software or not install your software, but instead prints out the
path of its build directory, so that you can QA test or review the software before
you install it.

=over

=item NOTICE

C<no_install> is a boolean configuration option, which just means its
values are limited to either true or false. True values are C<'True'>, C<'On'>,
C<1>, and false values are C<'False'>, C<'Off'>, and C<0>. All other values are
syntax errors.

=back

=head2 mirror 'somemirror0.com/some/optional/path';

Your Fetchwarefile needs to have at least one mirror specified. Although you can
specify as many as you want to.

This configuration option, unlike all the others, can be specified more than
once. So, for example you could put:

    mirror 'somemirror1.com';
    mirror 'somemirror2.com';
    mirror 'somemirror3.com';
    mirror 'somemirror4.com';
    mirror 'somemirror5.com';

When fetchware downloads files or directories it will try each one of these
mirrors in order, and only fail if all attempts at all mirrors fail.

If you specify a path in addition to just the hostname, then fetchware will try
to get whatever it wants to download at that alternate path as well.

    mirror 'somemirror6./com/alternate/path';

=head1 FURTHER CUSTOMIZING YOUR FETCHWAREFILE

Because fetchware's configuration files, its Fetchwarefiles, are little Perl
programs, you have the full power of Perl at your disposal to customize
fetchware's behavior to match what you need fetchware to do to install your
source code distributions.

Not only can you use arbitrary Perl code in your Fetchwarefile to customize
fetchware for programs that don't follow most FOSS mirroring's unwritten
standards or use a totally different build system, you can also create a
fetchware extension. Creating a fetchware extension even allows you to turn your
extension into a proper CPAN distribution, and upload it to CPAN to share it
with everybody else. See the section below,
L<CREATING A FETCHWARE EXTENSION>, for full details.

=head2 How Fetchware's configuration options are made

Each configuration option is created with L<App::Fetchware::CreateConfigOptions>
This package's import() is a simple code generator that generates configuration
subroutines.  These subroutines have the same names as fetchware's configuration
options, because that is exactly what they are. Perl's
L<Prototypes|perlsub/Prototypes> are used in the code that is generated, so
that you can remove the parentheses typically required around each configuration
subroutine. This turns what looks like a function call into what could
believably be considered a configuration file syntax.

These prototypes turn:

    lookup_url('http://somemirror.com/some/path');

Into:

    lookup_url 'http://somemirror.com/some/path';

Perl's prototypes are not perfect. The single quotes and semicolon are still
required, but the lack of parens instantly makes it look much more like a
configuration file syntax, then an actual programming language.

=head2 The magic of C<use App::Fetchware;>

The real magic power behind turning a Perl program into a configuration file
sytax comes from the C<use App::Fetchware;> line. This line is single handedly
responsible for making this work. This one line imports all of the configuration
subroutines that make up fetchware's configuration file syntax. And this
mechanism is also behind fetchware's extension mechanism. (To use a
App::Fetchware extension, you just C<use> it. Like
C<use App::FetchwareX::HTMLPageSync;>. That's all there is to it. This I<other>
App::Fetchware is responsible for exporting subroutines of the same names as
those that make up App::Fetchware's API. These subroutines are listed in the
section L<FETCHWAREFILE API SUBROUTINES> as well as their helper subroutines.
See the section below L<CREATING A FETCHWARE EXTENSION> for more information on
how to create App::Fetchware extensions.

=head2 So how do I add some custom Perl code to customize my Fetchwarefile?

You use hook() to override one of fetchware's API subroutines. Then when
fetchware goes to call that subroutine, your own subroutine is called in its
place. You can hook() as many of fetchware's API subroutines as you need to.

=over

Remember your replackement subroutine B<must> take the exact same arguments, and
return the same outputs that the standard fetchware API subroutines do!

All of the things these subroutines return are later used as parameters to later
API subroutines, so failing to return a correct value may cause fetchware to
fail.

=back

=head3 hook()

lib/App/Fetchware.pm  view on Meta::CPAN

            export FETCHWARE_LOCAL_BUILD_URL='/home/dly/software/ctags-5.8.tar.gz'
            export FETCHWARE_LOCAL_UPGRADE_URL='file:///home/dly/software/fetchware-upgrade'
            export FETCHWARE_NONROOT_USER='YOURUSERNAME'
            echo 'done.'
        else
            echo -n 'Deleting fetchware_release_testing environment variables...';
            unset FETCHWARE_RELEASE_TESTING
            unset FETCHWARE_FTP_LOOKUP_URL
            unset FETCHWARE_HTTP_LOOKUP_URL
            unset FETCHWARE_FTP_MIRROR_URL
            unset FETCHWARE_HTTP_MIRROR_URL
            unset FETCHWARE_FTP_DOWNLOAD_URL
            unset FETCHWARE_HTTP_DOWNLOAD_URL
            unset FETCHWARE_LOCAL_URL
            unset FETCHWARE_LOCAL_ZIP_URL
            unset FETCHWARE_LOCAL_BUILD_URL
            unset FETCHWARE_LOCAL_UPGRADE_URL
            unset FETCHWARE_NONROOT_USER
            echo 'done.'
        fi
    }

Just run C<frt> with no args to turn FETCHWARE_RELEASE_TESTING on, and run it
once more to turn it off. Don't forget to reload your shell's configuration
with:

    $ . ~/.bashrc # Or whatever file you added it to is named.

Then inside your test suite just import skip_all_unless_release_testing() from
Test::Fetchware:

    use Test::Fetchware ':TESTING';

=item 2. Call skip_all_unless_release_testing() as needed

In each subtest where you I<actually> install anything other than a test
distribution made with make_test_dist(), begin that subtest with a call to
skip_all_unless_release_testing(), which will skip the whole subtest if
FETCHWARE_RELEASE_TESTING is not setup properly.

    subtest 'test install() for real' => sub {
        # Only test if during release testing.
        skip_all_unless_release_testing();
        
        # Tests that actually install non-trivial distributions such as ones
        # made by make_test_dist() go here.

        ....
    };

If you dislike subtests, or otherwise don't want to use them, then put all of
the tests that I<actually> install something into a SKIP block after the other
tests that all users will run at the end of your test file.

=back

=head2 Share it on CPAN

Fetchware has no Web site or any other place to share fetchware extensions. But
fetchware is written in Perl, so fetchware can just use Perl's CPAN. To learn
how to create modules and upload them to CPAN please see Perl's own
documentation. L<perlnewmod> shows how to create new Perl modules, and how to
upload them to CPAN. See L<Module::Starter> for a simple way to create a
skeleton for a new Perl module, and L<dzil|http://dzil.org/index.html> is beyond
amazing, but has insane dependencies and a significant learning curve.

=head2 Fetchware Extension Best Practices

Below are a few points to keep in mind when creating a Fetchware extension that
the documentation itself fails to mention prominently or that otherwise slip
through the cracks, but are still important to keep in mind while designing and
implementing a Fetchware extension.

=over

=item * Actually implement the new() API subroutine to help your Fetchware
extension's users create new Fetchwarefiles for your Fetchware extension. And
use L<App::Fetchware::Fetchwarefile> to help you generate the Fetchwarefile.

=item * Only use L<run_prog()|App::Fetchware::Util/run_prog()> to execute
external programs, because it supports Fetchware's verbose (-v) and quiet (-q)
options. If you really do need something else, then checkout run_prog()'s
documentation for how to see if verbose or quiet options are enabled.

=item * Use L<App::Fetchware::Util>'s logging subroutines msg() and vmsg() to
print messages out to your user to tell them what you are doing. I recommened a
msg() or two in each or of your main API subroutines, and then at least one
vmsg() in each and any helper subroutines you may write and use.

=back

=head1 FETCHWAREFILE API SUBROUTINES

The subroutines below B<are> Fetchwarefile's API subroutines or helper
subroutines for App::Fetchware's API subroutines. If you want information on
fetchware's configuration file syntax, then see the section
L<FETCHWAREFILE CONFIGURATION SYNTAX> for more information. For additional
information on how to keep or override these API subroutines in a fetchware
extension see the section L<CREATING A FETCHWARE EXTENSION>

=head2 new()

    my ($program_name, $fetchwarefile) = new($term, $program_name);

    # Or in an extension, you can return whatever list of variables you want,
    # and then cmd_new() will provide them as arguments to new_install() except
    # a $term Term::ReadLine object will precede the others.
    my ($term, $program_name, $fetchwarefile, $custom_argument1, $custom_argument2)
        = new($term, $program_name);

new() is App::Fetchware's API subroutine that implements fetchware's new
command. new() calls a bunch of helper subroutines that implement
the algorithm fetchware uses to build new Fetchwarefiles automagically for the
user. The algorithm is dead stupid:

=over

=item 1. Ask for lookup_url & download it.

=item 2. Analyze the contents of the output from the lookup_url.

=item 3. Build the Fetchwarefile according to the output.

lib/App/Fetchware.pm  view on Meta::CPAN

=item md5_url

=item verify_method

=item verify_failure_ok

=item user_agent

=back

=back

Verifies the downloaded package stored in $package_path by downloading
$download_path.{asc,sha1,md5}> and comparing the two together. Uses the
helper subroutines C<{gpg,sha1,md5,digest}_verify()>.

=over

=item LIMITATIONS
Uses gpg command line, and the interface to gpg is a little brittle.
Crypt::OpenPGP is buggy and not currently maintainted again, so fetchware cannot
make use of it, so were stuck with using the command line gpg program.

=back

=over

=item drop_privs() NOTES

This section notes whatever problems you might come accross implementing and
debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See L<Util's drop_privs() subroutine for more info|App::Fetchware::Util/drop_privs()>.

=over

=item *

Under drop_privs() verify() is executed in the child with reduced privileges.

=back

=back

=head2 verify() API REFERENCE

The subroutines below are used by verify() to provide the verify
functionality for fetchware. If you have overridden the verify() handler, you
may want to use some of these subroutines so that you don't have to copy and
paste anything from verify().

App::Fetchware is B<not> object-oriented; therefore, you B<can not> subclass
App::Fetchware to extend it! 

=head3 gpg_verify()

    'Package Verified' = gpg_verify($download_path);

Uses the command-line program C<gpg> to cryptographically verify that the file
you download is the same as the file the author uploaded. It uses public-key
priviate-key cryptography. The author signs his software package using gpg or
some other OpenPGP compliant program creating a digital signature file with the
same filename as the software package, but usually with a C<.asc> file name
extension. gpg_verify() downloads the author's keys, imports them into
fetchware's own keyring unless the user sets C<user_keyring> to true in his
Fetchwarefile. Then Fetchware downloads a digital signature that usually
ends in C<.asc>. Afterwards, fetchware uses the gpg command line program to
verify the digital signature. gpg_verify returns true if successful, and throws
an exception otherwise.

You can use C<gpg_keys_url> to specify the URL of a file where the author has
uploaded his keys. And the C<gpg_sig_url> can be used to setup an alternative
location of where the C<.asc> digital signature is stored.

=head3 sha1_verify()

    'Package verified' = sha1_verify($download_path, $package_path);
    undef = sha1_verify($download_path, $package_path);

Verifies the downloaded software archive's integrity using the SHA Digest
specified by the C<sha_url 'ftp://sha.url/package.sha'> config option. Returns
true for sucess dies on error.

=over

=item SECURITY NOTE
If an attacker cracks a mirror and modifies a software package, they can also
modify the MD5 sum of that software package on that B<same mirror>. Because of
this limitation MD5 sums can only tell you if the software package was corrupted
while downloading. This can actually happen as I've had it happen to me once.

If your stuck with using MD5 sum, because your software package does not provide
gpg signing, I recommend that you download your SHA1 sums (and MD5 sums) from
your software package's master mirror. For example, Apache provides MD5 and SHA1
sums, but it does not mirror them--you must download them directly from Apache's
servers. To do this specify a C<sha1_url 'master.mirror/package.sha1';> in your
Fetchwarefile.

=back

=head3 md5_verify()

    'Package verified' = md5_verify($download_path, $package_path);
    undef = md5_verify($download_path, $package_path);

Verifies the downloaded software archive's integrity using the MD5 Digest
specified by the C<md5_url 'ftp://sha.url/package.sha'> config option. Returns
true for sucess and dies on error.

=over

=item SECURITY NOTE
If an attacker cracks a mirror and modifies a software package, they can also
modify the MD5 sum of that software package on that B<same mirror>. Because of
this limitation MD5 sums can only tell you if the software package was corrupted
while downloading. This can actually happen as I've had it happen to me once.

If your stuck with using MD5 sum, because your software package does not provide
gpg signing, I recommend that you download your MD5 sums (and SHA1 sums) from
your software package's master mirror. For example, Apache provides MD5 and SHA1
sums, but it does not mirror them--you must download them directly from Apache's
servers. To do this specify a C<md5_url 'master.mirror/package.md5';> in your
Fetchwarefile.

=back

=head3 digest_verify()

    'Package verified' = digest_verify($digest_type, $download_path, $package_path);
    undef = digest_verify($digest_type, $download_path, $package_path);

Verifies the downloaded software archive's integrity using the specified