App-Fetchware
view release on metacpan or search on metacpan
INTRODUCTION
Fetchware is a command line program written in Perl with help from CPAN that
brings package management to source code distributions. It is able to do this,
because most source code distributions use GNU autoconf, and thereby use the
exact same commands to build and install. Furthermore, everyone uses md5sums,
shasums, and/or gpg sigs, and ftp/http mirrors. Because everyone follows the
a similar format for their FOSS means a fairly simple but flexible command line
program should be able provide package management for source code distributions.
Fetchware is a command line perl program that supports install, uninstall (via
make uninstall), upgrade (for just one package), upgrade-all (for all installed
packages), and the best part a new command that is a console wizard question and
answer interface that easily builds brand new Fetchwarefiles and packages based
on fairly obvious information (mirrors, name, configure options, make options,
and so on.). If you have the need for fetchware (You already know how to install
source code distributions yourself.), then you should be able to figure out how
bin/fetchware view on Meta::CPAN
###BUGALERT### install installs no matter if the program is already
#installed!!! Change this to parse out the package from the
#download_urlif possible, compare with the one in the fetchware
#package database, and call exit right here if the current version
#is already installed unless of course --force is used!!!
my $download_url = lookup();
my $package_path = download($temp_dir, $download_url);
###BUGALERT### Add support for caching the key files gpg creates to
#the fetchwarefile, and for actually using them later on inside the
#fpkg.
verify($download_url, $package_path);
$P_build_path = unarchive($package_path);
build($P_build_path);
# Tell the parent, root, process the values of the variables the
# child calculated in this coderef, and write them across this pipe
bin/fetchware view on Meta::CPAN
# Call upgrade() to determine if the currently available version
# ($P_download_path) is newer than the currenlty installed version
# ($P_fetchware_package_path).
my $P_upgrade = upgrade($P_download_path, $P_fetchware_package_path);
if ($P_upgrade) {
msg 'New version available upgrading now.';
my $package_path = download($temp_dir, $P_download_path);
###BUGALERT### Add support for caching the key files gpg creates to the
#fetchwarefile, and for actually using them later on inside the fpkg.
verify($P_download_path, $package_path);
$P_build_path = unarchive($package_path);
build($P_build_path);
} else {
# If a new version is not available, then the child should do
# nothing, and let the parent call end() to clean up below.
# Set $P_build_path to something that will fail, and give a decent
bin/fetchware view on Meta::CPAN
my $output = drop_privs(
sub {
my $write_pipe = shift;
msg 'Downloading and unarchiving specified distribution.';
my $download_url = lookup();
my $package_path = download(cwd(), $download_url);
###BUGALERT### Add support for caching the key files gpg creates to
#the fetchwarefile, and for actually using them later on inside the
#fpkg.
verify($download_url, $package_path);
my $build_path = unarchive($package_path);
# end() is *not* run, because the point of look is to lookup,
# download, and unarchive, and then actually "look" at the files,
# and running end() would delete them.
lib/App/Fetchware.pm view on Meta::CPAN
fetchware_db_path
user
prefix
configure_options
make_options
build_commands
install_commands
uninstall_commands
lookup_url
lookup_method
gpg_keys_url
gpg_sig_url
sha1_url
md5_url
user_agent
verify_method
no_install
verify_failure_ok
user_keyring
stay_root
mirror
config
lib/App/Fetchware.pm view on Meta::CPAN
http_parse_filelist
file_parse_filelist
lookup_by_timestamp
lookup_by_versionstring
lookup_determine_downloadpath
)],
OVERRIDE_DOWNLOAD => [qw(
determine_package_path
)],
OVERRIDE_VERIFY => [qw(
gpg_verify
sha1_verify
md5_verify
digest_verify
)],
OVERRIDE_UNARCHIVE => [qw(
check_archive_files
list_files
list_files_tar
list_files_zip
unarchive_package
lib/App/Fetchware.pm view on Meta::CPAN
our @EXPORT_OK = @{$EXPORT_TAGS{OVERRIDE_ALL}};
###BUGALERT### Add strict argument checking to App::Fetchware's API subroutines
#to check for not being called correctly to aid extension debugging.
###BUGALERT### Recommend installing http://gpg4win.org if you use fetchware on
# Windows so you have gpg support.
# _make_config_sub() is an internal subroutine that only App::Fetchware and
# App::Fetchware::CreateConfigOptions should use. Use
# App::Fetchware::CreateConfigOptions to create any configuration option
# subroutines that you want your fetchware extensions to have.
lib/App/Fetchware.pm view on Meta::CPAN
[ fetchware_db_path => 'ONE' ],
[ user => 'ONE' ],
[ prefix => 'ONE' ],
[ configure_options=> 'ONEARRREF' ],
[ make_options => 'ONEARRREF' ],
[ build_commands => 'ONEARRREF' ],
[ install_commands => 'ONEARRREF' ],
[ uninstall_commands => 'ONEARRREF' ],
[ lookup_url => 'ONE' ],
[ lookup_method => 'ONE' ],
[ gpg_keys_url => 'ONE' ],
[ gpg_sig_url => 'ONE' ],
[ sha1_url => 'ONE' ],
[ md5_url => 'ONE' ],
[ user_agent => 'ONE' ],
[ verify_method => 'ONE' ],
[ mirror => 'MANY' ],
[ no_install => 'BOOLEAN' ],
[ verify_failure_ok => 'BOOLEAN' ],
[ stay_root => 'BOOLEAN' ],
[ user_keyring => 'BOOLEAN' ],
);
lib/App/Fetchware.pm view on Meta::CPAN
versions of your program are available. It should point to a directory listing
instead of a specific file.
EOA
lookup_method => <<EOA,
lookup_method specifies how fetchware determines what version of your program
to install. The default is the 'timestamp' algorithm, and then to try the
'versionstring' algorithm if 'timestamp' fails. lookup_method specifies which
one you would like to use. Only the strings 'timestamp' and 'versionstring'
are allowed options.
EOA
gpg_keys_url => <<EOA,
gpg_keys_url specifies the url that fetchware will use to download the author's
KEYS file that it uses for gpg verification.
EOA
gpg_sig_url => <<EOA,
gpg_sig_url specifies the url that fetchware uses to download digital
signatures of this program. They're files that usually end .asc.
EOA
sha1_url => <<EOA,
sha1_url specfies the url that fetchware uses to download sha1sum files of
this program. This url should be the program's main download site instead of a
mirror, because a hacked mirror could alter the sha1sum on that mirror.
EOA
md5_url => <<EOA,
md5_url specfies the url that fetchware uses to download md5sum files of
this program. This url should be the program's main download site instead of a
mirror, because a hacked mirror could alter the md5sum on that mirror.
EOA
verify_method => <<EOA,
verify_method specifes a specific method that fetchware should use to verify
your program. This method can be 'gpg', 'sha1', or 'md5'.
EOA
no_install => <<EOA,
no_install specifies that this software should not be installed. Instead, the
install step is skipped, and fetchware prints to STDOUT where it downloaded,
verified, and built your program. no_install must be a true or false value.
EOA
verify_failure_ok => <<EOA,
verify_failure_ok specifies that fetchware should not stop installing your
software and terminate with an error message if fetchware fails to verify your
software. You should never set this to true. Doing so could cause fetchware to
install software that may have been compromised, or had malware inserted into
it. Never use this option unless the author or maintainer of this program does
not gpg sign or checksum his software.
EOA
user_keyring => <<EOA,
users_keyring if enabled causes fetchware to use the user's own gpg keyring
instead of fetchware's own keyring.
EOA
mirror => <<EOA
The mirror configuration option provides fetchware with alternate servers to
try to download this program from. This option is used when the server
specified in the url options in this file is unavailable or times out.
EOA
}
);
###INSANEFEATUREENHANCEMENT### Prompt for name of program, and do a fuzzy
lib/App/Fetchware.pm view on Meta::CPAN
lookup_method specifies what how fetchware determines if a new version of your
software package is available. The available algorithms are 'timstamp' and
'versionstring'. 'timestamp' uses the timestamp listed in the FTP or HTTP
listing, and uses the software package that is the newest by filesystem
timestamp. The 'versionstring' algorithm uses the filename of the files in the
FTP or HTTP listing. It parses out the version information, sorts it highest to
lowest, and then picks the highest version of your software package. The default
is try 'timestamp' and if that doesn't work, then try 'versionstring'.
EOP
},
gpg_keys_url => {
prompt => <<EOP,
What gpg_keys_url configuration option would you like?
EOP
print_me => <<EOP
gpg_keys_url specifies a url similar to lookup_url in that it should specify a
directory instead a specific file. It is used to download KEYS files, which
contain your program author's gpg keys to import into gpg.
EOP
},
gpg_sig_url => {
prompt => <<EOP,
What gpg_sig_url configuration option would you like?
EOP
print_me => <<EOP
gpg_sig_url specifies a url similar to lookup_url in that it should specify a
directory instead a specific file. It is used to download gpg signatures to
verify your software package.
EOP
},
sha1_url => {
prompt => <<EOP,
What sha1_url configuration option would you like?
EOP
print_me => <<EOP
sha1_url specifies a url similar to lookup_url in that it should specify a
directory instead of a specific file. It is separate from lookup_url, because
lib/App/Fetchware.pm view on Meta::CPAN
if the mirror gets hacked.
EOP
},
verify_method => {
prompt => <<EOP,
What verify_method configuration option would you like?
EOP
print_me => <<EOP,
verify_method specifies what method of verification fetchware should use to
ensure the software you have downloaded has not been tampered with. The default
is to try gpg verification, then sha1, and then finally md5, and if they all
fail an error message is printed and fetchware exits, because if your software
package cannot be verified, then it should not be installed. This configuration
option allows you to remove the warnings by specifying a specific way of
verifying your software has not been tampered with. To disable verification set
the 'verify_failure_ok' configuration option to true.
EOP
},
###BUGALERT### replace no_install config su with a command line option that
#would be the opposite of --force???
# Nah! Leave it! Just create a command line option for it too!
lib/App/Fetchware.pm view on Meta::CPAN
option, because it makes it possible for fetchware to install source code that
may have been tampered with.
EOP
},
users_keyring => {
prompt => <<EOP,
Would you like to enable users_keyring configuration option?
EOP
print_me => <<EOP
users_keyring when enabled causes fetchware to use the user who calls
fetchware's gpg keyring instead of fetchware's own gpg keyring. Useful for
source code distributions that do not provide an easily accessible KEYS file.
Just remember to import the author's keys into your gpg keyring with gpg
--import.
EOP
},
);
vmsg 'User entered the following options.';
vmsg Dumper($other_options_hashref);
# Append all other options to the Fetchwarefile.
$fetchwarefile->config_options(%$other_options_hashref);
vmsg 'Appended all other options listed above to Fetchwarefile.';
lib/App/Fetchware.pm view on Meta::CPAN
sub get_verification {
my ($term, $filename_listing, $lookup_url) = @_;
my %options;
my %available_verify_methods;
# Determine what types of verification are available.
for my $file_and_timestamp (@$filename_listing) {
if ($file_and_timestamp->[0] =~ /\.(asc|sig|sign)$/) {
$available_verify_methods{gpg}++;
} elsif ($file_and_timestamp->[0] =~ /\.sha1?$/) {
$available_verify_methods{sha1}++;
} elsif ($file_and_timestamp->[0] =~ /\.md5$/) {
$available_verify_methods{md5}++;
}
}
my $verify_configed_flag = 0;
#If gpg is available prefer it over the others.
if (exists $available_verify_methods{gpg}
and defined $available_verify_methods{gpg}
and $available_verify_methods{gpg} > 0
) {
msg <<EOM;
gpg digital signatures found. Using gpg verification.
EOM
$options{verify_method} = 'gpg';
# Search for a KEYS file to use to import the author's keys.
if (grep {$_->[0] eq 'KEYS'} @$filename_listing) {
msg <<EOM;
KEYS file found using lookup_url. Adding gpg_keys_url to your Fetchwarefile.
EOM
# Add 'KEYS' or '/KEYS' to $lookup_url's path.
my ($scheme, $auth, $path, $query, $fragment) =
uri_split($lookup_url);
$path = catfile($path, 'KEYS');
$lookup_url = uri_join($scheme, $auth, $path, $query, $fragment);
$options{gpg_keys_url} = $lookup_url;
$verify_configed_flag++;
} else {
msg <<EOM;
KEYS file *not* found!
EOM
# Since autoconfiguration of KEYS failed, try asking the user if
# they would like to import the author's key themselves into their
# own keyring and have fetchware use that.
if (
$term->ask_yn(prompt =>
q{Would you like to import the author's key yourself after fetchware completes? },
default => 'n',
print_me => <<EOP,
Automatic KEYS file discovery failed. Fetchware needs the author's keys to
download and import into its own keyring, or you may specify the option
user_keyring, which if true will cause fetchware to use the user who runs
fetchware's keyring instead of fetchware's own keyring. But you, the user, needs
to import the author's keys into your own gpg keyring. You can do this now in a
separate shell, or after you finish configuring this Fetchwarefile. Just run the
command [gpg --import <name of file>].
EOP
)
) {
$options{user_keyring} = 'On';
$verify_configed_flag++;
}
# And if the user does not want to, then fallback to sha1 and/or md5
# if they're defined, which is done below.
}
}
# Only try sha1 and md5 if gpg failed.
unless ($verify_configed_flag == 1) {
if (exists $available_verify_methods{sha1}
and defined $available_verify_methods{sha1}
and $available_verify_methods{sha1} > 0
) {
msg <<EOM;
SHA1 checksums found. Using SHA1 verification.
EOM
$options{verify_method} = 'sha1';
} elsif (exists $available_verify_methods{md5}
lib/App/Fetchware.pm view on Meta::CPAN
) {
# If the user is ok with not properly verifying downloads, then
# ignore the failure, and install anyway.
$options{verify_failure_ok} = 'On';
} else {
# Otherwise, throw an exception.
die <<EOD;
fetchware: Fetchware *must* be able to verify any software packages that it
downloads. The Fetchwarefile that you were creating could not do this, because
you failed to specify how fetchware can verify its downloads. Please rerun
fetchware new again, and this time be sure to specify a gpg_keys_url, specify
user_keyring to use your own gpg keyring, or answer yes to the question
regarding adding verify_failure_ok to your Fetchwarefile to make failing
verificaton acceptable to fetchware.
EOD
}
}
}
return \%options;
}
lib/App/Fetchware.pm view on Meta::CPAN
sub verify {
my ($download_path, $package_path) = @_;
msg "Verifying the downloaded package [$package_path]";
my $retval;
unless (defined(config('verify_method'))) {
# if gpg fails try
# sha and if it fails try
# md5 and if it fails die
msg 'Trying to use gpg to cyptographically verify downloaded package.';
my ($gpg_err, $sha_err, $md5_err);
eval {$retval = gpg_verify($download_path)};
$gpg_err = $@;
if ($gpg_err) {
msg <<EOM;
Cyptographic verification using gpg failed!
GPG verification error [
$@
]
EOM
warn $gpg_err;
}
if (! $retval or $gpg_err) {
msg <<EOM;
Trying SHA1 verification of downloaded package.
EOM
eval {$retval = sha1_verify($download_path, $package_path)};
$sha_err = $@;
if ($sha_err) {
msg <<EOM;
SHA1 verification failed!
SHA1 verificaton error [
$@
lib/App/Fetchware.pm view on Meta::CPAN
You can also ignore the errors Fetchware printed out abover where it tried to
verify your downloaded file. See perldoc App::Fetchware.
EOW
vmsg <<EOM;
Verification Failed! But you asked to ignore verification failures, so this
failure is not fatal.
EOM
return 'warned due to verify_failure_ok'
}
}
} elsif (config('verify_method') =~ /gpg/i) {
vmsg <<EOM;
You selected gpg cryptographic verification. Verifying now.
EOM
###BUGALERT### Should trap the exception {gpg,sha1,md5}_verify()
#throws, and then add that error to the one here, otherwise the
#error message here is never seen.
gpg_verify($download_path)
or die <<EOD unless config('verify_failure_ok');
App-Fetchware: run-time error. You asked fetchware to only try to verify your
package with gpg or openpgp, but they both failed. See the warning above for
their error message. See perldoc App::Fetchware.
EOD
} elsif (config('verify_method') =~ /sha1?/i) {
vmsg <<EOM;
You selected SHA1 checksum verification. Verifying now.
EOM
sha1_verify($download_path, $package_path)
or die <<EOD unless config('verify_failure_ok');
App-Fetchware: run-time error. You asked fetchware to only try to verify your
package with sha, but it failed. See the warning above for their error message.
lib/App/Fetchware.pm view on Meta::CPAN
EOM
md5_verify($download_path, $package_path)
or die <<EOD unless config('verify_failure_ok');
App-Fetchware: run-time error. You asked fetchware to only try to verify your
package with md5, but it failed. See the warning above for their error message.
See perldoc App::Fetchware.
EOD
} else {
die <<EOD;
App-Fetchware: run-time error. Your fetchware file specified a wrong
verify_method option. The only supported types are 'gpg', 'sha', 'md5', but you
specified [@{[config('verify_method')]}]. See perldoc App::Fetchware.
EOD
}
msg 'Verification succeeded.';
}
sub gpg_verify {
my $download_path = shift;
my $keys_file;
# Attempt to download KEYS file in lookup_url's containing directory.
# If that fails, try gpg_keys_url if defined.
# Import downloaded KEYS file into a local gpg keyring using gpg command.
# Determine what URL to use to download the signature file *only* from
# lookup_url's host, so that we only download the signature from the
# project's main mirror.
# Download it.
# gpg verify the sig using the downloaded and imported keys in our local
# keyring.
# Skip downloading and importing keys if we're called from inside a
# fetchware package, which should already have a copy of our package's
# KEYS file.
unless (config('user_keyring')
or (-e './pubring.gpg' and -e './secring.gpg')) {
# Obtain a KEYS file listing everyone's key that signs this distribution.
if (defined config('gpg_keys_url')) {
$keys_file = no_mirror_download_file(config('gpg_keys_url'));
} else {
eval {
$keys_file = no_mirror_download_file(config('lookup_url'). '/KEYS');
};
die <<EOD if $@;
App-Fetchware: Fetchware was unable to download the gpg_key_url you specified or
that fetchware tried appending asc, sig, or sign to [@{[config('lookup_url')]}].
It needs to download this file to properly verify you software package. This is
a fatal error, because failing to verify packages is a perferable default over
potentially installing compromised ones. If failing to verify your software
package is ok to you, then you may disable verification by adding
verify_failure_ok 'On'; to your Fetchwarefile. See perldoc App::Fetchware.
EOD
}
# Import downloaded KEYS file into a local gpg keyring using gpg
# command.
eval {
# Add --homedir option if needed.
if (config('user_keyring')) {
run_prog('gpg', '--import', $keys_file);
} else {
run_prog('gpg', '--homedir', '.', '--import', $keys_file);
}
1;
} or msg <<EOM;
App-Fetchware: Warning: gpg exits nonzero when importing large KEY files such as
Apache's. However, despite exiting nonzero gpg still manages to import most of
the keys into its keyring. It only exits nonzero, because some of the keys in
the KEYS file had errors, and these key's errors were enough to cause gpg to
exit nonzero, but not enough to cause it to completely fail importing the keys.
EOM
}
# Download Signature using lookup_url.
my $sig_file;
my (undef, undef, $path, undef, undef) = uri_split($download_path);
my ($scheme, $auth, undef, undef, undef) = uri_split(config('lookup_url'));
my $sig_url;
for my $ext (qw(asc sig sign)) {
eval {
$sig_url = uri_join($scheme, $auth, "$path.$ext", undef, undef);
$sig_file = no_mirror_download_file($sig_url);
};
# If the file was downloaded stop trying other extensions.
last if defined $sig_file;
}
die <<EOD if not defined $sig_file;
App-Fetchware: Fetchware was unable to download the gpg_sig_url you specified or
that fetchware tried appending asc, sig, or sign to [$sig_url]. It needs
to download this file to properly verify you software package. This is a fatal
error, because failing to verify packages is a perferable default over
potentially installing compromised ones. If failing to verify your software
package is ok to you, then you may disable verification by adding
verify_failure_ok 'On'; to your Fetchwarefile. See perldoc App::Fetchware.
EOD
###BUGALERT### # Use Crypt::OpenPGP if its installed.
###BUGALERT### if (eval {use Crypt::OpenPGP}) {
##DOESNTWORK?? # Build a pubring needed for verify.
##DOESNTWORK?? my $pubring = Crypt::OpenPGP::KeyRing->new();
##DOESNTWORK?? my $secring = Crypt::OpenPGP::KeyRing->new();
##DOESNTWORK??
##DOESNTWORK?? # Turn on gpg compatibility just in case its needed.
##DOESNTWORK?? my $pgp = Crypt::OpenPGP->new(
##DOESNTWORK?? Compat => 'GnuPG',
##DOESNTWORK?? PubRing => $pubring,
##DOESNTWORK?? SecRing => $secring,
##DOESNTWORK?? # Automatically download public keys as needed.
##DOESNTWORK?? AutoKeyRetrieve => 1,
##DOESNTWORK?? # Use this keyserver to download them from.
##DOESNTWORK?? KeyServer => 'pool.sks-keyservers.net',
##DOESNTWORK?? );
##DOESNTWORK??
lib/App/Fetchware.pm view on Meta::CPAN
##DOESNTWORK?? } else {
##DOESNTWORK?? # print warning about $pgp errstr message.
##DOESNTWORK?? my $errstr = $pgp->errstr();
##DOESNTWORK?? warn "Crypt::OpenPGP failed with message: [$errstr]";
##DOESNTWORK?? # return failure, because Fetchware failed to verify the downloaded
##DOESNTWORK?? # file.
##DOESNTWORK?? return undef;
##DOESNTWORK?? }
###BUGALERT### } else {
###BUGALERT### ###BUGALERT### eval the run_prog()'s below & add better error reporting in
###BUGALERT### ###BUGALERT### if Crypt::OpenPGP works ok remove gpg support & this if &
###BUGALERT### }
#IPC::System::Simple dependency.
#my standard format.
# Use automatic key retrieval & a cool pool of keyservers
###BUGALERT## Give Crypt::OpenPGP another try with
#pool.sks-keyservers.net
###BUGALERT### Should I cache the files gpg puts in its "homedir"? They
#are the public keys that verify this fetchware package. Or should they
#always be downloaded on demand as they are now??? But if verify() can
#have keys cached inside the fetchware package does that mean that I
#should open up this as an API for fetchware extensions????? I don't
#know. I'll have to think more about this issue.
#run_prog('gpg', '--keyserver', 'pool.sks-keyservers.net',
# '--keyserver-options', 'auto-key-retrieve=1',
# '--homedir', '.', "$sig_file");
# Verify sig.
# Add --homedir option if needed.
if (config('user_keyring')) {
run_prog('gpg', '--verify', $sig_file);
} else {
run_prog('gpg', '--homedir', '.', '--verify', $sig_file);
}
# Return true indicating the package was verified.
return 'Package Verified';
}
sub sha1_verify {
my ($download_path, $package_path) = @_;
lib/App/Fetchware.pm view on Meta::CPAN
EOM
Mandatory => [ 'mirror', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a mirror configuration
option. Please add one, and try again.
EOM
Mandatory => [ 'lookup_url', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a lookup_url configuration
option. Please add one, and try again.
EOM
ConfigOptionEnum => ['lookup_method', [qw(timestamp versionstring)] ],
ConfigOptionEnum => ['verify_method', [qw(gpg sha1 md5)] ],
);
}
sub check_config_options {
my @args = @_;
lib/App/Fetchware.pm view on Meta::CPAN
=head1 SYNOPSIS
### App::Fetchware's use inside a Fetchwarefile.
### See fetchware's new command for an easy way to create Fetchwarefiles.
use App::Fetchware;
# Only program, lookup_url, one or more mirrors, and some method of
# verification are required.
program 'Your program';
lookup_url 'http://whatevermirror.your/program/is/on';
gpg_keys_url 'http://whatevermirror.your/program/gpg/key/url.asc';
mirror 'http://whatevermirror1.your/program/is/on';
mirror 'http://whatevermirror2.your/program/is/on';
mirror 'http://whatevermirror3.your/program/is/on';
mirror 'http://whatevermirror4.your/program/is/on';
mirror 'http://whatevermirror5.your/program/is/on';
# filter is not required, but is often needed to tell fetchware which
# program in the lookup_url directory or what specific version you would
# want to install. For example, Apache maintains 3 versions 2.0, 2.2, and
# 2.4. filter is what allows you to select which version you want fetchware
lib/App/Fetchware.pm view on Meta::CPAN
have the correct C<use App::Fetchware;> line at the top of your
Fetchwarefile.
=item B<2. Determine your lookup_url>
At the heart of App::Fetchware is its C<lookup_url>, which is
the URL to the FTP or HTTP mirror you want App::Fetchware to use to obtain a
directory listing to see if a new version of your program is available for
download. To figure this out just use your browser to find the program you
want fetchware to manage for you's Web site. Skip over the download link, and
instead look for the gpg, sha1, or md5 verify links, and copy and paste one of
those between the single quotes above in the lookup_url. Then delete the file
portion--from right to left until you reach a C</>. This is necessary, because
fetchware uses the lookup_url as a basis to download your the gpg, sha1, or md5
digital signatures or checksums to ensure that the packages fetchware downloads
and installs are exactly the same as the ones the author uploads.
lookup_url '';
And then after you copy the url.
lookup_url 'http://www.apache.org/dist/httpd/';
=item B<3. Determine your filter configuration option>
lib/App/Fetchware.pm view on Meta::CPAN
fetchware will refuse to build and install any software package that cannot be
verified. This limitation can be bypassed by setting the C<verify_failure_ok>
configuration option to true, but this is B<not> recommended.
Instead, if standard verification fails, please set up one or more of the
configuration options below that may allow verification to succeed if the author
has his download site set up differently then fetchware expects.
=over
=item gpg_keys_url - Should list a URL to a file most likely named C<KEYS> that
contains versions of the author's gpg verification keys that is suitable to be
imported into gpg using C<gpg --import [name of file]>. An example would be:
gpg_keys_url 'http://www.apache.org/dist/httpd/KEYS';
=item users_keyring - Tells fetchware to use the user who calls fetchware's gpg
keyring instead of fetchware's own keyring. This is handy for when you want to
install a program, but the author has no easily accessible C<KEYS> file, but the
author has listed his gpg key on his Website. With this option, you can import
this key into your own keyring using C<gpg --import [name of file]>, and then
specify this option in your Fetchwarefile as shown below.
users_keyring 'On';
=item gpg_sig_url - Should list a URL to a directory (not a file) that has files
with the same names as the software archives that contain your program, but with
a C<.asc>, C<.sig>, or C<.sign> file extension. An example would be:
gpg_sig_url 'http://www.apache.org/dist/httpd/';
=item sha1_url - Should list a URL to a directory (not a file) that has files
with the same names as the software archives that contain your program, but with
a C<.sha> or C<.sha1> file extension. An example would be:
sha1_url 'http://www.apache.org/dist/httpd/';
=item md5_url - Should list a URL to a directory (not a file) that has files
with the same names as the software archives that contain your program, but with
a C<.md5> file extension. An example would be:
lib/App/Fetchware.pm view on Meta::CPAN
used if C<lookup_method> is not specified.
=item versionstring
Versionstring parses out the version numbers that each downloadable program has,
and uses them to determine the downloadable archive with the highest version
number, which should also be the newest and best version of the archive to use.
=back
=head2 gpg_keys_url 'lookup_url.com/some/path';
Specifies a file not a directory URL for a C<KEYS> file that lists all of the
authors' gpg keys for fetchware to download and import before using them to
verify the downloaded software package.
If you come accross a software package whoose author uses gpg to sign his
software packages, but he does not include it in the form of a file on his main
mirror, then you can specify the C<user_keyring> option. This option forces
fetchware to use the user who runs fetchware's keyring instead of fetchware's
own keyring. This way you can then import the author's key into your own
keyring, and have fetchware use that keyring that already has the author's key
in it to verify your downloads.
=head2 user_keyring 'On';
When enabled fetchware will use the user who runs fetchware's keyring instead of
fetchware's own keyring. Fetchware uses its own keyring to avoid adding cruft to
your own keyring.
This is needed when the author of a software package does not maintain a KEYS
file that can easily be downloaded and imported into gpg. This option allows you
to import the author's key manually into your own gpg keyring, and then
fetchware will use your own keyring instead of its own to verify your downloads.
=over
=item LIMITAITON
C<user_keyring> when set to true requires that the user that fetchware is
running under have a real gpg keyring with keys that have been imported into it.
This is not the case B<unless> the C<user> option has been specified with a user
account with a proper home directory and gpg keyring for gpg to use. Because of
this limitation if you need to specify C<user_keyring> be sure to also specify
the C<user> option to specify a I<real> user account instead of the default fake
one C<nobody>.
Typically you would import the keys into your own user accounts gpg keyring, and
then you would specify your own username with the C<user> option to tell
fetchware to drop privs to your own user account to have access to your own gpg
keys.
=back
=head2 gpg_sig_url 'mirror.com/some/path';
Specifies an alternate url to use to download the cryptographic signature that
goes with your program. This is usually a file with the same name as the
download url with a C<.asc> file extension added on. Fetchware will also append
the extensions C<sig> and C<sign> if C<.asc> is not found, because some pgp
programs and authors use these extensions too.
=head2 sha1_url 'mastermirror.com/some/path';
Specifies an alternate url to download the SHA-1 checksum. This checksum is used
lib/App/Fetchware.pm view on Meta::CPAN
site, because if you download it from a mirror, its possible that both the
archive and the checksum could have been tampered with.
=head2 user_agent 'Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0'
Specifies what C<user_agent> you would like fetchware to pretend to be when
downloading files using the HTTP protocol. Some sites annoying prevent some
user agents from working while allowing others. This allows you to pretend to
be a real browser such as Firefox if you need to.
=head2 verify_method 'gpg';
Chooses a method to verify your program. The default is to try C<gpg>, then
C<sha1>, and finally C<md5>, and if all three fail, then the default is to exit
fetchware with an error message, because it is insecure to install archives that
cannot be verified. The availabel options are:
=over
=item gpg - Uses the gpg program to cryptographically verify that the program you downloaded is exactly the same as its author uploaded it.
=item sha1 - Uses the SHA-1 hash function to verify the integrity of the download. This is much less secure than gpg.
=item md5 - Uses the MD5 hash function to verify the integrity of the download. This is much less secure than gpg.
=back
=head2 verify_failure_ok 'True';
Fetchware's default regarding failing to verify your downloaded Archive with
gpg, sha1, or md5 is to exit with an error message, because installing software
that cannot be cryptographically verified should never be done.
=over
=item SECURITY NOTICE
However, if the author of a program you want to use fetchware to manage for you
does not offer a gpg, sha1, or md5 file to verify its integrity, then you can
use this option to force Fetchware to install this program anyway. However, do
not enable this option lightly. Please scour the program's mirrors and homepage
to see which C<gpg_keys_url>, C<gpg_sig_url>, C<sha1_url>, C<md5_url>, or
C<user_keyring> you can use to ensure that your archive is verified before it is
compiled and installed. Even mirrors from sites large and small get hacked
regularly:
L<http://www.itworld.com/security/322169/piwik-software-installer-rigged-back-door-following-website-compromise?page=0,0>
L<http://www.networkworld.com/news/2012/092612-compromised-sourceforge-mirror-distributes-backdoored-262815.html>
L<http://www.csoonline.com/article/685037/wordpress-warns-server-admins-of-trojans>
L<http://www.computerworld.com/s/article/9233822/Hackers_break_into_two_FreeBSD_Project_servers_using_stolen_SSH_keys>
So, Please give searching for a C<gpg_keys_url>, C<gpg_sig_url>, C<sha1_url>,
C<md5_url>, or C<user_keyring> for your program another try before simply
enabling this option.
=back
=over
=item NOTICE
C<verify_failure_ok> is a boolean configuration option, which just means its
lib/App/Fetchware.pm view on Meta::CPAN
=item L<OVERRIDE_LOOKUP|lookup() API REFERENCE> -
L</get_directory_listing()>, L</parse_directory_listing()>,
L</determine_download_path()>, L</ftp_parse_filelist()>, L</http_parse_filelist()>,
L</file_parse_filelist()>, L</lookup_by_timestamp()>,
L</lookup_by_versionstring()>, L</lookup_determine_downloadpath()>
=item L<OVERRIDE_DOWNLOAD|download() API REFERENCE> -
L</determine_package_path()>
=item L<OVERRIDE_VERIFY|verify() API REFERENCE> - L</gpg_verify()>,
L</sha1_verify()>, L</md5_verify()>, L</digest_verify()>
=item L<OVERRIDE_UNARCHIVE|unarchive() API REFERENCE> -
L</check_archive_files()>, L</list_files()>, L</list_files_tar()>,
L</list_files_zip()>, L</unarchive_package()>, L</unarchive_tar()>,
L</unarchive_zip()>
=item L<OVERRIDE_BUILD|build() API REFERENCE> - L</run_star_commands()> and
L</run_configure()>.
lib/App/Fetchware.pm view on Meta::CPAN
use App::Fetchware;
program 'Apache';
lookup_url 'http://www.apache.org/dist/httpd/';
filter 'httpd-2.2';
mirror 'http://apache.mirrors.pair.com/httpd/';
mirror 'http://mirrors.ibiblio.org/apache/httpd/';
mirror 'ftp://apache.cs.utah.edu/apache.org/httpd/';
verify_method 'gpg';
gpg_keys_url 'http://www.apache.org/dist/httpd/KEYS';
make_options '-j 4';
prefix '/home/dly/software/apache2.2';
# You can use heredocs to make gigantic options like this one more legible.
configure_options <<EOO;
--with-mpm=prefork
--enable-modules="access alias auth autoindex cgi logio log_config status vhost_alias userdir rewrite ssl"
--enable-so
EOO
lib/App/Fetchware.pm view on Meta::CPAN
needed. First, nginx does not seem to use any mirrors at all, which means
nginx's Fetchwarefile is going to look kind of stupid with the same exact URL
being used for both the C<lookup_url> and the C<mirror>, but such a
configuration is supported. Next, nginx does not have a KEYS file, but it does
list it's developer's keys on its Website. So, they have to be imported manually
into your keyring, and then specify the C<user_keyring> option to switch
fetchware from usings its own keyring to using your own keyring. Also, note the
comment regarding having to use the C<user> option to specify a real user
account. This is needed, because the verify step is done by fetchware's child
after that child drops its root privileges. Th default user is nobody, and
nobody has no real home, and therefore no keyring, so gpg won't be able to read
the keys you ask it to by using the C<user_keyring> option; therefore, C<user>
must be specified to change it to a real user, whoose keyring has had these keys
imported into it. Also, worth noting that this nginx configuration does not use
a C<filter> option. This is not actually needed, because the only source-code
packages availabe at the C<lookup_url> are the nginx software packages
themselves, but it might be a good idea to include one, because the nginx
developers could always change how their download server is structured. So,
including it is always a good idea.
=over
lib/App/Fetchware.pm view on Meta::CPAN
# lookup_url and mirror are the same thing, because nginx does not seem to have
# mirrors. Fetchware, however, requires one, so the same URL is simply
# duplicated.
lookup_url 'http://nginx.org/download/';
mirror 'http://nginx.org/download/';
# Must add the developers public keys to my own keyring. These keys are
# availabe from http://nginx.org/en/pgp_keys.html Do this with:
# gpg \
# --fetch-keys http://nginx.org/keys/aalexeev.key\
# --fetch-keys http://nginx.org/keys/is.key\
# --fetch-keys http://nginx.org/keys/mdounin.key\
# --fetch-keys http://nginx.org/keys/maxim.key\
# --fetch-keys http://nginx.org/keys/sb.key\
# --fetch-keys http://nginx.org/keys/glebius.key\
# --fetch-keys http://nginx.org/keys/nginx_signing.key
# You might think you could just set gpg_keys_url to the nginx-signing.key key,
# but that won't work, because like apache different releases are signed by
# different people. Perhaps I could change gpg_keys_url to be like mirror where
# you can specify more than one option?
user_keyring 'On';
# user_keyring specifies to use the user's own keyring instead of fetchware's.
# But fetchware drops privileges by default using he user 'nobody.' nobody is
# nobody, so that user account does not have a home directory for gpg to read a
# keyring from. Therefore, I'm using my own account instead.
user 'dly';
# The other option, which is commented out below, is to use root's own keyring,
# and the no_install option to ensure that root uses its own keyring instead of
# nobody's.
# noinstall 'On';
verify_method 'gpg';
=back
=head2 PHP Programming Language
PHP annoyingly uses a custom Web application on each of its mirror sites to
serve HTTP downloads. No simple directory listing is available. Therefore, to
use php with fetchware, custom C<lookup>, C<download>, and C<verify> hooks are
needed that override fetchware's internal behavior to customize fetchware as
needed so that it can work with how PHP's site is up.
lib/App/Fetchware.pm view on Meta::CPAN
It overrides lookup() to use a local git repo stored in the $git_repo_dir
variable. To create a repo just clone php's git repo (see
http://us1.php.net/git.php for details.). It runs git pull to update the repo,
and then it runs git tags, and ditches some older junk tags, and finds only the
tags used for new versions of php. These are sorted using the C<versonstring>
lookup() algorithm, and the latest one is returned.
download() uses C<git checkout [latesttag]> to "download" php by simply changing
the working directory to the latest tag. verify() uses git's cool C<verify-tag>
command to verify the gpg signature. unarchive() is updated to do nothing since
there is no archive to unarchive. However, because we reuse build(), archive()
must return a $build_path that build() will change its directory to. start() and
end() are also overridden, because managing a temporary directory is not needed,
so, instead, they just do a C<git checkout master> to switch from whatever the
latest tag is back to master, because git pull bases what it does on what branch
you're in, so we must actually be a real branch to update git.
=over
# php-using-git.Fetchwarefile: example fetchwarefile using php's git repo
lib/App/Fetchware.pm view on Meta::CPAN
my $latest_tag = $download_path;
# checkout the $latest_tag to download it.
run_prog('git checkout', "$latest_tag");
my $package_path = cwd();
return $package_path;
};
# You must manually add php's developer's gpg keys to your gpg keyring. Do
# this by going to the page: http://us1.php.net/downloads.php . At the
# bottom the gpg key "names are listed such as "7267B52D" or "5DA04B5D."
# These are their key "names." Use gpg to download them and import them into
# your keyring using: gpg --keyserver pgp.mit.edu --recv-keys [key id]
hook verify => sub {
my ($download_path, $package_path) = @_;
# the latest tag is the download path see lookup.
my $latest_tag = $download_path;
# Run git verify-tag to verify the latest tag
my $success = eval { run_prog('git verify-tag', "$latest_tag"); 1;};
# If the git verify-tag fails, *and* verify_failure_ok has been turned on,
lib/App/Fetchware.pm view on Meta::CPAN
at Fetchware's level. Perl level syntax errors are not checked. PPI is not used
to parse the file. Instead only high-level Fetchware specific syntax errors are
checked.
=item B<my $temp_dir = start(KeepTempDir => 0 | 1)> - Gives your extension a chance to do anything needed before the rest of the API subroutines get called. App::Fetchware's C<start()> manages App::Fetchware's temporary directory creation. If you wo...
=item B<my $download_url = lookup()> - Determines and returns a download url that C<download()> receives and uses to download the archive for the program.o
=item B<my $package_path = download($tempd_dir, $download_url)> - Downloads its provided $download_url argument.
=item B<verify($download_url, $package_path)> - Verifies the integrity of your downloaded archive using gpg, sha1, or md5.
=item B<my $build_path = unarchive($package_path)> - Unpacks the downloaded archive.
=item B<build($build_path)> - Configures and compiles the downloaded archive.
=item B<install()> - Installs the compiled archive.
=item B<end()> - Cleans up the temporary directory that start() created. Can be overridden to do any other clean up tasks that your archive needs.
=item B<uninstall($build_path)> - Uninstalls an already installed program installed with the same App::Fetchware extension.
lib/App/Fetchware.pm view on Meta::CPAN
# mirrors => [
# 'ftp://some.mirror/mirror',
# 'http://some.mirror/mirror',
# 'file://some.mirror/mirror',
# ],
# );
Asks the user to specify at least one mirror to use to download their archives.
It also reiterates to the user that the C<lookup_url> should point to the
author's original download site, and B<not> a 3rd party mirror, because md5sums,
sha1sums, and gpg signatures should B<only> be downloaded from the author's
download site to avoid them being modified by a hacked 3rd party mirror. While
C<mirror> should be configured to point to a 3rd party mirror to lessen the load
on the author's offical download site.
After the user enters at least one mirror, get_mirrors() asks the user if they
would like to add any additional mirrors, and it adds them if the user specifies
them.
The list of the mirrors the user specified is returned as a hash with only one
key C<mirror>, and a value that is an arrayref of mirrors that the user has
specified. The caller, then should call append_options_to_fetchwarefile() to add
this list of mirrors to the user's Fetchwarefile.
=head3 get_verification()
my $verification_hashref = get_verification($term, $filename_listing, $lookup_url);
# $verification_hashref = (
# gpg_keys_url => 'http://main.mirror/distdir',
# verification_method => 'gpg',
# );
Parses $filename_listing to determine what type of verification is available.
Prefering gpg, but falling back on sha1, and then md5 if gpg is not available.
If the type is gpg, then get_verification() will ask the user to specify a
C<gpg_keys_url>, which is required for gpg, because fetchware needs to be able
to import the needed keys to be able to use those keys to verify package
downloads. If this URL is not provided by the author, then get_verification()
will ask the user if they would like to import the author's key into their own
gpg public keyring. If they would, then get_verification() will use the
C<user_keyring> C<'On'> option to use the user's public keyring instead of
fetchware's own keyring. And if the user does not want to use their own gpg
public keyring, then get_verification will fall back to sha1 or md5 setting
C<verify_method> to sha1 or md5 as needed.
Also, adds a gpg_keys_url option if a C<KEYS> file is found in
$filename_listing.
If no verification methods are available, fetchware will print a big nasty
warning message, and offer to use C<verify_failure_ok> to make such a failure
cause fetchware to continue installing your software.
Returns a hashref of options for the user's Fetchwarefile. You're responsible
for calling append_options_to_fetchwarefile() to add them to the user's
Fetchwarefile, or perhaps the caller could analyze them in some way, before
adding them if needed. The keys are the names of the configuration options, and
lib/App/Fetchware.pm view on Meta::CPAN
=head2 verify()
verify($download_path, $package_path)
=over
=item Configuration subroutines used:
=over
=item gpg_keys_url
=item user_keyring
=item gpg_sig_url
=item sha1_url
=item md5_url
=item verify_method
=item verify_failure_ok
=item user_agent
=back
=back
Verifies the downloaded package stored in $package_path by downloading
$download_path.{asc,sha1,md5}> and comparing the two together. Uses the
helper subroutines C<{gpg,sha1,md5,digest}_verify()>.
=over
=item LIMITATIONS
Uses gpg command line, and the interface to gpg is a little brittle.
Crypt::OpenPGP is buggy and not currently maintainted again, so fetchware cannot
make use of it, so were stuck with using the command line gpg program.
=back
=over
=item drop_privs() NOTES
This section notes whatever problems you might come accross implementing and
debugging your Fetchware extension due to fetchware's drop_privs mechanism.
lib/App/Fetchware.pm view on Meta::CPAN
=head2 verify() API REFERENCE
The subroutines below are used by verify() to provide the verify
functionality for fetchware. If you have overridden the verify() handler, you
may want to use some of these subroutines so that you don't have to copy and
paste anything from verify().
App::Fetchware is B<not> object-oriented; therefore, you B<can not> subclass
App::Fetchware to extend it!
=head3 gpg_verify()
'Package Verified' = gpg_verify($download_path);
Uses the command-line program C<gpg> to cryptographically verify that the file
you download is the same as the file the author uploaded. It uses public-key
priviate-key cryptography. The author signs his software package using gpg or
some other OpenPGP compliant program creating a digital signature file with the
same filename as the software package, but usually with a C<.asc> file name
extension. gpg_verify() downloads the author's keys, imports them into
fetchware's own keyring unless the user sets C<user_keyring> to true in his
Fetchwarefile. Then Fetchware downloads a digital signature that usually
ends in C<.asc>. Afterwards, fetchware uses the gpg command line program to
verify the digital signature. gpg_verify returns true if successful, and throws
an exception otherwise.
You can use C<gpg_keys_url> to specify the URL of a file where the author has
uploaded his keys. And the C<gpg_sig_url> can be used to setup an alternative
location of where the C<.asc> digital signature is stored.
=head3 sha1_verify()
'Package verified' = sha1_verify($download_path, $package_path);
undef = sha1_verify($download_path, $package_path);
Verifies the downloaded software archive's integrity using the SHA Digest
specified by the C<sha_url 'ftp://sha.url/package.sha'> config option. Returns
true for sucess dies on error.
=over
=item SECURITY NOTE
If an attacker cracks a mirror and modifies a software package, they can also
modify the MD5 sum of that software package on that B<same mirror>. Because of
this limitation MD5 sums can only tell you if the software package was corrupted
while downloading. This can actually happen as I've had it happen to me once.
If your stuck with using MD5 sum, because your software package does not provide
gpg signing, I recommend that you download your SHA1 sums (and MD5 sums) from
your software package's master mirror. For example, Apache provides MD5 and SHA1
sums, but it does not mirror them--you must download them directly from Apache's
servers. To do this specify a C<sha1_url 'master.mirror/package.sha1';> in your
Fetchwarefile.
=back
=head3 md5_verify()
'Package verified' = md5_verify($download_path, $package_path);
lib/App/Fetchware.pm view on Meta::CPAN
=over
=item SECURITY NOTE
If an attacker cracks a mirror and modifies a software package, they can also
modify the MD5 sum of that software package on that B<same mirror>. Because of
this limitation MD5 sums can only tell you if the software package was corrupted
while downloading. This can actually happen as I've had it happen to me once.
If your stuck with using MD5 sum, because your software package does not provide
gpg signing, I recommend that you download your MD5 sums (and SHA1 sums) from
your software package's master mirror. For example, Apache provides MD5 and SHA1
sums, but it does not mirror them--you must download them directly from Apache's
servers. To do this specify a C<md5_url 'master.mirror/package.md5';> in your
Fetchwarefile.
=back
=head3 digest_verify()
'Package verified' = digest_verify($digest_type, $download_path, $package_path);
lib/App/Fetchware.pm view on Meta::CPAN
=over
=item SECURITY NOTE
If an attacker cracks a mirror and modifies a software package, they can also
modify the $digest_type sum of that software package on that B<same mirror>.
Because of this limitation $digest_type sums can only tell you if the software
package was corrupted while downloading. This can actually happen as I've had
it happen to me once.
If your stuck with using $digest_type sum, because your software package does
not provide gpg signing, I recommend that you download your $digest_type sums
(and SHA1 sums) from your software package's master mirror. For example, Apache
provides MD5 and SHA1 sums, but it does not mirror them--you must download them
directly from Apache's servers. To do this specify a
C<$digest_type_url 'master.mirror/package.$digest_type';>' in your Fetchwarefile.
=back
=head2 unarchive()
my $build_path = unarchive($package_path)
lib/App/Fetchware.pm view on Meta::CPAN
make_options.
=back
=item * Ensures some options have only allowable options specified.
=over
=item * lookup_method can only have 'timestamp' or 'versionstring'. as options.
=item * And verify_method can only have 'gpg', 'sha1', or 'md5' specified.
=back
=back
=over
=item drop_privs() NOTES
This section notes whatever problems you might come accross implementing and
lib/App/Fetchware.pm view on Meta::CPAN
EOM
Mandatory => [ 'mirror', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a mirror configuration
option. Please add one, and try again.
EOM
Mandatory => [ 'lookup_url', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a lookup_url configuration
option. Please add one, and try again.
EOM
ConfigOptionEnum => ['lookup_method', [qw(timestamp versionstring)] ],
ConfigOptionEnum => ['verify_method', [qw(gpg sha1 md5)] ],
);
Uses config() to test that no configuration options that clash with each other
are used.
It's parameters are specified in a list with an even number of parameters. Each
group of 2 parameters specifies a type of test that check_config_options() will
test for. There are three types of tests. Also, note that the parameters are
specified as a list not as a hash, because multiple "keys" are allowed, and hash
keys must be unique; therefore, the parameters are a list instead of a hash.
lib/App/Fetchware.pm view on Meta::CPAN
This extension mechanism is also very easy for Perl programmers, because you're
basically I<subclassing> App::Fetchware, only you do it using
L<App::Fetchware::ExportAPI> and L<App::Fetchware::CreateConfigOptions>. See
section L<Implement your fetchware extension.> for full details.
=head2 How do I fix the verification failed error.
Fetchware is designed to always attempt to verify the software archives it
downloads even if you failed to configure fetchware's verification settings. It
will try to guess what those setting should be using simple heuristics. First it
will try gpg verificaton, then sha1 verification, and finally md5 verification.
If all fail, then fetchware exit failure with an appropriate error message.
When you get this error message
L<read fetchware's documentation on how to set this up|/4. Add mandatory verification settings>.
=head2 How do I make fetchware log to a file instead of STDOUT?
You can't fetchware does not have any log file support. However, you can simply
redirect STDOUT to a file to make your shell redirect STDOUT to a file for you.
lib/App/Fetchware.pm view on Meta::CPAN
fetchware install <some-program.Fetchwarefile> 2>&1 fetchware.log
And to throw away all messages use:
fetchware -q install <some-progra.Fetchwarefile>
or use the shell
fetchware install <some-program.Fetchwarefile 2>&1 /dev/null
=head2 Why don't you use Crypt::OpenPGP instead of the gpg command line program?
I tried to use Crypt::OpenPGP, but I couldn't get it to work. And getting gpg to
work was a breeze after digging through its manpage to find the right command
line options that did what I need it to.
Also, unfortunately Crypt::OpenPGP is buggy, out-of-date, and seems to have
lost another maintainer. If it ever gets another maintainer, who fixes the newer
bugs, perhaps I'll add support for Crypt::OpenPGP again. Because of how
fetchware works it needs to use supported but not popular options of
Crypt::OpenPGP, which may be where the bugs preventing it from working reside.
Supporting Crypt::OpenPGP is still on my TODO list. It's just not very high on
that list. Patches are welcome to add support for it, and the old code is still
there commented out, but it needs updating if anyone is interested.
In the meantime if you're on Windows without simple access to a gpg command line
program, try installing gpg from the L<gpg4win project|http://gpg4win.org/>,
which packages up gpg and a bunch of other tools for easier use on Windows.
=head2 Does fetchware support Windows?
Yes and no. I intend to support Windows, but right now I'm running Linux, and my
Windows virtual machine is broken, so I can't easily test it on Windows. The
codebase makes heavy use of File::Spec and Path::Class, so all of its file
operations should work on Windows.
I currently have not tested fetchware on Windows. There are probably some test
failures on Windows, but Windows support should be just a few patches away.
lib/App/Fetchware/Util.pm view on Meta::CPAN
# Don't CLEANUP if KeepTempDir is set.
push @args, CLEANUP => 1 if not defined $opts{KeepTempDir};
# Call tempdir() with the @args I've built.
$temp_dir = tempdir(@args);
# Only when we do *not* drop privs...
if (config('stay_root')
or ($< != 0 or $> != 0)
) {
# ...Must chmod 700 so gpg's localized keyfiles are good.
chmod(0700, $temp_dir) or die <<EOD;
App-Fetchware-Util: Fetchware failed to change the permissions of its temporary
directory [$temp_dir] to 0700. This should not happen, and is a bug, or perhaps
your system's temporary directory is full. The OS error was [$!].
EOD
}
$exception = $@;
1; # return true unless an exception is thrown.
} or die <<EOD;
t/App-Fetchware-check_syntax.t view on Meta::CPAN
option. Please add one, and try again.
EOM
is($retval, 'Syntax Ok', 'checked check_config_options() Mandatory success.');
__clear_CONFIG();
# ConfigOptionEnum only triggers when the specified option, verify_method in
# this case, has been specified. Therefore, I must specify verify_method in
# order to test ConfigOptionEnum.
config(verify_method => 'notgpg');
eval_ok(sub{check_config_options(
ConfigOptionEnum => ['verify_method', [qw(gpg sha1 md5)] ],)
}, <<EOD, 'checked check_config_options() ConfigOptionEnum exception');
App-Fetchware: You specified the option [verify_method], but failed to specify only
one of its acceptable values [gpg sha1 md5]. Please change the value you
specified [notgpg] to one of the acceptable ones listed above, and try again.
EOD
__clear_CONFIG();
config(verify_method => 'gpg');
is(check_config_options(
ConfigOptionEnum => ['verify_method', [qw(gpg sha1 md5)] ],),
'Syntax Ok', 'checked check_config_options() ConfigOptionEnum success.');
__clear_CONFIG();
# Define a test Fetchwarefile with just config().
config(program => 'Some Program');
config(lookup_url => 'scheme://fake.url');
config(mirror => 'scheme://fake.url');
is(check_config_options(
t/App-Fetchware-check_syntax.t view on Meta::CPAN
EOM
Mandatory => [ 'mirror', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a mirror configuration
option. Please add one, and try again.
EOM
Mandatory => [ 'lookup_url', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a lookup_url configuration
option. Please add one, and try again.
EOM
ConfigOptionEnum => ['lookup_method', [qw(timestamp versionstring)] ],
ConfigOptionEnum => ['verify_method', [qw(gpg sha1 md5)] ],
), 'Syntax Ok', 'checked check_config_options() all options together.');
};
subtest 'test check_syntax()' => sub {
is(check_syntax(), 'Syntax Ok',
'checked check_syntax() success.');
};
t/App-Fetchware-config-file.t view on Meta::CPAN
temp_dir 'test';
fetchware_db_path 'test';
user 'test';
prefix 'test';
configure_options 'test';
make_options 'test';
build_commands 'test';
install_commands 'test';
lookup_url 'test';
lookup_method 'test';
gpg_sig_url 'test';
gpg_keys_url 'test';
sha1_url 'test';
md5_url 'test';
verify_method 'test';
no_install 'test';
verify_failure_ok 'test';
stay_root 'test';
user_keyring 'test';
debug_CONFIG();
for my $config_sub (qw(
temp_dir
user
prefix
configure_options
make_options
build_commands
install_commands
lookup_url
lookup_method
gpg_sig_url
verify_method
no_install
verify_failure_ok
stay_root
)) {
is(config($config_sub), 'test', "checked config sub $config_sub");
}
# Test 'MANY' config subs.
mirror 'test';
t/App-Fetchware-config-file.t view on Meta::CPAN
[ fetchware_db_path => 'ONE' ],
[ user => 'ONE' ],
[ prefix => 'ONE' ],
[ configure_options=> 'ONEARRREF' ],
[ make_options => 'ONEARRREF' ],
[ build_commands => 'ONEARRREF' ],
[ install_commands => 'ONEARRREF' ],
[ uninstall_commands => 'ONEARRREF' ],
[ lookup_url => 'ONE' ],
[ lookup_method => 'ONE' ],
[ gpg_keys_url => 'ONE' ],
[ gpg_sig_url => 'ONE' ],
[ sha1_url => 'ONE' ],
[ md5_url => 'ONE' ],
[ verify_method => 'ONE' ],
[ mirror => 'MANY' ],
[ no_install => 'BOOLEAN' ],
[ verify_failure_ok => 'BOOLEAN' ],
[ stay_root => 'BOOLEAN' ],
[ user_keyring => 'BOOLEAN' ],
);
t/App-Fetchware-lookup.t view on Meta::CPAN
<p>Always use the signature files to verify the authenticity
of the distribution, <i>e.g.</i>,</p>
<pre>
% pgpk -a KEYS
% pgpv httpd-2.2.8.tar.gz.asc
<i>or</i>,
% pgp -ka KEYS
% pgp httpd-2.2.8.tar.gz.asc
<i>or</i>,
% gpg --import KEYS
% gpg --verify httpd-2.2.8.tar.gz.asc
</pre>
<p>We offer MD5 hashes as an alternative to validate the integrity
of the downloaded files. A unix program called <code>md5</code> or
<code>md5sum</code> is included in many unix distributions. It is
also available as part of <a
href="http://www.gnu.org/software/textutils/textutils.html">GNU
Textutils</a>. Windows users can get binary md5 programs from <a
href="http://www.fourmilab.ch/md5/">here</a>, <a
href="http://www.pc-tools.net/win32/freeware/console/">here</a>, or
t/App-Fetchware-new-get_verification view on Meta::CPAN
my $verification_hashref = get_verification($term, $current_file_list, $lookup_url);
ok(exists $verification_hashref->{user_keyring},
'checked get_verification() keyring success.');
like($verification_hashref->{user_keyring}, qr/on|true/i,
'checked get_verification() keyring value.');
ok(exists $verification_hashref->{verify_method},
'checked get_verification method sucess.');
like($verification_hashref->{verify_method}, qr/gpg/i,
'checked get_verification() verify_method value.');
# Spit out # of tests run.
done_testing();
# Print a bogus "prompt" to keep Expect from freaking out, because it presumes
# the prompt works like it does in a shell, but fetchware new is not a shell.
print "Bogus shell: \n";
# Because we're in a child process not the same one that is running the main
t/App-Fetchware-new.t view on Meta::CPAN
##TODO####BROKEN##
##TODO####BROKEN## expect_send("$ENV{FETCHWARE_HTTP_MIRROR_URL}",
##TODO####BROKEN## 'checked cmd_new() say mirror.');
##TODO####BROKEN##
##TODO####BROKEN## expect_like(qr/In addition to the one required mirror that you must define in order for/ms,
##TODO####BROKEN## 'checked cmd_new() received more mirrors prompt.');
##TODO####BROKEN##
##TODO####BROKEN## expect_send('N',
##TODO####BROKEN## 'checked cmd_new() say N for more mirrors.');
##TODO####BROKEN##
##TODO####BROKEN## #expect_like(qr!\[y/N\]|gpg digital signatures found. Using gpg verification.!ms,
##TODO####BROKEN## expect_like(qr!.*|gpg digital signatures found. Using gpg verification.!ms,
##TODO####BROKEN## 'checked cmd_new() received filter prompt.');
##TODO####BROKEN##
##TODO####BROKEN## expect_send('httpd-2.2',
##TODO####BROKEN## 'checked cmd_new() say httpd-2.2 for filter option.');
##TODO####BROKEN##
##TODO####BROKEN## expect_like(qr/Fetchware has many different configuration options that allow you to control its/ms,
##TODO####BROKEN## 'checked cmd_new() received extra config prompt.');
##TODO####BROKEN##
##TODO####BROKEN## expect_send('N',
##TODO####BROKEN## 'checked cmd_new() say N for more config options prompt.');
t/App-Fetchware-new_install.t view on Meta::CPAN
my $fetchwarefile = <<EOF;
use App::Fetchware;
program 'Apache 2.2';
lookup_url '$ENV{FETCHWARE_HTTP_LOOKUP_URL}';
mirror '$ENV{FETCHWARE_FTP_MIRROR_URL}';
gpg_keys_url "$ENV{FETCHWARE_HTTP_LOOKUP_URL}/KEYS";
filter 'httpd-2.2';
EOF
note('FETCHWAREFILE');
note("$fetchwarefile");
my $new_fetchware_package_path =
t/App-Fetchware-new_install.t view on Meta::CPAN
my $fetchwarefile = <<EOF;
use App::Fetchware;
program 'Apache 2.2';
lookup_url '$ENV{FETCHWARE_HTTP_LOOKUP_URL}';
mirror '$ENV{FETCHWARE_FTP_MIRROR_URL}';
gpg_keys_url "$ENV{FETCHWARE_HTTP_LOOKUP_URL}/KEYS";
filter 'httpd-2.2';
EOF
note('FETCHWAREFILE');
note("$fetchwarefile");
my $fetchware_package_path = new_install($term, 'Apache 2.2',
$fetchwarefile);
t/App-Fetchware-verify.t view on Meta::CPAN
#!perl
# App-Fetchware-verify.t tests App::Fetchware's verify() subroutine, which gpg
# verifies your downloaded archive if possible. If not it will also try md5/sha.
# Pretend to be bin/fetchware, so that I can test App::Fetchware as though
# bin/fetchware was calling it.
package fetchware;
use strict;
use warnings;
use 5.010001;
# Set a umask of 022 just like bin/fetchware does. Not all fetchware tests load
# bin/fetchware, and so all fetchware tests must set a umask of 0022 to ensure
t/App-Fetchware-verify.t view on Meta::CPAN
BEGIN { use_ok('App::Fetchware', qw(:DEFAULT :OVERRIDE_VERIFY)); }
# Print the subroutines that App::Fetchware imported by default when I used it.
note("App::Fetchware's default imports [@App::Fetchware::EXPORT]");
subtest 'OVERRIDE_VERIFY exports what it should' => sub {
my @expected_overide_verify_exports = qw(
gpg_verify
sha1_verify
md5_verify
digest_verify
);
# sort them to make the testing their equality very easy.
my @sorted_verify_tag = sort @{$App::Fetchware::EXPORT_TAGS{OVERRIDE_VERIFY}};
@expected_overide_verify_exports = sort @expected_overide_verify_exports;
is_deeply(\@sorted_verify_tag, \@expected_overide_verify_exports,
'checked for correct OVERRIDE_VERIFY @EXPORT_TAG');
};
t/App-Fetchware-verify.t view on Meta::CPAN
"checked sha1_verify() success.");
sha1_url 'http://www.apache.org/dist/httpd/';
ok(sha1_verify($download_path, $package_path),
'checked sha1_verify() sha_url success.');
};
subtest 'test gpg_verify()' => sub {
skip_all_unless_release_testing();
# Clean the gunk of of %CONFIG.
__clear_CONFIG();
# Test gpg_verify() success.
lookup_url 'http://www.apache.org/dist/httpd';
ok(gpg_verify($download_path), 'checked gpg_verify() success');
# Test gpg_verify() success using gpg_keys_url.
gpg_keys_url config('lookup_url') . '/KEYS';
ok(gpg_verify($download_path), 'checked gpg_verify() success');
eval_ok(sub {
gpg_verify('ftp://fake.url/will.fail');
}, <<EOE, 'checked gpg_verify() download gpg_sig_url failure');
App-Fetchware: Fetchware was unable to download the gpg_sig_url you specified or
that fetchware tried appending asc, sig, or sign to [http://www.apache.org/will.fail.sign]. It needs
to download this file to properly verify you software package. This is a fatal
error, because failing to verify packages is a perferable default over
potentially installing compromised ones. If failing to verify your software
package is ok to you, then you may disable verification by adding
verify_failure_ok 'On'; to your Fetchwarefile. See perldoc App::Fetchware.
EOE
};
subtest 'test verify()' => sub {
skip_all_unless_release_testing();
# test verify_method
# test gpg verify_method
# test sha1 verify_method
# test md5 verify_method
# Specify a DownloadURL to test some gpg_verify() guessing magic.
for my $verify_method (qw(gpg sha md5)) {
config_replace('verify_method', "$verify_method");
eval {verify($download_path, $package_path)};
unless ($@) {
pass("checked verify() verify_method $verify_method");
} else {
fail("checked verify() verify_method $verify_method");
}
}
# test using copied gpg_verify setup from above.
eval {verify($download_path, $package_path)};
note("exe[$@]");
unless ($@) {
pass("checked verify() automatic method gpg");
} else {
fail("checked verify() automatic method gpg");
}
# test for skiping gpg & using sha1. Can't find a site that does this.
###BUGALERT### Figure out how to test for this. I may have to wait until I
#implement testing webserver to download files from using maybe
#Test::Fake::HTTPD or something else.
###HOWTOTEST?? eval {verify()};
###HOWTOTEST?? unless ($@) {
###HOWTOTEST?? pass("checked verify() automatic method sha");
###HOWTOTEST?? } else {
###HOWTOTEST?? fail("checked verify() automatic method sha");
###HOWTOTEST?? }
# test using just a plain old md5sum.
t/App-Fetchware-verify.t view on Meta::CPAN
note("vfo[@{[config('verify_failure_ok')]}]");
is(verify('ftp://fake.url/doesnt/exist.ever', $package_path),
'warned due to verify_failure_ok',
'checked verify() verify_failure_ok');
# Test an invalid verify_method.
verify_method 'invalid';
eval_ok(sub {verify($download_path, $package_path)},
<<EOE, 'checked verify() invalid verify_method');
App-Fetchware: run-time error. Your fetchware file specified a wrong
verify_method option. The only supported types are 'gpg', 'sha', 'md5', but you
specified [invalid]. See perldoc App::Fetchware.
EOE
config_delete('verify_method');
};
subtest 'Call end() to clean up temporary directory.' => sub {
# Skip this simple test, because start() is only caled with the other
# prereques, which are skipped except during release testing.
t/bin-fetchware-Fetchwarefile.t view on Meta::CPAN
my $apache_fetchwarefile = <<'EOF';
use App::Fetchware;
program 'Apache';
lookup_url 'http://www.apache.org/dist/httpd/';
filter 'httpd-2.2';
mirror 'http://mirrors.ibiblio.org/apache/httpd/';
mirror 'ftp://apache.cs.utah.edu/apache.org/httpd/';
verify_method 'gpg';
gpg_keys_url 'http://www.apache.org/dist/httpd/KEYS';
make_options '-j 4';
prefix '/home/dly/software/apache2.2';
# You can use heredocs to make gigantic options like this one more legible.
configure_options <<EOO;
--with-mpm=prefork
--enable-modules="access alias auth autoindex cgi logio log_config status vhost_alias userdir rewrite ssl"
--enable-so
EOO
EOF
t/bin-fetchware-Fetchwarefile.t view on Meta::CPAN
# lookup_url and mirror are the same thing, because nginx does not seem to have
# mirrors. Fetchware, however, requires one, so the same URL is simply
# duplicated.
lookup_url 'http://nginx.org/download/';
mirror 'http://nginx.org/download/';
# Must add the developers public keys to my own keyring. These keys are
# availabe from http://nginx.org/en/pgp_keys.html Do this with:
# gpg \
# --fetch-keys http://nginx.org/keys/aalexeev.key\
# --fetch-keys http://nginx.org/keys/is.key\
# --fetch-keys http://nginx.org/keys/mdounin.key\
# --fetch-keys http://nginx.org/keys/maxim.key\
# --fetch-keys http://nginx.org/keys/sb.key\
# --fetch-keys http://nginx.org/keys/glebius.key\
# --fetch-keys http://nginx.org/keys/nginx_signing.key
# You might think you could just set gpg_keys_url to the nginx-signing.key key,
# but that won't work, because like apache different releases are signed by
# different people. Perhaps I could change gpg_keys_url to be like mirror where
# you can specify more than one option?
user_keyring 'On';
# user_keyring specifies to use the user's own keyring instead of fetchware's.
# But fetchware drops privileges by default using he user 'nobody.' nobody is
# nobody, so that user account does not have a home directory for gpg to read a
# keyring from. Therefore, I'm using my own account instead.
user 'dly';
# The other option, which is commented out below, is to use root's own keyring,
# and the no_install option to ensure that root uses its own keyring instead of
# nobody's.
# noinstall 'On';
verify_method 'gpg';
EOF
# Create a tempfile to store the Fetchwarefile in.
my ($fh, $filename) = tempfile("fetchware-test-$$-XXXXXXXXXXX", TMPDIR => 1,
UNLINK => 1);
# Write the $apache_fetchwarefile to disk, so bin/fetchware can access it.
print $fh $nginx_fetchwarefile;
close $fh; # Close $fh to ensure its contents make it out to disk.
# Just execute bin/fetchware install with the newly created
t/bin-fetchware-Fetchwarefile.t view on Meta::CPAN
# The directory where the php source code's local git repo is.
my $git_repo_dir = '/home/dly/Desktop/Code/php-src';
# By default Fetchware drops privs, and since the source code repo is stored in
# the user dly's home directory, I should drop privs to dly, so that I have
# permission to access it.
user 'dly';
# Turn on verify failure ok, because the current version as of this writing (php
# 5.4.5) is signed by someone who has not shared uploaded their gpg key to a
# keyserver somewhere making git verifgy-tag fail, because I can't find the key.
# Hopefully, php bug# 65840 will result in this being fixed.
verify_failure_ok 'On';
# Determine latest version by using the tags developers create to determine the
# latest version.
hook lookup => sub {
# chdir to git repo.
chdir $git_repo_dir or die <<EOD;
php.Fetchwarefile: Failed to chdir to git repo at
t/bin-fetchware-Fetchwarefile.t view on Meta::CPAN
my $latest_tag = $download_path;
# checkout the $latest_tag to download it.
run_prog('git checkout', "$latest_tag");
my $package_path = cwd();
return $package_path;
};
# You must manually add php's developer's gpg keys to your gpg keyring. Do
# this by going to the page: http://us1.php.net/downloads.php . At the
# bottom the gpg key "names are listed such as "7267B52D" or "5DA04B5D."
# These are their key "names." Use gpg to download them and import them into
# your keyring using: gpg --keyserver pgp.mit.edu --recv-keys [key id]
hook verify => sub {
my ($download_path, $package_path) = @_;
# the latest tag is the download path see lookup.
my $latest_tag = $download_path;
# Run git verify-tag to verify the latest tag
my $success = eval { run_prog('git verify-tag', "$latest_tag"); 1;};
# If the git verify-tag fails, *and* verify_failure_ok has been turned on,
t/bin-fetchware-new.t view on Meta::CPAN
##TODO####BROKEN##
##TODO####BROKEN## expect_send("$ENV{FETCHWARE_HTTP_MIRROR_URL}",
##TODO####BROKEN## 'checked cmd_new() say mirror.');
##TODO####BROKEN##
##TODO####BROKEN## expect_like(qr/In addition to the one required mirror that you must define in order for/ms,
##TODO####BROKEN## 'checked cmd_new() received more mirrors prompt.');
##TODO####BROKEN##
##TODO####BROKEN## expect_send('N',
##TODO####BROKEN## 'checked cmd_new() say N for more mirrors.');
##TODO####BROKEN##
##TODO####BROKEN## #expect_like(qr!\[y/N\]|gpg digital signatures found. Using gpg verification.!ms,
##TODO####BROKEN## expect_like(qr!.*|gpg digital signatures found. Using gpg verification.!ms,
##TODO####BROKEN## 'checked cmd_new() received filter prompt.');
##TODO####BROKEN##
##TODO####BROKEN## expect_send('httpd-2.2',
##TODO####BROKEN## 'checked cmd_new() say httpd-2.2 for filter option.');
##TODO####BROKEN##
##TODO####BROKEN## expect_like(qr/Fetchware has many different configuration options that allow you to control its/ms,
##TODO####BROKEN## 'checked cmd_new() received extra config prompt.');
##TODO####BROKEN##
##TODO####BROKEN## expect_send('N',
##TODO####BROKEN## 'checked cmd_new() say N for more config options prompt.');
t/bin-fetchware-util.t view on Meta::CPAN
withdraw('install');
EOS
eval_ok(sub {parse_fetchwarefile(\$api_subs_exported)},
qr/fetchware: The App::Fetchware module you choose in your fetchwarefile does not/,
'checked parse_fetchwarefile() failed to export api subs.');
};
subtest 'test create_fetchware_package()' => sub {
###BUGALERT### Must add tests for adding the gpg generated files to the
#fetchware package, so that gpg doesn't have to download the keys again.
#Also, I must actually add code for this in bin/fetchware.
my $fetchwarefile = '# Fake Fetchwarefile for testing';
# Create a hopefully successful fetchware package using the current working
# directory (my Fetchware git checkout) and the fake Fetchwarefile I created
# above.
my $cwd = dir(cwd());
my $cwd_parent = $cwd->parent();
my $cwd_lastdir = $cwd->dir_list(-1, 1);
( run in 1.432 second using v1.01-cache-2.11-cpan-df04353d9ac )