view release on metacpan or  search on metacpan

lib/App/Fetchware/Util.pm  view on Meta::CPAN

EOD
        # Clear possibly tainted variables. It's a weird bug that makes no
        # sense. I don't turn -t or -T on, so what gives??? If you're curious
        # try commenting out the taint clearing code below, and running the
        # t/bin-fetchware-install.t test file (Or any other ones that call
        # drop_privs().).
        my $untainted;
        # Need the m//ms options to match strings with newlines in them.
        if ($variable =~ /(.*)/ms) {
            $untainted = $1;
        } else {
            die <<EOD;
App::Fetchware::Util: Untaint failed! Huh! This just shouldn't happen! It's
probably a bug. 
EOD
        }

        # Push $untainted instead of just $variable, because I want to return
        # untatined data instead of potentially tainted data.
        push @variables, $untainted;
    }

    return @variables;
}
###BUGALERT### Add some pipe parsers that use Storable too.

} # End $MAGIC_NUMBER bare block.







sub do_nothing {
    return;
}






{ # Begin scope block for $original_cwd.

    # $original_cwd is a scalar variable that stores fetchware's original
    # working directory for later use if its needed. It is access with
    # original_cwd() below.
    my $original_cwd;
    # $fh_sem is a semaphore lock file that create_tempdir() creates, and
    # cleanup_tempdir() closes clearing the lock. This is used to support
    # fetchware clean. The filehandle needs to be declared outside
    # create_tempdir()'s scope, because when this filehandle goes out of scope
    # the file is closed, and the lock is released, but fetchware needs to keep
    # hold of this lock for the life of fetchware to ensure that any fetchware
    # clean won't delete this fetchware temporary directory.
    my $fh_sem;


###BUGALERT### Add support for the -f/--force option to force deleting fetchware
#temp dirs even if locked.
sub create_tempdir {
    my %opts = @_;

    msg 'Creating temp dir to use to install your package.';

    # Ask for better security.
    File::Temp->safe_level( File::Temp::HIGH );

    # Create the temp dir in the portable locations as returned by
    # File::Spec->tempdir() using the specified template (the weird $$ is this
    # processes process id), and cleaning up at program exit.
    my $exception = '';
    my $temp_dir;
    eval {
        local $@;

        # Determine tempdir()'s arguments.
        my @args = ("fetchware-$$-XXXXXXXXXX");#, TMPDIR => 1);

        # Specify the caller's TempDir (DIR) if they specify it.
        push @args, DIR => $opts{TempDir} if defined $opts{TempDir};

        # Specify either system temp directory or user specified directory.
        push @args,
            (defined $opts{TempDir} ? (DIR => $opts{TempDir}) : (TMPDIR => 1));

        # Don't CLEANUP if KeepTempDir is set.
        push @args, CLEANUP => 1 if not defined $opts{KeepTempDir};

        # Call tempdir() with the @args I've built.
        $temp_dir = tempdir(@args);

        # Only when we do *not* drop privs...
        if (config('stay_root')
                or ($< != 0 or $> != 0)
        ) {
            # ...Must chmod 700 so gpg's localized keyfiles are good.
            chmod(0700, $temp_dir) or die <<EOD;
App-Fetchware-Util: Fetchware failed to change the permissions of its temporary
directory [$temp_dir] to 0700. This should not happen, and is a bug, or perhaps
your system's temporary directory is full. The OS error was [$!].
EOD
        }

        $exception = $@;
        1; # return true unless an exception is thrown.
    } or die <<EOD;
App-Fetchware: run-time error. Fetchware tried to use File::Temp's tempdir()
subroutine to create a temporary file, but tempdir() threw an exception. That
exception was [$exception]. See perldoc App::Fetchware.
EOD

    $original_cwd = cwd();
    vmsg "Saving original working directory as [$original_cwd]";

    # Change directory to $CONFIG{TempDir} to make unarchiving and building happen
    # in a temporary directory, and to allow for multiple concurrent fetchware
    # runs at the same time.
    chdir $temp_dir or die <<EOD;
App-Fetchware: run-time error. Fetchware failed to change its directory to the
temporary directory that it successfully created. This just shouldn't happen,
and is weird, and may be a bug. See perldoc App::Fetchware.
EOD
    vmsg "Successfully changed working directory to [$temp_dir].";

    # Create 'fetcwhare.sem' - the fetchware semaphore lock file.
    open $fh_sem, '>', 'fetchware.sem' or die <<EOD;
App-Fetchware-Util: Failed to create [fetchware.sem] semaphore lock file! This
should not happen, because fetchware is creating this file in a brand new
directory that only fetchware should be accessing. You simply shouldn't see this
error unless some one is messing with fetchware, or perphaps there actually is a
bug? I don't know, but this just shouldn't happen. It's so hard to trigger it to
happen, it can't easily be tested in fetchware's test suite. OS error [$!].
EOD
    vmsg "Successfully created [fetchware.sem] semaphore lock file.";
    # Now flock 'fetchware.sem.' This should
    # Use LOCK_NB so flock won't stupidly wait forever and ever until the lock
    # becomes available.
    flock $fh_sem, LOCK_EX | LOCK_NB or die <<EOD;
App-Fetchware-Util: Failed to flock [fetchware.sem] semaphore lock file! This
should not happen, because this is being done in a brand new temporary directory
that only this instance of fetchware cares about. This just shouldn't happen. OS
error [$!].
EOD
    vmsg "Successfully locked [fetchware.sem] semaphore lock file using flock.";

    msg "Temporary directory created [$temp_dir]";

    return $temp_dir;
}



    sub original_cwd {
        return $original_cwd;
    }



sub cleanup_tempdir {
    msg 'Cleaning up temporary directory temporary directory.';

    # Close and unlock the fetchware semaphore lock file, 'fetchware.sem.'
    if (defined $fh_sem) {
        close $fh_sem or die <<EOD;
App-Fetchware-Util: Huh? close() failed! Fetchware failed to close(\$fh_sem).
Perhaps some one or something deleted it under us? Maybe a fetchware clean was
run with the force flag (--force) while this other fetchware was running?
OS error [$!].
EOD
        vmsg <<EOM;
Closed [fetchware.sem] filehandle to unlock this fetchware temporary directory from any
fetchware clean runs.
EOM
    }

    # chdir to original_cwd() directory, so File::Temp can delete the tempdir.
    # This is necessary, because operating systems do not allow you to delete a
    # directory that a running program has as its cwd.
    if (defined(original_cwd())) {
        vmsg "Changing directory to [@{[original_cwd()]}].";
        chdir(original_cwd()) or die <<EOD;
App-Fetchware: run-time error. Fetchware failed to chdir() to
[@{[original_cwd()]}]. See perldoc App::Fetchware.
EOD
    }

    # cleanup_tempdir() used to actually delete the temporary directory by using
    # File::Temp's cleanup() subroutine, but that subroutine deletes *all*
    # temporary directories that File::Temp has created and marked for deletion,
    # which might include directories created before this call to
    # cleanup_tempdir(), but are needed after. Therefore, cleanup_tempdir() no
    # longer actually deletes anything; instead, File::Temp can do it in its END
    # handler.
    #
    # The code below is left here on purpose, to remind everyone *not* to call
    # File::Temp's cleanup() here!! Do not do it!
    ###DONOTDO#### Call File::Temp's cleanup subrouttine to delete fetchware's temp
    ###DONOTDO#### directory.
    ###DONOTDO###vmsg 'Cleaning up temporary directory.';
    ###DONOTDO###File::Temp::cleanup();

    vmsg "Leaving tempdir alone. File::Temp's END handler will delete it.";

    vmsg 'Clearing internal %CONFIG variable that hold your parsed Fetchwarefile.';

lib/App/Fetchware/Util.pm  view on Meta::CPAN

Returns an scalar of the HTML ladden directory listing.

If an even number of other options are specified (a faux hash), then those
options are forwarded on to L<HTTP::Tiny>'s new() method. See L<HTTP::Tiny> for
details about what these options are. For example, you couse use this to add a
C<Referrer> header to your request if a download site annoying checks referrers.

=head2 file_download_dirlist()

    my $file_listing = file_download_dirlist($local_lookup_url)

Glob's provided $local_lookup_url, and builds a directory listing of all files
in the provided directory.

=head2 download_file()

    my $filename = download_file($url)

    my $filename = download_file(PATH => $path)

Can be called with either a $url or a PATH parameter. When called with a $url
parameter, the specified $url is downloaded using no_mirror_download_file(),
and returned if successful. If it fails then each C<mirror> the user specified
is also tried unitl there are no more mirrors, and then an exception is thrown.

If you specify a PATH parameter instead of a $url parameter, then that path is
appended to each C<mirror>, and the resultant url is downloaded using
no_mirror_download_file().

=head2 no_mirror_download_file()

    my $filename = no_mirror_download_file($url)

Downloads one $url and assumes it is a file that will be downloaded instead of a
file listing that will be returned. no_mirror_download_file() returns the file
name of the file it downloads.

Like its name says it does not try any configured mirrors at all. This
subroutine should not be used; instead download_file() should be used, because
you should respect your user's desired mirrors.

=head2 download_ftp_url()

    my $filename = download_ftp_url($url);

Uses Net::FTP to download the specified FTP URL using binary mode.

=head2 download_http_url()

    my $filename = download_http_url($url);

Uses HTTP::Tiny to download the specified HTTP URL.

Supports adding extra arguments to HTTP::Tiny's new() constructor. These
arguments are B<not> checked for correctness; instead, they are simply forwarded
to HTTP::Tiny, which does not check them for correctness either. HTTP::Tiny
simply loops over its internal listing of what is arguments should be, and then
accesses the arguments if they exist.

This was really only implemented to allow App::FetchwareX::HTMLPageSync to change
its user agent string to avoid being blocked or freaking out Web developers that
they're being screen scraped by some obnoxious bot as HTMLPageSync is wimpy and
harmless, and only downloads one page. 

You would add an argument like this:
download_http_url($http_url, agent => 'Firefox');

See HTTP::Tiny's documentation for what these options are.

=head2 download_file_url()

    my $filename = download_file_url($url);

Uses File::Copy to copy ("download") the local file to the current working
directory.

=head1 TEMPDIR SUBROUTINES

These subroutines manage the creation of a temporary directory for you. They
also implement the original_cwd() getter subroutine that returns the current
working directory fetchware was at before create_tempdir() chdir()'d to the
temporary directory you specify. File::Temp's tempdir() is used, and
cleanup_tempdir() manages the C<fetchware.sem> fetchware semaphore file.

=over 
=item NOTICE
App::Fetchware::Util's temporary directory creation utilities, create_tempdir(),
original_cwd(), and cleanup_tempdir(), only keep track of one tempdir at a time. If
you create another tempdir with create_tempdir() it will override the value of
original_cwd(), which may mess up other functions that call create_tempdir(),
original_cwd(), and cleanup_tempdir(). Therefore, becareful when you call these
functions, and do B<not> use them inside a fetchware extension if you reuse
App::Fetchware's start() and end(), because App::Fetchware's start() and end()
use these functions, so your use of them will conflict. If you still need to
create a tempdir just call File::Temp's tempdir() directly.

=back

=head2 create_tempdir()

    my $temp_dir = create_tempdir();

Creates a temporary directory, chmod 700's it, and chdir()'s into it.

Accepts the fake hash argument C<KeepTempDir => 1>, which tells create_tempdir()
to B<not> delete the temporary directory when the program exits.

Also, accepts C<TempDir =E<gt> '/tmp'> to specify what temporary directory to
use. The default with out this argument is to use tempdir()'s default, which is
whatever File::Spec's tmpdir() says to use.

The C<NoChown =E<gt> 1> option causes create_tempdir() to B<not> chown to
config('user').

=head3 Locking Fetchware's temp directories with a semaphore file.

In order to support C<fetchware clean>, create_tempdir() creates a semaphore
file. The file is used by C<fetchware clean> (via bin/fetchware's cmd_clean())
to determine if another fetchware process out there is currently using this
temporary directory, and if it is not, the file is not currently locked with
flock, then the entire directory is deleted using File::Path's remove_path()
function. If the file is there and locked, then the directory is skipped by
cmd_clean().

cleanup_tempdir() is responsible for unlocking the semaphore file that
create_tempdir() creates. However, the coolest part of using flock is that if
fetchware is killed in any manner whether its C<END> block or File::Temp's
C<END>block run, the OS will still unlock the file, so no edge cases need
handling, because the OS will do them for us!

=head2 original_cwd()

    my $original_cwd = original_cwd();

original_cwd() simply returns the value of fetchware's $original_cwd that is
saved inside each create_tempdir() call. A new call to create_tempdir() will
reset this value. Note: App::Fetchware's start() also calls create_tempdir(), so
another call to start() will also reset original_cwd().

=head2 cleanup_tempdir()

    cleanup_tempdir();

Cleans up B<any> temporary files or directories that anything in this process used
File::Temp to create. You cannot only clean up one directory or another;
instead, you must just use this sparingly or in an END block although file::Temp
takes care of that for you unless you asked it not to.

It also closes $fh_sem, which is the filehandle of the 'fetchware.sem' file
create_tempdir() opens and I<locks>. By closing it in cleanup_tempdir(), we're
unlocking it. According to MJD's "File Locking Tips and Traps," it's better to
just close the file, then use flock to unlock it.

=head1 SECURITY SUBROUTINES

This section describes Utilty subroutines that can be used for checking security
of files on the file system to see if fetchware should open and use them.

=head2 safe_open()

    my $fh = safe_open($file_to_check, <<EOE);
    App-Fetchware-Extension???: Failed to open file [$file_to_check]! Because of
    OS error [$!].
    EOE

    # To open for writing instead of reading 
    my $fh = safe_open($file_to_check, <<EOE, MODE => '>');
    App-Fetchware-Extension???: Failed to open file [$file_to_check]! Because of
    OS error [$!].
    EOE

safe_open() takes $file_to_check and does a bunch of file checks on that
file to determine if it's safe to open and use the contents of that file in
your program. Instead of returning true or false, it returns a file handle of
the file you want to check that has already been open for you. This is done to
prevent race conditions between the time safe_open() checks the file's safety
and the time the caller actually opens the file.

safe_open() also takes an optional second argument that specifies a caller
specific error message that replaces the generic default one.

Fetchware occasionally needs to write files especially in fetchware's new()