Brackup
view release on metacpan or search on metacpan
1.10 (2010-10-31)
- permit 0 as a filename. https://rt.cpan.org/Ticket/Display.html?id=62004
- add Riak target, allowing backups to a riak cluster (Gavin Carr)
- add uid/gid info to metafile, and use in restores (where possible)
(Gavin Carr)
- allow multiple gpg recipients to be specified, any can restore (Alex
Vandiver)
- if IO::Compress::Gzip is available, write a compressed brackup metafile in
unencrypted mode, and handle properly for reads and restores (Gavin Carr)
- remove orphaned chunks from inventory as part of garbage collection
(Gavin Carr)
- add simple config section inheritance ('inherit' directive) (Gavin Carr)
compatible with the old layout. also, if max-link count (max
files in a directory) is hit, this new version will carefully
rearrange the minimum files necessary to the new layout to
make room, all automatically. the new format is xx/xx/*,
rather than xxxx/xxxx/xxxx/xxxx/* which was stupid and overkill.
stupid because that's 65k of files in the root, twice ext3's
limit, and overkill because leaves were always just 1 file.
thanks to Max Kanat-Alexander for pointing this out, and
part of the patch to use new layout pattern.
- quieter (no) error messages on death/control-C from gpg child
processes who were previously confused by their parent processes
going away and cleaning up their shared temp files.
- actually respect the --just flag on restore
1.05 (2007-08-02)
- 'prune' and 'gc' commands commands for both Amazon
and Filesystem targets. from Alessandro Ranellucci <aar@cpan.org>.
target too), from Alessandro Ranellucci <aar@cpan.org>
- make tests pass on OS X (Jesse Vincent)
1.03 (2007-05-23)
- brackup-restore's verbose flag is more verbose now, showing files
as they're restored.
- brackup-restore can restore from an encrypted *.brackup file now,
firing up gpg for user to decrypt to a tempfile
- brackup-target tool, to list/get brackup files from a target,
and in the future do garbage collection on no-longer-referenced
chunks (once a command exists to delete a brackup file from a target)
- stop leaking temp files
- doc fixes/additions
1.02 (2007-05-22)
- don't upload meta files when in dry-run mode
- update amazon target support to work again, with the new inventory
database support (now separated from the old digest database)
- merge in the refactoring branch, in which a lot of long-standing
pet peeves in the design were rethought/redone.
- make decryption --use-agent and --batch, and help out if env not set
and gpg-agent probably not running
- support putting .meta files besides .chunk files on the Target
to enable reconstructing the digest database in the future, should
it get lost. also start to flesh out per-chunk digests, which
would enable backing up large databases (say, InnoDB tablespaces) where
large chunks of the file never change.
- new --du-stats to command to act like the du(1) command, but
based on a root in brackup.conf, and skipping ignored directories.
good to let you know how big a backup will be.
ahead of time w/o errors/warnings later.
- start of stats code (to give stats after a backup). not done.
0.91 (2006-09-29)
- there's now a restore command (brackup-restore)
- amazon restore support
- use gpg --trust-model=always for new gpg that is more paranoid.
- mostly usable. some more switches would be nice later. real
1.00 release will come after few weeks/months of testing/tweaks.
0.80
- restore works
- lot more tests
- notable bug fix with encrypted backups. metafiles could have wrong sizes.
lib/Brackup/Target/Riak.pm
lib/Brackup/Target/Sftp.pm
lib/Brackup/TargetBackupStatInfo.pm
lib/Brackup/Test.pm
lib/Brackup/Util.pm
t/00-use.t
t/01-backup-filesystem.t
t/01-backup-ftp.t
t/01-backup-riak.t
t/01-backup-sftp.t
t/02-gpg.t
t/03-composite-filesystem.t
t/03-composite-ftp.t
t/03-composite-sftp.t
t/04-gc-filesystem.t
t/04-gc-ftp.t
t/04-gc-sftp.t
t/05-filename-escaping.t
t/06-config-inheritance.t
t/data-2/000-dup1.txt
t/data-2/000-dup2.txt
t/data-2/README
t/data-2/gzip.txt.gz
t/data-2/my-link.txt
t/data-2/my_dir/sub_dir/another-file.txt
t/data-2/my_dir/sub_dir/program.sh
t/data-2/pubring-test.gpg
t/data-2/pubring-test2.gpg
t/data-2/secring-test.gpg
t/data-2/secring-test2.gpg
t/data-2/test-file.txt
't/data-weird-filenames/filename whitespace'
't/data-weird-filenames/filename_trailing_whitespace '
t/data-weird-filenames/000-dup1.txt
t/data-weird-filenames/000-dup2.txt
t/data-weird-filenames/huge-file.txt
t/data-weird-filenames/my-link.txt
t/data-weird-filenames/pubring-test.gpg
t/data-weird-filenames/secring-test.gpg
t/data-weird-filenames/test-file.txt
t/data/000-dup1.txt
t/data/000-dup2.txt
t/data/gzip.txt.gz
t/data/huge-file.txt
t/data/my-link.txt
t/data/my_dir/sub_dir/another-file.txt
t/data/my_dir/sub_dir/program.sh
t/data/pubring-test.gpg
t/data/pubring-test2.gpg
t/data/secring-test.gpg
t/data/secring-test2.gpg
t/data/test-file.txt
t/misc/brackup.conf
META.yml Module meta-data (added by MakeMaker)
doc/data-structures.txt view on Meta::CPAN
The keys/values used are:
<FileCacheKey> --> <TypedDigest(original_unencrypted_file)>
<ChunkCacheKey> --> <ChunkDetails>
Where:
FileCacheKey ::= "[" <RootName> "]" <FileRelativePath> ":" join(",", <ctime>, <mtime>, <size>, <inode>)
ChunkCacheKey ::= <TypedDigest(original_unencrypted_file)> "-" <raw_offset> "-" <raw_length> "-" <gpg-recipient>
ChunkDetails ::= <EncryptedLength> " " <TypedDigest(encrypted_chunk)>
TypedDigest ::= <DigestAlgo> ":" <hex_digest>
DigestAlgo ::= { "sha1" }
----------------------------------------------------------------------------
[backup-name].brackup format (RFC-822-like)
doc/exampleconfig.txt view on Meta::CPAN
# a sample ~/.brackup.conf
[TARGET:raidbackups]
type = Filesystem # this can be any Brackup::Target::<foo> subclass
path = /raid/backup/brackup
[SOURCE:proj]
path = /raid/bradfitz/proj/
chunk_size = 5m
gpg_recipient = 5E1B3EC5
[SOURCE:bradhome]
chunk_size = 64MB
path = /raid/bradfitz/
ignore = ^\.thumbnails/
ignore = ^\.kde/share/thumbnails/
ignore = ^\.ee/minis/
ignore = ^build/
ignore = ^(gqview|nautilus)/thumbnails/
lib/Brackup/Backup.pm view on Meta::CPAN
# returns true (a Brackup::BackupStats object) on success, or dies with error
sub backup {
my ($self, $backup_file) = @_;
my $root = $self->{root};
my $target = $self->{target};
my $stats = Brackup::BackupStats->new;
my @gpg_rcpts = $self->{root}->gpg_rcpts;
my $n_kb = 0.0; # num: kb of all files in root
my $n_files = 0; # int: # of files in root
my $n_kb_done = 0.0; # num: kb of files already done with (uploaded or skipped)
# if we're pre-calculating the amount of data we'll
# actually need to upload, store it here.
my $n_files_up = 0;
my $n_kb_up = 0.0;
my $n_kb_up_need = 0.0; # by default, not calculated/used.
lib/Brackup/Backup.pm view on Meta::CPAN
}
warn "kb need to upload = $n_kb_up_need\n";
$stats->timestamp('Calc Needed');
}
my $chunk_iterator = Brackup::ChunkIterator->new(@files);
undef @files;
$stats->timestamp('Chunk Iterator');
my $gpg_iter;
my $gpg_pm; # gpg ProcessManager
if (@gpg_rcpts) {
($chunk_iterator, $gpg_iter) = $chunk_iterator->mux_into(2);
$gpg_pm = Brackup::GPGProcManager->new($gpg_iter, $target);
}
# begin temp backup_file
my ($metafh, $meta_filename);
unless ($self->{dryrun}) {
($metafh, $meta_filename) = tempfile(
'.' . basename($backup_file) . 'XXXXX',
DIR => dirname($backup_file),
);
if (! @gpg_rcpts) {
if (eval { require IO::Compress::Gzip }) {
close $metafh;
$metafh = IO::Compress::Gzip->new($meta_filename)
or die "Cannot open tempfile with IO::Compress::Gzip: $IO::Compress::Gzip::GzipError";
}
}
print $metafh $self->backup_header;
}
my $cur_file; # current (last seen) file
lib/Brackup/Backup.pm view on Meta::CPAN
$cur_file->path, $n_files_done, $n_files, $percdone,
$mb_remain));
$self->report_progress($percdone);
};
my $start_file = sub {
$end_file->();
$cur_file = shift;
@stored_chunks = ();
$show_status->() if $cur_file->is_dir;
if ($gpg_iter) {
# catch our gpg iterator up. we want it to be ahead of us,
# nothing iteresting is behind us.
$gpg_iter->next while $gpg_iter->behind_by > 1;
}
$file_has_shown_status = 0;
};
# records are either Brackup::File (for symlinks, directories, etc), or
# PositionedChunks, in which case the file can asked of the chunk
while (my $rec = $chunk_iterator->next) {
if ($rec->isa("Brackup::File")) {
$start_file->($rec);
next;
lib/Brackup/Backup.pm view on Meta::CPAN
$show_status->();
$n_files_up++;
}
$self->debug(" * storing chunk: ", $pchunk->as_string, "\n");
$self->report_progress(undef, $pchunk->file->path . " (" . $pchunk->offset . "," . $pchunk->length . ")");
unless ($self->{dryrun}) {
$schunk = Brackup::StoredChunk->new($pchunk);
# encrypt it
if (@gpg_rcpts) {
$schunk->set_encrypted_chunkref($gpg_pm->enc_chunkref_of($pchunk));
}
# see if we should pack it into a bigger blob
my $chunk_size = $schunk->backup_length;
# see if we should merge this chunk (in this case, file) together with
# other small files we encountered earlier, into a "composite chunk",
# to be stored on the target in one go.
# Note: no technical reason for only merging small files (is_entire_file),
lib/Brackup/Backup.pm view on Meta::CPAN
# store it regularly, as its own chunk on the target
$target->store_chunk($schunk)
or die "Chunk storage failed.\n";
$target->add_to_inventory($pchunk => $schunk);
}
# if only this worked... (LWP protocol handler seems to
# get confused by its syscalls getting interrupted?)
#local $SIG{CHLD} = sub {
# print "some child finished!\n";
# $gpg_pm->start_some_processes;
#};
$n_kb_up += $pchunk->length / 1024;
$schunk->forget_chunkref;
push @stored_chunks, $schunk;
}
#$stats->note_stored_chunk($schunk);
lib/Brackup/Backup.pm view on Meta::CPAN
unless ($self->{dryrun}) {
close $metafh or die "Close on metafile '$backup_file' failed: $!";
rename $meta_filename, $backup_file
or die "Failed to rename temporary backup_file: $!\n";
my ($store_fh, $store_filename);
my $is_encrypted = 0;
# store the metafile, encrypted, on the target
if (@gpg_rcpts) {
my $encfile = $backup_file . ".enc";
my @recipients = map {("--recipient", $_)} @gpg_rcpts;
system($self->{root}->gpg_path, $self->{root}->gpg_args,
@recipients,
"--trust-model=always",
"--batch",
"--encrypt",
"--output=$encfile",
"--yes",
$backup_file)
and die "Failed to run gpg while encryping metafile: $!\n";
open ($store_fh, $encfile) or die "Failed to open encrypted metafile '$encfile': $!\n";
$store_filename = $encfile;
$is_encrypted = 1;
} else {
# Reopen $metafh to reset file pointer (no backward seek with IO::Compress::Gzip)
open($store_fh, $backup_file) or die "Failed to open metafile '$backup_file': $!\n";
$store_filename = $backup_file;
}
# store it on the target
lib/Brackup/Backup.pm view on Meta::CPAN
}
$ret .= "RootName: " . $self->{root}->name . "\n";
$ret .= "RootPath: " . $self->{root}->path . "\n";
$ret .= "TargetName: " . $self->{target}->name . "\n";
$ret .= "DefaultFileMode: " . $self->default_file_mode . "\n";
$ret .= "DefaultDirMode: " . $self->default_directory_mode . "\n";
$ret .= "DefaultUID: " . $self->default_uid . "\n";
$ret .= "DefaultGID: " . $self->default_gid . "\n";
$ret .= "UIDMap: " . $self->uid_map . "\n";
$ret .= "GIDMap: " . $self->gid_map . "\n";
$ret .= "GPG-Recipient: $_\n" for $self->{root}->gpg_rcpts;
$ret .= "\n";
return $ret;
}
sub record_mode_ids {
my ($self, $file) = @_;
$self->{modecounts}{$file->type}{$file->mode}++;
$self->{idcounts}{u}{$file->uid}++;
$self->{idcounts}{g}{$file->gid}++;
}
lib/Brackup/Config.pm view on Meta::CPAN
#[TARGET:amazon]
#type = Amazon
#aws_access_key_id = XXXXXXXXXX
#aws_secret_access_key = XXXXXXXXXXXX
#keep_backups = 10
#[SOURCE:proj]
#path = /raid/bradfitz/proj/
#chunk_size = 5m
#gpg_recipient = 5E1B3EC5
#[SOURCE:bradhome]
#path = /raid/bradfitz/
#noatime = 1
#chunk_size = 64MB
#ignore = ^\.thumbnails/
#ignore = ^\.kde/share/thumbnails/
#ignore = ^\.ee/minis/
#ignore = ^build/
#ignore = ^(gqview|nautilus)/thumbnails/
lib/Brackup/Decrypt.pm view on Meta::CPAN
warn "Decrypted ${filename} to ${new_file}.\n";
}
return $new_file;
}
return undef;
}
# Decrypt a file into a new file
# Return the new file's name, or undef.
our $warned_about_gpg_agent = 0;
sub decrypt_file {
my ($encrypted_file,%opts) = @_;
my $no_batch = delete $opts{no_batch};
my $meta = delete $opts{meta};
croak("Unknown options: " . join(', ', keys %opts)) if %opts;
# find which key we're using to decrypt it
if ($meta) {
my $rcpt = $meta->{"GPG-Recipient"} or
return undef;
}
unless ($ENV{'GPG_AGENT_INFO'} ||
@Brackup::GPG_ARGS ||
$warned_about_gpg_agent++)
{
my $err = q{
#
# WARNING: trying to restore encrypted files,
# but $ENV{'GPG_AGENT_INFO'} not present.
# Are you running gpg-agent?
#
};
$err =~ s/^\s+//gm;
warn $err;
}
my $output_temp = ( (tempfile())[1] || die );
my @list = ("gpg", @Brackup::GPG_ARGS,
"--use-agent",
!$opts{no_batch} ? ("--batch") : (),
"--trust-model=always",
"--output", $output_temp,
"--yes", "--quiet",
"--decrypt", $encrypted_file);
system(@list)
and die "Failed to decrypt with gpg: $!\n";
return $output_temp;
}
1;
lib/Brackup/InventoryDatabase.pm view on Meta::CPAN
=item B<2a) Exists on target; not in inventory database (without encryption)>
You re-upload it to the target, so you waste time & bandwidth, but no
extra disk space is wasted, and no chunks are orphaned. Actually,
chunks are un-orphaned, as the inventory database is now updated and
contains the chunk you just uploaded.
=item B<2b) Exists on target; not in inventory database (with encryption)>
When using encryption, each time a chunk is encrypted with gpg, the
contents are different. So if the inventory database says a given
chunk isn't already stored on the server, it will be re-encrypted and
stored (uploaded) again. You may or may not have an orphaned chunk on
the server, depending on whether or not it's referenced by any other
*.brackup meta files.
=back
For those reasons, it's somewhat important that your inventory
database be kept around and not deleted. If you're running brackup to
lib/Brackup/InventoryDatabase.pm view on Meta::CPAN
B<Keys>
The key is the digest of the "raw" (pre-compression/encryption)
file/chunk (with GPG recipient, if using encryption), and the value is
the digest of the chunk stored on the target, which contains the raw
chunk. The chunk stored on the target may contain other chunks, may
be compressed, encrypted, etc.
<raw_digest> --> <stored_digest> <stored_length>
<raw_digest>;to=<gpg_rcpt> --> <stored_digest> <stored_length>
For example:
sha1:e23c4b5f685e046e7cc50e30e378ab11391e528e;to=6BAFF35F =>
sha1:d7257184899c9e6c4e26506f1c46f8b6562d9ee7 71223
Means that the chunk with sha1 contents "e23c4...", intended to be
en/de-crypted for 6BAFF35F, can be got by asking the target for the
chunk with digest "d72571848...", with length 71,223 bytes.
When using the Brackup feature which combines small files into larger
blobs, the inventory database instead stores values like:
<raw_digest>[;to=<gpg_rcpt>] -->
<stored_digest> <stored_length> <from_offset>-<to_offset>
Which is the same thing, but after fetching the composite chunk using
the stored digest provided, only the range provided from C<from_offset> to
C<to_offset> should be used.
=head1 SEE ALSO
L<Brackup>
lib/Brackup/Manual/Overview.pod view on Meta::CPAN
your private key getting stolen. (however, you'd still worry about
your machine getting compromised for lots of other reasons...)
In any case, you encrypt files I<to yourself>, and this is a property
on a backup source (see L<Brackup::Root>). For example, in my config
file, I have:
[SOURCE:bradhome]
...
path = /home/bradfitz/
gpg_recipient = 5E1B3EC5
...
Where 5E1B3EC5 corresponds to the key signature for myself as seen in:
$ gpg --list-keys
...
pub 1024D/5E1B3EC5 2006-03-20
uid Brad Fitzpatrick <brad@danga.com>
....
While you backup automatically without a human present, a restore from
encryption requires an interactive session for you to enter your
private key's passphrase into gpg-agent.
To create a new key, run:
$ gpg --gen-key
But really, you should go read a gpg manual first. Notably, B<backing
up your gpg private key is very important!>. If you lose the disk
with your files which also contain your private key, your encrypted backups on
Amazon won't do you much good, since you'll have no way to decrypt them.
I recommend burning your private key to a CD, as well as printing it out
on paper. (Worst case you can type it back in, or use OCR.) Export with:
$ gpg --export-secret-keys --armor
You can encrypt to multiple keys by providing multiple C<gpg_recipient>
lines; any of the keys provided will be able to decrypt the backups.
=head1 Restores
To do a restore, you'll need your *.brackup file handy. If you lost
it, you can re-download it from your backup target with
L<brackup-target>. Then run:
brackup-restore --from=foo.brackup --to=<dir> --all
lib/Brackup/PositionedChunk.pm view on Meta::CPAN
return $self->{_raw_chunkref} = $ifh;
}
# useful string for targets to key on. of one of the forms:
# "<digest>;to=<enc_to>"
# "<digest>;raw"
# "<digest>;gz" (future)
sub inventory_key {
my $self = shift;
my $key = $self->raw_digest;
if (my @rcpts = $self->root->gpg_rcpts) {
$key .= ";to=@rcpts";
} else {
$key .= ";raw";
}
return $key;
}
sub forget_chunkref {
my $self = shift;
delete $self->{_raw_chunkref};
lib/Brackup/Root.pm view on Meta::CPAN
sub new {
my ($class, $conf) = @_;
my $self = bless {}, $class;
($self->{name}) = $conf->name =~ m/^SOURCE:(.+)$/
or die "No backup-root name provided.";
die "Backup-root name must be only a-z, A-Z, 0-9, and _." unless $self->{name} =~ /^\w+/;
$self->{dir} = $conf->path_value('path');
$self->{gpg_path} = $conf->value('gpg_path') || "gpg";
$self->{gpg_rcpt} = [ $conf->values('gpg_recipient') ];
$self->{chunk_size} = $conf->byte_value('chunk_size');
$self->{ignore} = [];
$self->{smart_mp3_chunking} = $conf->bool_value('smart_mp3_chunking');
$self->{merge_files_under} = $conf->byte_value('merge_files_under');
$self->{max_composite_size} = $conf->byte_value('max_composite_chunk_size') || 2**20;
die "'max_composite_chunk_size' must be greater than 'merge_files_under'\n" unless
$self->{max_composite_size} > $self->{merge_files_under};
$self->{gpg_args} = []; # TODO: let user set this. for now, not possible
$self->{digcache} = Brackup::DigestCache->new($self, $conf);
$self->{digcache_file} = $self->{digcache}->backing_file; # may be empty, if digest cache doesn't use a file
$self->{noatime} = $conf->value('noatime');
return $self;
}
sub merge_files_under { $_[0]{merge_files_under} }
sub max_composite_size { $_[0]{max_composite_size} }
sub smart_mp3_chunking { $_[0]{smart_mp3_chunking} }
sub gpg_path {
my $self = shift;
return $self->{gpg_path};
}
sub gpg_args {
my $self = shift;
return @{ $self->{gpg_args} };
}
sub gpg_rcpts {
my $self = shift;
return @{ $self->{gpg_rcpt} };
}
# returns Brackup::DigestCache object
sub digest_cache {
my $self = shift;
return $self->{digcache};
}
sub chunk_size {
my $self = shift;
lib/Brackup/Root.pm view on Meta::CPAN
next if $dentry eq "." || $dentry eq "..";
my $path = "$dir/$dentry";
$path =~ s!^\./!!;
# skip the digest database file. not sure if this is smart or not.
# for now it'd be kinda nice to have, but it's re-creatable from
# the backup meta files later, so let's skip it.
next if $self->{digcache_file} && $path eq $self->{digcache_file};
# GC: seems to work fine as of at least gpg 1.4.5, so commenting out
# gpg seems to barf on files ending in whitespace, blowing
# stuff up, so we just skip them instead...
#if ($self->gpg_rcpts && $path =~ /\s+$/) {
# warn "Skipping file ending in whitespace: <$path>\n";
# next;
#}
my $statobj = File::stat::lstat($path);
my $is_dir = -d _;
foreach my $pattern (@{ $self->{ignore} }) {
next DENTRY if $path =~ /$pattern/;
next DENTRY if $is_dir && "$path/" =~ /$pattern/;
lib/Brackup/Root.pm view on Meta::CPAN
$dir_size{$_} += $kB foreach @dir_stack;
}
});
$pop_dir->() while @dir_stack;
}
# given filehandle to data, returns encrypted data
sub encrypt {
my ($self, $data_fh, $outfn) = @_;
my @gpg_rcpts = $self->gpg_rcpts
or Carp::confess("Encryption not setup for this root");
my $cout = Symbol::gensym();
my $cin = Symbol::gensym();
my @recipients = map {("--recipient", $_)} @gpg_rcpts;
my $pid = IPC::Open2::open2($cout, $cin,
$self->gpg_path, $self->gpg_args,
@recipients,
"--trust-model=always",
"--batch",
"--encrypt",
"--output", $outfn,
"--yes",
"-" # read from stdin
);
# send data to gpg
binmode $cin;
my $bytes = io_print_to_fh($data_fh, $cin)
or die "Sending data to gpg failed: $!";
close $cin;
close $cout;
waitpid($pid, 0);
die "GPG failed: $!" if $? != 0; # If gpg return status is non-zero
}
1;
__END__
=head1 NAME
Brackup::Root - describes the source directory (and options) for a backup
=head1 EXAMPLE
In your ~/.brackup.conf file:
[SOURCE:bradhome]
path = /home/bradfitz/
gpg_recipient = 5E1B3EC5
chunk_size = 64MB
ignore = ^\.thumbnails/
ignore = ^\.kde/share/thumbnails/
ignore = ^\.ee/(minis|icons|previews)/
ignore = ^build/
noatime = 1
=head1 CONFIG OPTIONS
=over
=item B<path>
The directory to backup (recursively)
=item B<gpg_recipient>
The public key signature to encrypt data with. See L<Brackup::Manual::Overview/"Using encryption">.
=item B<chunk_size>
In units of bytes, kB, MB, etc. The max size of a chunk to be stored
on the target. Files over this size are cut up into chunks of this
size or smaller. The default is 64 MB if not specified.
=item B<ignore>
lib/Brackup/Root.pm view on Meta::CPAN
for any parameters that are not already defined in the current section.
The example above could also be written:
[SOURCE:defaults]
chunk_size = 64MB
noatime = 0
[SOURCE:bradhome]
inherit = defaults
path = /home/bradfitz/
gpg_recipient = 5E1B3EC5
ignore = ^\.thumbnails/
ignore = ^\.kde/share/thumbnails/
ignore = ^\.ee/(minis|icons|previews)/
ignore = ^build/
noatime = 1
=back
lib/Brackup/StoredChunk.pm view on Meta::CPAN
}
sub root {
my $self = shift;
return $self->file->root;
}
# returns true if encrypted, false otherwise
sub encrypted {
my $self = shift;
return $self->root->gpg_rcpts ? 1 : 0;
}
sub compressed {
my $self = shift;
# TODO/FUTURE: support compressed chunks (for non-encrypted
# content; gpg already compresses)
return 0;
}
# the original length, pre-encryption
sub length {
my $self = shift;
return $self->{pchunk}->length;
}
# the length, either encrypted or not
lib/Brackup/Test.pm view on Meta::CPAN
my $has_diff = eval "use Text::Diff; 1;";
my @to_unlink;
my $par_pid = $$;
END {
if ($$ == $par_pid) {
my $rv = unlink @to_unlink unless $ENV{BRACKUP_TEST_NOCLEANUP};
}
}
# Set the gpg directory, so we don't rely on users having a ~/.gnupg
$ENV{GNUPGHOME} = tempdir( CLEANUP => 1 );
sub do_backup {
my %opts = @_;
my $with_confsec = delete $opts{'with_confsec'} || sub {};
my $with_targetsec = delete $opts{'with_targetsec'} || sub {};
my $with_root = delete $opts{'with_root'} || sub {};
my $target = delete $opts{'with_target'};
die if %opts;
lib/Brackup/Test.pm view on Meta::CPAN
my ($meta_fh, $meta_filename) = tempfile();
ok(-e $meta_filename, "metafile exists");
push @to_unlink, $meta_filename;
ok(eval { $backup->backup($meta_filename) }, "backup succeeded");
if ($@) {
warn "Died running backup: $@\n";
}
ok(-s $meta_filename, "backup file has size");
check_inventory_db($target, [$root->gpg_args]);
return wantarray ? ($meta_filename, $backup, $target) : $meta_filename;
}
sub check_inventory_db {
my ($target, $gpg_args) = @_;
my $inv_db_file;
eval {
my $inv_db = $target->inventory_db or die 'cannot open inventory db';
$inv_db_file = $inv_db->backing_file ? (' ' . $inv_db->backing_file) : '';
while (my ($key, $value) = $inv_db->each) {
my ($raw_dig, $gpg_rcpt) = split /;/, $key;
my ($enc_dig, $enc_size, $range) = split /\s+/, $value;
# check the stored data
my $dataref = $target->load_chunk($enc_dig) or die "cannot load chunk $enc_dig";
length $$dataref == $enc_size or die "chunk $enc_dig has wrong size, not $enc_size";
$enc_dig eq "sha1:".sha1_hex($$dataref) or die "chunk $enc_dig has wrong digest";
# if we are in a composite chunk, keep only the part we want
if($range) {
my ($from, $to) = split '-', $range;
my $part = substr $$dataref, $from, $to-$from;
$dataref = \$part;
}
# decrypt if encrypted
my $dec_ref;
if($gpg_rcpt =~ /^to=(.*)$/) {
my $meta = { 'GPG-Recipient' => $1 };
local @Brackup::GPG_ARGS = @$gpg_args;
$dec_ref = Brackup::Decrypt::decrypt_data($dataref, meta => $meta);
} else {
$dec_ref = $dataref;
}
# check the raw data
$raw_dig eq "sha1:".sha1_hex($$dec_ref) or die "chunk $enc_dig has wrong raw digest";
}
};
lib/Brackup/Test.pm view on Meta::CPAN
if ($meta->{is_link}) {
$meta->{link} = readlink $path;
} else {
# we ignore these for links, since Linux doesn't let us restore anyway,
# as Linux as no lutimes(2) syscall, as of Linux 2.6.16 at least
$meta->{atime} = $st->atime if 0; # TODO: make tests work with atimes
$meta->{mtime} = $st->mtime;
$meta->{mode} = sprintf('%#o', $st->mode & 0777);
}
# the gpg tests open/close the rings in the root, so
# mtimes get bumped around or something. the proper fix
# is too ugly for what it's worth, so let's just ignore
# the mtime of top-level
delete $meta->{mtime} if $path eq ".";
return $meta;
}
# given a directory, returns a hashref of its contentn
sub dir_structure {
lib/Brackup/Test.pm view on Meta::CPAN
chdir($cwd) or die "Failed to chdir back to $cwd";
return \%files;
}
# add a random number of orphan chunks to $target
sub add_orphan_chunks {
my ($root, $target, $orphan_chunks_count) = @_;
for (1..$orphan_chunks_count) {
# HACK: to avoid worse hacks, we need a pchunk to store an orphan chunk.
# We use small segments of 'pubring-test.gpg' so that they are different
# than all other chunks
my $pchunk = Brackup::PositionedChunk->new(
file => Brackup::File->new(root => $root,
path => 'pubring-test.gpg'),
offset => $_ * 10,
length => 10,
);
# no encryption, copy raw data and store schunk
my $schunk = Brackup::StoredChunk->new($pchunk);
# $schunk->copy_raw_data;
$target->store_chunk($schunk);
}
}
use strict;
use Test::More;
use Brackup::Test;
use FindBin qw($Bin);
use Brackup::Util qw(tempfile);
############### Backup
if (`gpg --version`) {
plan tests => 17;
} else {
plan skip_all => 'gpg binary not found, skipping encrypted tests';
}
my $gpg_args = ["--no-default-keyring",
"--quiet",
"--keyring=$Bin/data/pubring-test.gpg",
"--secret-keyring=$Bin/data/secring-test.gpg"];
my $gpg_args2 = ["--no-default-keyring",
"--quiet",
"--keyring=$Bin/data/pubring-test2.gpg",
"--secret-keyring=$Bin/data/secring-test2.gpg"];
my ($digdb_fh, $digdb_fn) = tempfile();
close($digdb_fh);
my $root_dir = "$Bin/data";
ok(-d $root_dir, "test data to backup exists");
my $backup_file = do_backup(
with_confsec => sub {
my $csec = shift;
$csec->add("path", $root_dir);
$csec->add("chunk_size", "2k");
$csec->add("digestdb_file", $digdb_fn);
$csec->add("gpg_recipient", "2149C469");
$csec->add("gpg_recipient", "1AD2C5EB");
},
with_root => sub {
my $root = shift;
$root->{gpg_args} = $gpg_args;
},
);
############### Restore
# Restore as first recipient
my $restore_dir = do {
local @Brackup::GPG_ARGS = @$gpg_args;
do_restore($backup_file);
};
ok_dirs_match($restore_dir, $root_dir);
# Restore as second recipient
$restore_dir = do {
local @Brackup::GPG_ARGS = @$gpg_args2;
do_restore($backup_file);
};
ok_dirs_match($restore_dir, $root_dir);
t/03-composite-filesystem.t view on Meta::CPAN
use strict;
use Test::More;
use Brackup::Test;
use FindBin qw($Bin);
use Brackup::Util qw(tempfile);
############### Backup
my $gpg_version;
if ($gpg_version = `gpg --version`) {
plan tests => 31;
} else {
plan tests => 16;
}
my $gpg_args = ["--no-default-keyring",
"--quiet",
"--keyring=$Bin/data/pubring-test.gpg",
"--secret-keyring=$Bin/data/secring-test.gpg"];
my ($digdb_fh, $digdb_fn) = tempfile();
close($digdb_fh);
my $root_dir = "$Bin/data";
ok(-d $root_dir, "test data to backup exists");
############### Unencrypted backup
my ($backup_file, $backup) =
do_backup(
with_confsec => sub {
t/03-composite-filesystem.t view on Meta::CPAN
is((%seen)[-1], 2, "and stored it twice");
like((%seen)[0], qr/-/, "and it was stored in a range");
############### Restore
my $restore_dir = do_restore($backup_file);
ok_dirs_match($restore_dir, $root_dir);
exit unless $gpg_version;
############### Encrypted backup
($backup_file, $backup) =
do_backup(
with_confsec => sub {
my $csec = shift;
$csec->add("path", $root_dir);
$csec->add("merge_files_under", "1k");
$csec->add("max_composite_chunk_size", "500k");
$csec->add("digestdb_file", $digdb_fn);
$csec->add("gpg_recipient", "2149C469");
},
with_root => sub {
my $root = shift;
$root->{gpg_args} = $gpg_args;
},
);
# see if dup files were only stored once
%seen = ();
$backup->foreach_saved_file(sub {
my ($file, $slist) = @_;
return unless $file->path =~ /000-dup[12]\.txt$/;
foreach my $sc (@$slist) {
$seen{$sc->to_meta}++;
t/03-composite-filesystem.t view on Meta::CPAN
});
is(scalar keys %seen, 1, "stored just one uniq copy of 000-dup[12]");
is((%seen)[-1], 2, "and stored it twice");
like((%seen)[0], qr/-/, "and it was stored in a range");
############### Restore
$restore_dir = do {
local @Brackup::GPG_ARGS = @$gpg_args;
do_restore($backup_file);
};
ok_dirs_match($restore_dir, $root_dir);
t/03-composite-ftp.t view on Meta::CPAN
use strict;
use Test::More;
use Brackup::Test;
use FindBin qw($Bin);
use Brackup::Util qw(tempfile);
############### Setup
my $gpg_version;
if ($ENV{BRACKUP_TEST_FTP}) {
if ($gpg_version = `gpg --version`) {
plan tests => 31;
} else {
plan tests => 16;
}
} else {
plan skip_all => "\$ENV{BRACKUP_TEST_FTP} not set";
}
my $gpg_args = ["--no-default-keyring",
"--quiet",
"--keyring=$Bin/data/pubring-test.gpg",
"--secret-keyring=$Bin/data/secring-test.gpg"];
my ($digdb_fh, $digdb_fn) = tempfile();
close($digdb_fh);
my $root_dir = "$Bin/data";
ok(-d $root_dir, "test data to backup exists");
############### Unencrypted backup
my ($backup_file, $backup) =
do_backup(
with_confsec => sub {
t/03-composite-ftp.t view on Meta::CPAN
############### Restore
$ENV{FTP_PASSWORD} ||= 'user@example.com';
my $restore_dir = do_restore($backup_file);
ok_dirs_match($restore_dir, $root_dir);
exit unless $gpg_version;
############### Encrypted backup
($backup_file, $backup) =
do_backup(
with_confsec => sub {
my $csec = shift;
$csec->add("path", $root_dir);
$csec->add("merge_files_under", "1k");
$csec->add("max_composite_chunk_size", "500k");
$csec->add("digestdb_file", $digdb_fn);
$csec->add("gpg_recipient", "2149C469");
},
with_targetsec => sub {
my $tsec = shift;
$tsec->add("type", 'Ftp');
$tsec->add("ftp_host", $ENV{FTP_HOST} || 'localhost');
$tsec->add("ftp_user", $ENV{FTP_USER} || 'anonymous');
$tsec->add("ftp_password", $ENV{FTP_PASSWORD} || 'user@example.com');
},
with_root => sub {
my $root = shift;
$root->{gpg_args} = $gpg_args;
},
);
# see if dup files were only stored once
%seen = ();
$backup->foreach_saved_file(sub {
my ($file, $slist) = @_;
return unless $file->path =~ /000-dup[12]\.txt$/;
foreach my $sc (@$slist) {
$seen{$sc->to_meta}++;
t/03-composite-ftp.t view on Meta::CPAN
});
is(scalar keys %seen, 1, "stored just one uniq copy of 000-dup[12]");
is((%seen)[-1], 2, "and stored it twice");
like((%seen)[0], qr/-/, "and it was stored in a range");
############### Restore
$restore_dir = do {
local @Brackup::GPG_ARGS = @$gpg_args;
do_restore($backup_file);
};
ok_dirs_match($restore_dir, $root_dir);
t/03-composite-sftp.t view on Meta::CPAN
use strict;
use Test::More;
use Brackup::Test;
use FindBin qw($Bin);
use Brackup::Util qw(tempfile);
############### Backup
my $gpg_version;
if ($ENV{BRACKUP_TEST_SFTP}) {
if ($gpg_version = `gpg --version`) {
plan tests => 31;
} else {
plan tests => 16;
}
} else {
plan skip_all => "\$ENV{BRACKUP_TEST_SFTP} not set";
}
my $gpg_args = ["--no-default-keyring",
"--quiet",
"--keyring=$Bin/data/pubring-test.gpg",
"--secret-keyring=$Bin/data/secring-test.gpg"];
my ($digdb_fh, $digdb_fn) = tempfile();
close($digdb_fh);
my $root_dir = "$Bin/data";
ok(-d $root_dir, "test data to backup exists");
############### Unencrypted backup
my ($backup_file, $backup) =
do_backup(
with_confsec => sub {
t/03-composite-sftp.t view on Meta::CPAN
is((%seen)[-1], 2, "and stored it twice");
like((%seen)[0], qr/-/, "and it was stored in a range");
############### Restore
my $restore_dir = do_restore($backup_file);
ok_dirs_match($restore_dir, $root_dir);
exit unless $gpg_version;
############### Encrypted backup
($backup_file, $backup) =
do_backup(
with_confsec => sub {
my $csec = shift;
$csec->add("path", $root_dir);
$csec->add("merge_files_under", "1k");
$csec->add("max_composite_chunk_size", "500k");
$csec->add("digestdb_file", $digdb_fn);
$csec->add("gpg_recipient", "2149C469");
},
with_targetsec => sub {
my $tsec = shift;
$tsec->add("type", 'Sftp');
$tsec->add("sftp_host", $ENV{SFTP_HOST} || 'localhost');
$tsec->add("sftp_port", $ENV{SFTP_PORT}) if $ENV{SFTP_PORT};
$tsec->add("sftp_user", $ENV{SFTP_USER} || '');
},
with_root => sub {
my $root = shift;
$root->{gpg_args} = $gpg_args;
},
);
# see if dup files were only stored once
%seen = ();
$backup->foreach_saved_file(sub {
my ($file, $slist) = @_;
return unless $file->path =~ /000-dup[12]\.txt$/;
foreach my $sc (@$slist) {
$seen{$sc->to_meta}++;
t/03-composite-sftp.t view on Meta::CPAN
});
is(scalar keys %seen, 1, "stored just one uniq copy of 000-dup[12]");
is((%seen)[-1], 2, "and stored it twice");
like((%seen)[0], qr/-/, "and it was stored in a range");
############### Restore
$restore_dir = do {
local @Brackup::GPG_ARGS = @$gpg_args;
do_restore($backup_file);
};
ok_dirs_match($restore_dir, $root_dir);
t/05-filename-escaping.t view on Meta::CPAN
use strict;
use Test::More;
use Brackup::Test;
use FindBin qw($Bin);
use Brackup::Util qw(tempfile);
############### Backup
my $gpg_version;
if ($gpg_version = `gpg --version`) {
plan tests => 25;
} else {
plan tests => 13;
}
my $gpg_args = ["--no-default-keyring",
"--quiet",
"--keyring=$Bin/data/pubring-test.gpg",
"--secret-keyring=$Bin/data/secring-test.gpg"];
my ($digdb_fh, $digdb_fn) = tempfile();
close($digdb_fh);
my $root_dir = "$Bin/data-weird-filenames";
ok(-d $root_dir, "test data to backup exists ($root_dir)");
############### Unencrypted backup
my $backup_file = do_backup(
with_confsec => sub {
my $csec = shift;
t/05-filename-escaping.t view on Meta::CPAN
$csec->add("digestdb_file", $digdb_fn);
},
);
############### Restore
# Full restore
my $restore_dir = do_restore($backup_file);
ok_dirs_match($restore_dir, $root_dir);
exit unless $gpg_version;
############### Encrypted backup
$backup_file = do_backup(
with_confsec => sub {
my $csec = shift;
$csec->add("path", $root_dir);
$csec->add("chunk_size", "2k");
$csec->add("digestdb_file", $digdb_fn);
$csec->add("gpg_recipient", "2149C469");
},
with_root => sub {
my $root = shift;
$root->{gpg_args} = $gpg_args;
},
);
############### Restore
# Full restore
$restore_dir = do {
local @Brackup::GPG_ARGS = @$gpg_args;
do_restore($backup_file);
};
ok_dirs_match($restore_dir, $root_dir);
( run in 1.384 second using v1.01-cache-2.11-cpan-df04353d9ac )