Brackup
view release on metacpan or search on metacpan
1.10 (2010-10-31)
- permit 0 as a filename. https://rt.cpan.org/Ticket/Display.html?id=62004
- add Riak target, allowing backups to a riak cluster (Gavin Carr)
- add uid/gid info to metafile, and use in restores (where possible)
(Gavin Carr)
- allow multiple gpg recipients to be specified, any can restore (Alex
Vandiver)
- if IO::Compress::Gzip is available, write a compressed brackup metafile in
unencrypted mode, and handle properly for reads and restores (Gavin Carr)
- add brackup-verify-inventory utility (Gavin Carr)
- add chunkpath and size methods to Amazon S3 target (Gavin Carr)
1.09 (2009-05-18)
- move all decryption to one place; support decrypting metafiles.
(Stephane Alnet, stephane@shimaore.net, Chris Anderson, cva@pobox.com)
- added CloudFiles target to enable backups to Rackspace/Mosso's online
data storage service. (William Wolf)
1.08 (2009-04-26)
- make brackup-target --verbose prune and gc more verbose (Gavin Carr)
- add a --interactive option to brackup-target gc (Gavin Carr)
- add a --source option to brackup-target prune, to restrict a prune
run to a particular source (Gavin Carr)
- fix utime warnings on restore (Gavin Carr)
- actually strip non-ASCII characters (gary.richardson@gmail.com)
- smarts to filesystem target, noticed when using sshfs (slow filesystem).
does .partial files now, is smart about not overwriting existing chunk
that's there, etc.
- bradfitz: optional smart mp3 chunking. (start of file-aware chunking
strategies) with smart mp3 chunking, the ID3 tags are kept in separate
chunks, so future re-tagging of your music won't force iterative backups
to re-upload all the music bytes again... just the updated
metadata (tiny)
- Add a new option to the Amazon S3 target to force a prefix to be
added to the names of any saved backups.
It might be worth moving this up into Brackup itself at some point,
since it's probably useful for other network-based targets.
- Fix Restore.pm to use binmode for operating systems that care about
such things.
- Added a first whack at a FUSE filesystem for mounting a backup
as a real filesystem.
- Added FTP target.
- added the aws_location option to set the datacenter location for
S3 targets. from Alessandro Ranellucci <aar@cpan.org>.
The Amazon S3 target now depends on version 0.41 of Net::Amazon::S3.
- fixed tempfile creation in gc (orphaned files were left in the
working directory). from Alessandro Ranellucci <aar@cpan.org>.
- added the aws_prefix option to configure multiple backup targets
on a single Amazon account. from Alessandro Ranellucci <aar@cpan.org>.
1.06 (october 20, 2007)
- new on-disk layout for the Filesystem target. fully backwards
compatible with the old layout. also, if max-link count (max
files in a directory) is hit, this new version will carefully
rearrange the minimum files necessary to the new layout to
make room, all automatically. the new format is xx/xx/*,
rather than xxxx/xxxx/xxxx/xxxx/* which was stupid and overkill.
- actually respect the --just flag on restore
1.05 (2007-08-02)
- 'prune' and 'gc' commands commands for both Amazon
and Filesystem targets. from Alessandro Ranellucci <aar@cpan.org>.
1.04 (2007-06-30)
- Amazon list_backups and delete backups (and delete for filesystem
target too), from Alessandro Ranellucci <aar@cpan.org>
- make tests pass on OS X (Jesse Vincent)
1.03 (2007-05-23)
- brackup-restore's verbose flag is more verbose now, showing files
as they're restored.
- brackup-restore can restore from an encrypted *.brackup file now,
and in the future do garbage collection on no-longer-referenced
chunks (once a command exists to delete a brackup file from a target)
- stop leaking temp files
- doc fixes/additions
1.02 (2007-05-22)
- support for merging little files together into big chunks
on the backup target. aka "tail packing". requires no changes
to target drivers. this should speed backups, as less network
round-trips. will also be cheaper, once Amazon starts charging
per number of HTTP requests in June.
- improved docs
1.01 (2007-05-21)
- lot of new/updated docs
1.00 (2007-05-21)
- rename digestdb back to digestcache, now that it's purely a cache
again.
- fix memory leak in case where chunk exists on target, but local
digest database was lost, and digest of chunk had to be recomputed.
in that case, the raw chunk was kept in memory until the end
(which it likely would never reach, accumulating GBs of RAM)
- make PositionedChunk use the digest cache (which I guess was
re-fleshed out in the big refactor but never used...). so
iterative backups are fast again... no re-reading all files
in, blowing away all caches.
- clean up old, dead code in Amazon target (the old inventory db which
is now an official part of the core, and in the Target base class)
- retry PUTs to Amazon on failure, a few times, pausing in-between,
in case it was a transient error, as seems to happen occasionally
- halve number of stats when walking backup root
- cleanups, strictness
- don't upload meta files when in dry-run mode
- update amazon target support to work again, with the new inventory
database support (now separated from the old digest database)
- merge in the refactoring branch, in which a lot of long-standing
pet peeves in the design were rethought/redone.
and gpg-agent probably not running
- support putting .meta files besides .chunk files on the Target
to enable reconstructing the digest database in the future, should
it get lost. also start to flesh out per-chunk digests, which
would enable backing up large databases (say, InnoDB tablespaces) where
large chunks of the file never change.
- new --du-stats to command to act like the du(1) command, but
based on a root in brackup.conf, and skipping ignored directories.
good to let you know how big a backup will be.
- walk directories smarter: jump over directories early which ignore
patterns show as never matching.
- deal w/ encryption better: tell chunks when the backup target
will need data, so it can forget cached digest/backlength
ahead of time w/o errors/warnings later.
- start of stats code (to give stats after a backup). not done.
0.91 (2006-09-29)
- there's now a restore command (brackup-restore)
- amazon restore support
- use gpg --trust-model=always for new gpg that is more paranoid.
- mostly usable. some more switches would be nice later. real
1.00 release will come after few weeks/months of testing/tweaks.
0.80
- restore works
- lot more tests
- notable bug fix with encrypted backups. metafiles could have wrong sizes.
0.71
- first release to CPAN, didn't support restoring yet.
also didn't have a Changes file
lib/Brackup/Target/Filebased.pm
lib/Brackup/Target/Filesystem.pm
lib/Brackup/Target/Ftp.pm
lib/Brackup/Target/GoogleAppEngine.pm
lib/Brackup/Target/Riak.pm
lib/Brackup/Target/Sftp.pm
lib/Brackup/TargetBackupStatInfo.pm
lib/Brackup/Test.pm
lib/Brackup/Util.pm
t/00-use.t
t/01-backup-filesystem.t
t/01-backup-ftp.t
t/01-backup-riak.t
t/01-backup-sftp.t
t/02-gpg.t
t/03-composite-filesystem.t
t/03-composite-ftp.t
t/03-composite-sftp.t
t/04-gc-filesystem.t
t/04-gc-ftp.t
t/04-gc-sftp.t
t/05-filename-escaping.t
t/06-config-inheritance.t
t/data-2/000-dup1.txt
MANIFEST.SKIP view on Meta::CPAN
\bblib/
\bMakeMaker-\d
\bpm_to_blib\.ts$
\bpm_to_blib$
\bblibdirs\.ts$ # 6.18 through 6.25 generated this
# Avoid Module::Build generated and utility files.
\bBuild$
\b_build/
# Avoid temp and backup files.
~$
\.old$
\#$
\b\.#
\.bak$
# Avoid Devel::Cover files.
\bcover_db\b
--- #YAML:1.0
name: Brackup
version: 1.10
abstract: Flexible backup tool. Slices, dices, encrypts, and sprays across the net.
author:
- Brad Fitzpatrick <brad@danga.com>
license: unknown
distribution_type: module
configure_requires:
ExtUtils::MakeMaker: 0
build_requires:
ExtUtils::MakeMaker: 0
requires:
DBD::SQLite: 0
#!/usr/bin/perl
=head1 NAME
brackup - do a backup using Brackup
=head1 SYNOPSIS
$ brackup [-v] --from=<source> --to=<target> [--output=my_backup.brackup] [--save-stats]
=head2 OPTIONS
=over 4
=item --from=NAME
Required. The source or root of your backup. Must match a [SOURCE:NAME]
config section in your ~/.brackup.conf (which is auto-created for you
on first run, so then you just have to go modify it). See L<Brackup::Root>
for more.
=item --to=NAME
Required. The destination or target for your backup. Must match a
[TARGET:NAME] config section in your ~/.brackup.conf. See L<Brackup::Target>
for more.
=item --output=FILE
Optional. Defaults to "source-target-YYYYMMDD.brackup". This is the
"metafile" index you'll need to do a restore.
=item --config=FILE
Specify the configuration file to use; defaults to ~/.brackup.
=item --save-stats[=FILE]
Optional. Flag to indicate that stats output should be recorded to a
file. If =FILE is omitted, defaults to "source-target-YYYYMMDD.stats."
=item --verbose|-v
Show status during backup.
=item --dry-run
Don't actually store any data on the target.
=item --du-stats
Prints the size, in kB, of data underneath each directory
(recursively) which will be backed up.
=item --zenityprogress
Produces output suitable for piping into C<zenity --progress> to get a
pretty GUI progress bar while running a backup. This option is
incompatable with C<--verbose>, as both print to STDOUT.
=item --list-sources
List the names of the sources defined in your configuration file.
=item --list-targets
List the names of the targets defined in your configuration file.
use warnings;
use Getopt::Long;
use Cwd;
use FindBin qw($Bin);
use lib "$Bin/lib";
use Brackup;
use Brackup::Util qw(noclobber_filename);
my ($src_name, $target_name, $backup_file, $stats_file, $opt_help);
my $opt_dryrun;
my $opt_verbose;
my $opt_du_stats;
my $opt_zenityprogress;
my ($opt_list_sources, $opt_list_targets);
my $config_file = Brackup::Config->default_config_file_name;
my $arguments = join(' ', @ARGV);
usage() unless
GetOptions(
'from=s' => \$src_name,
'to=s' => \$target_name,
'verbose' => \$opt_verbose,
'zenity-progress' => \$opt_zenityprogress,
'output=s' => \$backup_file,
'save-stats:s' => \$stats_file,
'help' => \$opt_help,
'dry-run' => \$opt_dryrun,
'du-stats' => \$opt_du_stats,
'config=s' => \$config_file,
'list-sources' => \$opt_list_sources,
'list-targets' => \$opt_list_targets,
);
usage() if @ARGV;
usage() unless $src_name && $target_name;
my $cwd = getcwd();
sub usage {
my $why = shift || "";
if ($why) {
$why =~ s/\s+$//;
$why = "Error: $why\n\n";
}
die "${why}brackup --from=[source_name] --to=[target_name] [--output=<backup_metafile.brackup>]\nbrackup --help\n";
}
my $root = eval { $config->load_root($src_name); } or
usage($@);
my $target = eval { $config->load_target($target_name); } or
usage($@);
my @now = localtime();
$backup_file ||= sprintf("%s-%s-%04d%02d%02d.brackup",
$root->name, $target->name, $now[5]+1900, $now[4]+1, $now[3]);
$backup_file =~ s!^~/!$ENV{HOME}/! if $ENV{HOME};
$backup_file = "$cwd/$backup_file" unless $backup_file =~ m!^/!;
if (defined $stats_file) {
if ($stats_file eq '') {
($stats_file = $backup_file) =~ s/(\.brackup)?$/.stats/;
}
else {
$stats_file = "$cwd/$stats_file" unless $stats_file =~ m!^/!;
}
}
$backup_file = noclobber_filename($backup_file);
$stats_file = noclobber_filename($stats_file) if $stats_file;
my $backup = Brackup::Backup->new(
root => $root,
target => $target,
dryrun => $opt_dryrun,
verbose => $opt_verbose,
zenityprogress => $opt_zenityprogress,
);
if (my $stats = eval { $backup->backup($backup_file) }) {
warn "Backup complete.\n" if $opt_verbose;
$stats->set('Run Arguments:' => $arguments);
if ($opt_dryrun || $opt_verbose) {
$stats->print;
}
if ($stats_file) {
$stats->print($stats_file);
}
exit 0;
} else {
warn "Error running backup: $@\n";
exit 1;
}
brackup-mount view on Meta::CPAN
#!/usr/bin/perl
use strict;
=head1 NAME
brackup-mount - mount a backup as a filesystem using FUSE
=cut
# Make a friendly error message if Fuse isn't installed
BEGIN {
eval { require Fuse; };
if ($@) {
print STDERR "brackup-mount requires the 'Fuse' library from CPAN\n";
exit(1);
brackup-mount view on Meta::CPAN
exit(1);
}
=head1 SYNOPSIS
brackup-mount <metafile> <mountpoint>
=head1 DESCRIPTION
C<brackup-mount> allows you to mount a backup into your filesystem
at a particular mount point. Once it's mounted, you'll have a
read-only view of the directories and files in the backup
at the mountpoint given.
For example:
brackup-mount somebackup-20080203.brackup /mnt
This might be useful if you need to refer to something from a backup
but you don't want to do a full restore. You can also, if you like, do
something resembling a restore by mounting a backup and copying the
contents into your "real" filesystem.
=head1 PREREQUISITES
Before using this utility, you'll need to install the C<Fuse> library
from CPAN:
perl -MCPAN -e "install Fuse"
If you're on a Debian-like system then this might be a better idea:
brackup-mount view on Meta::CPAN
=head1 HOW IT WORKS
C<brackup-mount> reads the metafile it is given and uses the metadata
within to create a filesystem that is exposed via FUSE. All operations
apart from reading from files operate purely on the in-memory data structure
created from the metafile, and so you can C<ls> and C<stat> files to
your heart's content without worrying about expensive calls to your
target.
When a process calls C<open> on a file, the file will be effectively
"restored" from the backup target into a temporary directory, where it
will remain until it is ultimately C<close>d. All C<read> operations
on the file are performed on the temporary file. This means that you
can expect the C<open> call to be the most expensive call against this
filesystem.
If you're paying for data transfer from your target, be aware that
the local copy retrieved on C<open> is thrown away on C<close>, so if you
plan to be opening and closing the same file repeatedly you might
want to force the local copy to be retained for the duration by running
something like C<tail -f filename> in another terminal.
brackup-restore view on Meta::CPAN
$ brackup-restore [-v] --from=foo.brackup --to=<base_directory> --all
$ brackup-restore [-v] --from=foo.brackup --to=<base_directory> --just=<file>
$ brackup-restore [-v] --from=foo.brackup --to=<base_directory> --just=<dir>
=head2 OPTIONS
=over 4
=item --from=NAME
Required. The backup metafile, describing the tree you want to restore.
Probably named like "source-target-YYYYMMDD.brackup". If you lost it,
it's also stored on your backup target, and you can fetch it with
L<brackup-target>.
=item --to=NAME
Required. The destination root directory for your restored files.
Will be created if it doesn't exist.
=item --all
Restore all files.
brackup-target view on Meta::CPAN
#!/usr/bin/perl
=head1 NAME
brackup-target - Manage your backup targets
=head1 SYNOPSIS
$ brackup-target [opts] <target_name> list_backups
$ brackup-target [opts] <target_name> get_backup <backup_file>
$ brackup-target [opts] <target_name> get_backups
$ brackup-target [opts] <target_name> delete_backup <backup_file>
$ brackup-target [opts] <target_name> prune # remove old backups
$ brackup-target [opts] <target_name> gc # run garbage collector
=head2 OPTIONS
=over 4
=item --dest=DIR
Destination to write files to. Defaults to current working directory.
=item --verbose|-v
Be verbose with status.
=item --dry-run
Do not actually execute write operations.
=item --keep-backups
To be used in combination with the I<prune> command. This overrides the
I<keep_backups> option specified in the configuration file.
=item --source <source>
To be used in combination with the I<prune> command. This restricts the
prune operation to only delete backup files from the given I<source>.
=item --interactive
To be used in combination with the I<gc> command. Runs interactively,
requiring an explicit confirmation before deleting chunks that have been
garbage collected. Implies --verbose.
=back
=head1 WARRANTY
brackup-target view on Meta::CPAN
use Cwd;
use FindBin qw($Bin);
use lib "$Bin/lib";
use Brackup;
my $config_file;
my $destdir;
my $opt_help;
my $opt_verbose;
my $opt_keep_backups;
my $opt_dryrun;
my $opt_interactive;
my $opt_source;
usage() unless
GetOptions(
'verbose+' => \$opt_verbose,
'dest=s' => \$destdir,
'config=s' => \$config_file,
'keep-backups=i' => \$opt_keep_backups,
'dry-run' => \$opt_dryrun,
'interactive' => \$opt_interactive,
'source=s' => \$opt_source,
'help' => \$opt_help,
);
if ($destdir) {
chdir $destdir or die "Failed to chdir to $destdir: $!\n";
}
brackup-target view on Meta::CPAN
my $target = eval { $config->load_target($target_name); } or
usage($@);
my $code = __PACKAGE__->can("CMD_$cmd_name") or
usage("Unknown/unimplemented command.");
exit($code->() ? 0 : 1);
sub CMD_list_backups {
printf("%-32s %-30s %10s\n",
'Backup File',
'Backup Date',
'Size (B)');
printf("%-32s %-30s %10s\n",
'-' x 11,
'-' x 11,
'-' x 8);
foreach my $si (sort { $b->time <=> $a->time } $target->backups) {
printf("%-32s %-30s %10s\n",
$si->filename,
$si->localtime,
$si->size || '?');
}
return 1;
}
sub CMD_get_backup {
my $name = shift @ARGV or
die "get_backup requires a filename to download";
$target->get_backup($name)
or die "Failed to retrieve backup $name\n";
}
sub CMD_get_backups {
foreach my $si ($target->backups) {
my $size = $si->size;
my $name = $si->filename;
no warnings 'uninitialized';
if (-s "$name.brackup" == $size || -s "$name.brackup.orig" == $size) {
debug("Skipping $name; already have it");
next;
}
debug("Fetching $name");
$target->get_backup($si->filename);
}
}
sub CMD_delete_backup {
my $name = shift @ARGV or
die "delete_backup requires a filename to download";
$target->delete_backup($name)
or die "Failed to delete backup $name\n";
}
sub CMD_prune {
my $removed_count = $target->prune( keep_backups => $opt_keep_backups,
dryrun => $opt_dryrun,
source => $opt_source,
verbose => $opt_verbose);
debug("$removed_count backups " . ($opt_dryrun ? "would be " : "") . "removed from target");
}
sub CMD_gc {
my ($removed_chunks, $total_chunks) = $target->gc(dryrun => $opt_dryrun,
verbose => $opt_verbose,
interactive => $opt_interactive);
debug("$removed_chunks/$total_chunks chunks " . ($opt_dryrun ? "would be " : "") .
"removed from target");
}
doc/data-structures.txt view on Meta::CPAN
----------------------------------------------------------------------------
Class-wise, we have:
----------------------------------------------------------------------------
Root -- describes a path on the filesystem to be backed up. has as
properties how small large files are cut up into
("chunk_size"), what files to ignore, and the encryption
settings.
Target -- a destination for the backups.
File -- a directory, symlink, or file in a Root.
Chunk -- part of a file, defined as an offset and length. depending
on encryption settings, the serialized backup length can be
more or less than the unencrypted length.
Backup -- a snapshot in time of all a Root's Files and Chunks.
during the backup, the Target is consulted to see if it
has chunks before they're re-stored. The backup upon
completion writes a structured file as described below.
DigestCache -- the digest cache, a property of the Root, acts
mostly as a cache, but is pretty important when
using encryption. If you lose the database, all your
files will need to be re-encrypted, as Brackup won't
know if the chunks already exist, as encryption makes
different files each time. Note that you don't need
the digest database to do a restore.
doc/data-structures.txt view on Meta::CPAN
ChunkCacheKey ::= <TypedDigest(original_unencrypted_file)> "-" <raw_offset> "-" <raw_length> "-" <gpg-recipient>
ChunkDetails ::= <EncryptedLength> " " <TypedDigest(encrypted_chunk)>
TypedDigest ::= <DigestAlgo> ":" <hex_digest>
DigestAlgo ::= { "sha1" }
----------------------------------------------------------------------------
[backup-name].brackup format (RFC-822-like)
----------------------------------------------------------------------------
Keys:
-----
Path: relative path
Size: unencrypted size
Digest: unencrypted digest (see TypedDigest format above)
Type: "f" or blank for regular file
"l" for symlink
doc/databases.txt view on Meta::CPAN
inventory DB:
-------------
default loc:
[TARGET:foo]'s "inventory_db" key, or
"$ENV{HOME}/.brackup-target-$name.invdb";
mapping:
pchunk->inventory_key ---> join(" ", $schunk->backup_digest, $schunk->backup_length))
<dig>;to=<rcpt>
<dig>;raw
Digest DB:
----------
default loc:
[SOURCE:foo]'s 'digestdb_file' key, or
"$SOURCE_DIR/.brackup-digest.db"
doc/design-decisions.txt view on Meta::CPAN
-- you should be able to restore without setting up a config file.
if you lost data, that'd be annoying. restoring from a config
file will be supported in the future, but it's not yet.
-- backups must be automatable, never requiring user input. hence public
key encryption.
-- restores may prompt for user input ("What's your Amazon S3
password?" and "Enter your GPG passphrase."), because they won't be
automated or common. and I don't want a restore to require a fully
setup ~/.brackup.conf. You probably lost it anyway. So a *.brackup
metafile (the one you get after a backup) should contain all the
metadata necessary to restore (say, Amazon S3 username), but not
secret stuff.
-- targets shouldn't include passwords (say, Amazon S3 password)
in the *.brackup (backup "index"/"meta" file). let the user
enter that on restore. you should, however, put in metadata
that'll ease restoring.... like Amazon username, or path, etc.
doc/exampleconfig.txt view on Meta::CPAN
# a sample ~/.brackup.conf
[TARGET:raidbackups]
type = Filesystem # this can be any Brackup::Target::<foo> subclass
path = /raid/backup/brackup
[SOURCE:proj]
path = /raid/bradfitz/proj/
chunk_size = 5m
gpg_recipient = 5E1B3EC5
[SOURCE:bradhome]
chunk_size = 64MB
path = /raid/bradfitz/
ignore = ^\.thumbnails/
doc/overview.txt view on Meta::CPAN
Originally posted to:
<http://brad.livejournal.com/2205732.html>
There are lots of ways to store files on the net lately:
-- Amazon S3 is the most interesting,
-- Google's rumored GDrive is surely soon coming
-- Apple has .Mac
I want to back up to them. And more than one. So first off, abstract
out net-wide storage.... my backup tool (wsbackup) isn't targetting
one. They're all just providers.
Also, don't trust sending my data in cleartext, and having it stored
in cleartext, so public key encryption is a must. Then I can run
automated backups from many hosts, without much fear of keys being
compromised.
Don't want people being able to do size-analysis, and huge files are a pain anyway, so big files are cut into chunks.
Files stored on Amazon/Google are of form:
-- meta files: backup_rootname-yyyymmddnn.meta, encrypted (YAML?) file mapping relative paths from backup directory root to the stat() information, original SHA1, and array of chunk keys (SHA1s of encrypted chunks) that comprise the file.
-- [sha1ofencryptedchunk].chunk -- content being <= ,say, 20MB chunk of encrypted data.
Then every night different hosts/laptops recurse directory trees,
consult a stat() cache (on, say, inode number, mtime, size, whatever)
and do SHA1 calculations on changed files, lookup rest from cache, and
build the metafile, upload any new chunks, encrypt the metafile,
upload the metafile.
Result:
lib/Brackup.pm view on Meta::CPAN
use Brackup::Restore;
use Brackup::Target;
use Brackup::BackupStats;
1;
__END__
=head1 NAME
Brackup - Flexible backup tool. Slices, dices, encrypts, and sprays across the net.
=head1 FURTHER READING
L<Brackup::Manual::Overview>
L<brackup>
L<brackup-restore>
L<brackup-target>
lib/Brackup/Backup.pm view on Meta::CPAN
$self->{savefiles} = delete $opts{savefiles}; # bool
$self->{zenityprogress} = delete $opts{zenityprogress}; # bool
$self->{modecounts} = {}; # type -> mode(octal) -> count
$self->{idcounts} = {}; # type -> uid/gid -> count
$self->{_uid_map} = {}; # uid -> username
$self->{_gid_map} = {}; # gid -> group
$self->{saved_files} = []; # list of Brackup::File objects backed up
$self->{unflushed_files} = []; # list of Brackup::File objects not in backup_file
croak("Unknown options: " . join(', ', keys %opts)) if %opts;
return $self;
}
# returns true (a Brackup::BackupStats object) on success, or dies with error
sub backup {
my ($self, $backup_file) = @_;
my $root = $self->{root};
my $target = $self->{target};
my $stats = Brackup::BackupStats->new;
my @gpg_rcpts = $self->{root}->gpg_rcpts;
my $n_kb = 0.0; # num: kb of all files in root
my $n_files = 0; # int: # of files in root
lib/Brackup/Backup.pm view on Meta::CPAN
undef @files;
$stats->timestamp('Chunk Iterator');
my $gpg_iter;
my $gpg_pm; # gpg ProcessManager
if (@gpg_rcpts) {
($chunk_iterator, $gpg_iter) = $chunk_iterator->mux_into(2);
$gpg_pm = Brackup::GPGProcManager->new($gpg_iter, $target);
}
# begin temp backup_file
my ($metafh, $meta_filename);
unless ($self->{dryrun}) {
($metafh, $meta_filename) = tempfile(
'.' . basename($backup_file) . 'XXXXX',
DIR => dirname($backup_file),
);
if (! @gpg_rcpts) {
if (eval { require IO::Compress::Gzip }) {
close $metafh;
$metafh = IO::Compress::Gzip->new($meta_filename)
or die "Cannot open tempfile with IO::Compress::Gzip: $IO::Compress::Gzip::GzipError";
}
}
print $metafh $self->backup_header;
}
my $cur_file; # current (last seen) file
my @stored_chunks;
my $file_has_shown_status = 0;
my $merge_under = $root->merge_files_under;
my $comp_chunk = undef;
my $end_file = sub {
return unless $cur_file;
if ($merge_under && $comp_chunk) {
# defer recording to backup_file until CompositeChunk finalization
$self->add_unflushed_file($cur_file, [ @stored_chunks ]);
}
else {
print $metafh $cur_file->as_rfc822([ @stored_chunks ], $self) if $metafh;
}
$self->add_saved_file($cur_file, [ @stored_chunks ]) if $self->{savefiles};
$n_files_done++;
$n_kb_done += $cur_file->size / 1024;
$cur_file = undef;
};
lib/Brackup/Backup.pm view on Meta::CPAN
while (my $rec = $chunk_iterator->next) {
if ($rec->isa("Brackup::File")) {
$start_file->($rec);
next;
}
my $pchunk = $rec;
if ($pchunk->file != $cur_file) {
$start_file->($pchunk->file);
}
# have we already stored this chunk before? (iterative backup)
my $schunk;
if ($schunk = $target->stored_chunk_from_inventory($pchunk)) {
$pchunk->forget_chunkref;
push @stored_chunks, $schunk;
next;
}
# weird case... have we stored this same pchunk digest in the
# current comp_chunk we're building? these aren't caught by
# the above inventory check, because chunks in a composite
lib/Brackup/Backup.pm view on Meta::CPAN
unless ($self->{dryrun}) {
$schunk = Brackup::StoredChunk->new($pchunk);
# encrypt it
if (@gpg_rcpts) {
$schunk->set_encrypted_chunkref($gpg_pm->enc_chunkref_of($pchunk));
}
# see if we should pack it into a bigger blob
my $chunk_size = $schunk->backup_length;
# see if we should merge this chunk (in this case, file) together with
# other small files we encountered earlier, into a "composite chunk",
# to be stored on the target in one go.
# Note: no technical reason for only merging small files (is_entire_file),
# and not the tails of larger files. just don't like the idea of files being
# both split up (for big head) and also merged together (for little end).
# would rather just have 1 type of magic per file. (split it or join it)
if ($merge_under && $chunk_size < $merge_under && $pchunk->is_entire_file) {
lib/Brackup/Backup.pm view on Meta::CPAN
push @stored_chunks, $schunk;
}
#$stats->note_stored_chunk($schunk);
# DEBUG: verify it got written correctly
if ($ENV{BRACKUP_PARANOID}) {
die "FIX UP TO NEW API";
#my $saved_ref = $target->load_chunk($handle);
#my $saved_len = length $$saved_ref;
#unless ($saved_len == $chunk->backup_length) {
# warn "Saved length of $saved_len doesn't match our length of " . $chunk->backup_length . "\n";
# die;
#}
}
$stats->check_maxmem;
$pchunk->forget_chunkref;
}
$end_file->();
$comp_chunk->finalize if $comp_chunk;
$self->flush_files($metafh);
$stats->timestamp('Chunk Storage');
$stats->set('Number of Files Uploaded:', $n_files_up);
$stats->set('Total File Size Uploaded:', sprintf('%0.01f MB', $n_kb_up / 1024));
unless ($self->{dryrun}) {
close $metafh or die "Close on metafile '$backup_file' failed: $!";
rename $meta_filename, $backup_file
or die "Failed to rename temporary backup_file: $!\n";
my ($store_fh, $store_filename);
my $is_encrypted = 0;
# store the metafile, encrypted, on the target
if (@gpg_rcpts) {
my $encfile = $backup_file . ".enc";
my @recipients = map {("--recipient", $_)} @gpg_rcpts;
system($self->{root}->gpg_path, $self->{root}->gpg_args,
@recipients,
"--trust-model=always",
"--batch",
"--encrypt",
"--output=$encfile",
"--yes",
$backup_file)
and die "Failed to run gpg while encryping metafile: $!\n";
open ($store_fh, $encfile) or die "Failed to open encrypted metafile '$encfile': $!\n";
$store_filename = $encfile;
$is_encrypted = 1;
} else {
# Reopen $metafh to reset file pointer (no backward seek with IO::Compress::Gzip)
open($store_fh, $backup_file) or die "Failed to open metafile '$backup_file': $!\n";
$store_filename = $backup_file;
}
# store it on the target
$self->debug("Storing metafile to " . ref($target));
my $name = $self->{root}->publicname . "-" . $self->backup_time;
$target->store_backup_meta($name, $store_fh, { filename => $store_filename, is_encrypted => $is_encrypted });
$stats->timestamp('Metafile Storage');
# cleanup encrypted metafile
if ($is_encrypted) {
close $store_fh or die "Close on encrypted metafile failed: $!";
unlink $store_filename;
}
}
$self->report_progress(100, "Backup complete.");
lib/Brackup/Backup.pm view on Meta::CPAN
my @map;
my $gidcounts = $self->{idcounts}{g};
for my $gid (sort { $a <=> $b } keys %$gidcounts) {
if (my $name = getgrgid($gid)) {
push @map, "$gid:$name";
}
}
return join(' ', @map);
}
sub backup_time {
my $self = shift;
return $self->{backup_time} ||= time();
}
sub backup_header {
my $self = shift;
my $ret = "";
my $now = $self->backup_time;
$ret .= "BackupTime: " . $now . " (" . localtime($now) . ")\n";
$ret .= "BackupDriver: " . ref($self->{target}) . "\n";
if (my $fields = $self->{target}->backup_header) {
foreach my $k (sort keys %$fields) {
die "Bogus header field from driver" unless $k =~ /^\w+$/;
my $val = $fields->{$k};
next if ! defined $val || $val eq ''; # skip keys with empty values
die "Bogus header value from driver" if $val =~ /[\r\n]/;
$ret .= "Driver-$k: $val\n";
}
}
$ret .= "RootName: " . $self->{root}->name . "\n";
$ret .= "RootPath: " . $self->{root}->path . "\n";
lib/Brackup/Chunker/MP3.pm view on Meta::CPAN
__END__
=head1 NAME
Brackup::Chunker::MP3 - an mp3-aware file chunker
=head1 ABOUT
This chunker knows about the structure of MP3 files (by using
L<MP3::Info>) and will instruct Brackup to backup the metadata and the
audio bytes separately. That way if you re-tag your music and later
do an interative backup, you only back up the new metadata bits
(tiny).
lib/Brackup/CompositeChunk.pm view on Meta::CPAN
$self->{sha1} = Digest::SHA1->new;
$self->{_chunk_fh} = tempfile_obj();
return $self;
}
sub append_little_chunk {
my ($self, $schunk) = @_;
die "ASSERT" if $self->{digest}; # its digest was already requested?
my $from = $self->{used_up};
$self->{used_up} += $schunk->backup_length;
io_print_to_fh($schunk->chunkref, $self->{_chunk_fh}, $self->{sha1});
my $to = $self->{used_up};
$schunk->set_composite_chunk($self, $from, $to);
push @{$self->{subchunks}}, $schunk;
}
sub digest {
my $self = shift;
return $self->{digest} ||= "sha1:" . $self->{sha1}->hexdigest;
lib/Brackup/CompositeChunk.pm view on Meta::CPAN
foreach my $schunk (@{$self->{subchunks}}) {
next unless $schunk->pchunk->inventory_key eq $ikey;
# match! found a duplicate within ourselves
return $schunk->clone_but_for_pchunk($pchunk);
}
return undef;
}
# <duck-typing>
# make this duck-typed like a StoredChunk, so targets can store it
*backup_digest = \&digest;
sub backup_length {
my $self = shift;
return $self->{used_up};
}
# return handle to data
sub chunkref {
my $self = shift;
croak "ASSERT: _chunk_fh not opened" unless $self->{_chunk_fh}->opened;
seek($self->{_chunk_fh}, 0, SEEK_SET);
return $self->{_chunk_fh};
}
lib/Brackup/Config.pm view on Meta::CPAN
}
}
sub write_dummy_config {
my $file = shift;
sysopen (my $fh, $file, O_WRONLY | O_CREAT | O_EXCL, 0600) or return;
print $fh <<ENDCONF;
# This is an example config
#[TARGET:raidbackups]
#type = Filesystem
#path = /raid/backup/brackup
#keep_backups = 10
#[TARGET:amazon]
#type = Amazon
#aws_access_key_id = XXXXXXXXXX
#aws_secret_access_key = XXXXXXXXXXXX
#keep_backups = 10
#[SOURCE:proj]
#path = /raid/bradfitz/proj/
#chunk_size = 5m
#gpg_recipient = 5E1B3EC5
#[SOURCE:bradhome]
#path = /raid/bradfitz/
#noatime = 1
#chunk_size = 64MB
lib/Brackup/DigestCache.pm view on Meta::CPAN
__END__
=head1 NAME
Brackup::DigestCache - cache digests of file and chunk contents
=head1 DESCRIPTION
The brackup DigestCache caches the digests (currently SHA1) of files
and file chunks, to prevent untouched files from needing to be re-read
on subsequent, iterative backups.
The digest cache is I<purely> a cache. It has no critical data in it,
so if you lose it, subsequent backups will just take longer while the
digest cache is re-built.
Note that you don't need the digest cache to do a restore.
=head1 DETAILS
=head2 Storage type
The digest cache makes use of Dictionary modules (Brackup::Dict::*) to
handle the storage of the cache. The default dictionary used is
lib/Brackup/File.pm view on Meta::CPAN
my $self = shift;
return $self->stat->uid;
}
sub gid {
my $self = shift;
return $self->stat->gid;
}
sub as_rfc822 {
my ($self, $schunk_list, $backup) = @_;
my $ret = "";
my $set = sub {
my ($key, $val) = @_;
return unless length $val;
$ret .= "$key: $val\n";
};
my $st = $self->stat;
$set->("Path", printable($self->{path}));
my $type = $self->type;
lib/Brackup/File.pm view on Meta::CPAN
$set->("Link", $self->link_target);
}
}
$set->("Chunks", join("\n ", map { $_->to_meta } @$schunk_list));
unless ($self->is_link) {
$set->("Mtime", $st->mtime);
$set->("Atime", $st->atime) unless $self->root->noatime;
my $mode = $self->mode;
unless (($type eq "d" && $mode eq $backup->default_directory_mode) ||
($type eq "f" && $mode eq $backup->default_file_mode)) {
$set->("Mode", $mode);
}
}
my $uid = $self->uid;
unless ($uid eq $backup->default_uid) {
$set->("UID", $uid);
}
my $gid = $self->gid;
unless ($gid eq $backup->default_gid) {
$set->("GID", $gid);
}
return $ret . "\n";
}
1;
lib/Brackup/InventoryDatabase.pm view on Meta::CPAN
1;
__END__
=head1 NAME
Brackup::InventoryDatabase - track what chunks are already on a target
=head1 DESCRIPTION
The Brackup InventoryDatabase keeps track of which chunks (files) are
already on a given target, speeding up future iterative backups,
negating the need to ask the target for each chunk whether it exists
or not (which may be a lot of network roundtrips, slow even in the
best of network conditions).
Unlike the L<Digest Cache|Brackup::DigestCache>, the inventory
database is not a cache... its contents matter. Consider what happens
when the inventory database doesn't match reality:
=over
=item B<1) Exists in inventory database; not on target>
If a chunk exists in the inventory database, but not on the target, brackup
won't store it on the target, and you'll think a backup succeeded, but
it's not actually there.
=item B<2a) Exists on target; not in inventory database (without encryption)>
You re-upload it to the target, so you waste time & bandwidth, but no
extra disk space is wasted, and no chunks are orphaned. Actually,
chunks are un-orphaned, as the inventory database is now updated and
contains the chunk you just uploaded.
=item B<2b) Exists on target; not in inventory database (with encryption)>
lib/Brackup/InventoryDatabase.pm view on Meta::CPAN
[TARGET] declaration in ~/.brackup.conf e.g.:
[TARGET:amazon]
type = Amazon
aws_access_key_id = ...
aws_secret_access_key = ...
inventorydb_file = /home/bradfitz/.amazon-already-has-these-chunks.db
Or, more commonly (and recommended), is to not specify it and accept
the default location, which is ".brackup-target-TARGETNAME.invdb" in
your home directory (where it might be shared by multiple backup
roots).
=head2 SQLite Schema
This is made automatically for you, but if you want to look around in
it, the schema is:
CREATE TABLE target_inv (
key TEXT PRIMARY KEY,
value TEXT
lib/Brackup/Manual/Overview.pod view on Meta::CPAN
=head2 Setup your config file
Run B<brackup> to initialize your config file. You'll see:
$ brackup
Error:
Your config file needs tweaking.
I put a commented-out template at: /home/bradfitz/.brackup.conf
brackup --from=[source_name] --to=[target_name] [--output=<backup_metafile.brackup>]
brackup --help
Now, go edit your config file:
$ $EDITOR ~/.brackup.conf
Tweak as appropriate.
For details on what's tweakable, see L<Brackup::Root> (a "source"), or
L<Brackup::Target> (a destination).
=head2 Do a backup
Now that you've got a source and target named, run a backup. I like
watching it all happen with the --verbose (or -v) option:
$ brackup --from=myhome --to=amazon -v
=head2 What just happened?
Let's look around at what just happened.
First, you'll notice a file named, by default,
"myhome-yyyymmdd.brackup" in your current directory. Go look at it.
lib/Brackup/Manual/Overview.pod view on Meta::CPAN
not critical. (You can always re-download lost .brackup files with
L<brackup-target>)
You might also notice two SQLite files at:
$SOURCE_ROOT/.brackup-digest.db
$HOME/.brackup-target-amazon.invdb
These are the L<Brackup::DigestCache> and
L<Brackup::InventoryDatabase> files, both of which make future
incremental backups fast.
=head1 Incremental backups
Incremental backups are essentially free, only storing new chunks,
even if you rearrange your directory tree or rename all your files.
Brackup doesn't use the I<name> of your files to decide what's new in
an incremental backup, only the contents.
For two back-to-back backups, with no data changes in-between, the
only cost of an incremental backup is that another metafile (*.brackup) is
produced, which is proportional in size to the I<number of files>
you're backing up (not the size of the files).
Another good side-effect of storing backups based on their digests is
that multiple, duplicate files on your source are only stored on the
target once. (but yes, they're restored to all original locations)
=head1 Using encryption
Brackup supports backing up with public key encryption, using GNU
Privacy Guard (GnuPG). One of the great advantage of using public key
encryption is that your machines doing backups only need your public
key, so you can run automated backups from hosts which are on the
public Internet and might be get compromised, without worrying about
your private key getting stolen. (however, you'd still worry about
your machine getting compromised for lots of other reasons...)
In any case, you encrypt files I<to yourself>, and this is a property
on a backup source (see L<Brackup::Root>). For example, in my config
file, I have:
[SOURCE:bradhome]
...
path = /home/bradfitz/
gpg_recipient = 5E1B3EC5
...
Where 5E1B3EC5 corresponds to the key signature for myself as seen in:
$ gpg --list-keys
...
pub 1024D/5E1B3EC5 2006-03-20
uid Brad Fitzpatrick <brad@danga.com>
....
While you backup automatically without a human present, a restore from
encryption requires an interactive session for you to enter your
private key's passphrase into gpg-agent.
To create a new key, run:
$ gpg --gen-key
But really, you should go read a gpg manual first. Notably, B<backing
up your gpg private key is very important!>. If you lose the disk
with your files which also contain your private key, your encrypted backups on
Amazon won't do you much good, since you'll have no way to decrypt them.
I recommend burning your private key to a CD, as well as printing it out
on paper. (Worst case you can type it back in, or use OCR.) Export with:
$ gpg --export-secret-keys --armor
You can encrypt to multiple keys by providing multiple C<gpg_recipient>
lines; any of the keys provided will be able to decrypt the backups.
=head1 Restores
To do a restore, you'll need your *.brackup file handy. If you lost
it, you can re-download it from your backup target with
L<brackup-target>. Then run:
brackup-restore --from=foo.brackup --to=<dir> --all
For more options, see:
brackup-restore --help
=head1 Number of backups to keep
To free space on your target you can remove old backups. There are two steps
to do this:
brackup-target <target> prune
brackup-target <target> gc
The first command will look for backup metafiles in your target and remove the
oldest ones according to the I<keep_backups> option you specified in the config
file. Thus, if you have, say, 15 backups stored and I<keep_backups> is set to 10
then I<prune> will remove the oldest 5 backups.
The second command will remove from your target the orphaned chunks that are no
more referenced by any metafile. This will free some space while preserving chunks
that are still referenced by recent backups.
=head1 SEE ALSO
L<Brackup>
lib/Brackup/Mount.pm view on Meta::CPAN
my $p = Brackup::Metafile->open($metafile) or die "Failed to open metafile $metafile";
my $header = $p->readline();
my $driver_header = Brackup::Restore::_driver_meta($header);
my $driver_class = $header->{BackupDriver};
die "No driver specified" unless $driver_class;
eval "use $driver_class; 1;" or die "Failed to load driver ($driver_class) to restore from: $@\n";
my $target = "$driver_class"->new_from_backup_header($driver_header);
my $meta = $class->_build_metadata($header, $p);
my $tempdir = File::Temp::tempdir(CLEANUP => 1);
my $file_temp_path = sub {
my ($record) = @_;
return $tempdir."/".$record->{digest};
};
lib/Brackup/Mount.pm view on Meta::CPAN
$meta{$path_key} = $record;
push @{$meta{$parent_path}{child_nodes}}, $local_name;
}
return \%meta;
}
=head1 NAME
Brackup::Mount - Mount a backup as a usable filesystem using FUSE
=cut
1;
lib/Brackup/Restore.pm view on Meta::CPAN
my $confsec;
if ($self->{config} && $meta->{TargetName}) {
$confsec = eval { $self->{config}->get_section('TARGET:' . $meta->{TargetName}) };
}
# If no config section, use an empty one up with no keys to simplify Target handling
$confsec ||= Brackup::ConfigSection->new('fake');
eval "use $driver_class; 1;" or die
"Failed to load driver ($driver_class) to restore from: $@\n";
my $target = eval {"$driver_class"->new_from_backup_header($driver_meta, $confsec); };
if ($@) {
die "Failed to instantiate target ($driver_class) for restore. Perhaps it doesn't support restoring yet?\n\nThe error was: $@";
}
$self->{_target} = $target;
$self->{_meta} = $meta;
# handle absolute prefixes by stripping off RootPath to relativise
if ($self->{prefix} && $self->{prefix} =~ m/^\//) {
$self->{prefix} =~ s/^\Q$meta->{RootPath}\E\/?//;
}
lib/Brackup/Root.pm view on Meta::CPAN
use Brackup::DigestCache;
use Brackup::Util qw(io_print_to_fh);
use IPC::Open2;
use Symbol;
sub new {
my ($class, $conf) = @_;
my $self = bless {}, $class;
($self->{name}) = $conf->name =~ m/^SOURCE:(.+)$/
or die "No backup-root name provided.";
die "Backup-root name must be only a-z, A-Z, 0-9, and _." unless $self->{name} =~ /^\w+/;
$self->{dir} = $conf->path_value('path');
$self->{gpg_path} = $conf->value('gpg_path') || "gpg";
$self->{gpg_rcpt} = [ $conf->values('gpg_recipient') ];
$self->{chunk_size} = $conf->byte_value('chunk_size');
$self->{ignore} = [];
$self->{smart_mp3_chunking} = $conf->bool_value('smart_mp3_chunking');
lib/Brackup/Root.pm view on Meta::CPAN
my @good_dentries;
DENTRY:
foreach my $dentry (@_) {
next if $dentry eq "." || $dentry eq "..";
my $path = "$dir/$dentry";
$path =~ s!^\./!!;
# skip the digest database file. not sure if this is smart or not.
# for now it'd be kinda nice to have, but it's re-creatable from
# the backup meta files later, so let's skip it.
next if $self->{digcache_file} && $path eq $self->{digcache_file};
# GC: seems to work fine as of at least gpg 1.4.5, so commenting out
# gpg seems to barf on files ending in whitespace, blowing
# stuff up, so we just skip them instead...
#if ($self->gpg_rcpts && $path =~ /\s+$/) {
# warn "Skipping file ending in whitespace: <$path>\n";
# next;
#}
lib/Brackup/Root.pm view on Meta::CPAN
waitpid($pid, 0);
die "GPG failed: $!" if $? != 0; # If gpg return status is non-zero
}
1;
__END__
=head1 NAME
Brackup::Root - describes the source directory (and options) for a backup
=head1 EXAMPLE
In your ~/.brackup.conf file:
[SOURCE:bradhome]
path = /home/bradfitz/
gpg_recipient = 5E1B3EC5
chunk_size = 64MB
ignore = ^\.thumbnails/
lib/Brackup/Root.pm view on Meta::CPAN
ignore = ^\.ee/(minis|icons|previews)/
ignore = ^build/
noatime = 1
=head1 CONFIG OPTIONS
=over
=item B<path>
The directory to backup (recursively)
=item B<gpg_recipient>
The public key signature to encrypt data with. See L<Brackup::Manual::Overview/"Using encryption">.
=item B<chunk_size>
In units of bytes, kB, MB, etc. The max size of a chunk to be stored
on the target. Files over this size are cut up into chunks of this
size or smaller. The default is 64 MB if not specified.
=item B<ignore>
Perl5 regular expression of files not to backup. You may have multiple ignore lines.
=item B<noatime>
If true, don't backup access times. They're kinda useless anyway, and
just make the *.brackup metafiles larger.
=item B<merge_files_under>
In units of bytes, kB, MB, etc. If files are under this size. By
default this feature is off (value 0), purely because it's new, but 1
kB is a recommended size, and will probably be the default in the
future. Set it to 0 to explicitly disable.
=item B<max_composite_chunk_size>
In units of bytes, kB, MB, etc. The maximum size of a composite
chunk, holding lots of little files. If this is too big, you'll waste
more space with future iterative backups updating files locked into
this chunk with unchanged chunks.
Recommended, and default value, is 1 MB.
=item B<smart_mp3_chunking>
Boolean parameter. Set to one of {on,yes,true,1} to make mp3 files
chunked along their metadata boundaries. If a file has both ID3v1 and
ID3v2 chunks, the file will be cut into three parts: two little ones
for the ID3 tags, and one big one for the music bytes.
lib/Brackup/StoredChunk.pm view on Meta::CPAN
$copy->{$f} = $self->{$f};
}
$copy->{pchunk} = $pchunk;
return $copy;
}
sub set_composite_chunk {
my ($self, $cchunk, $from, $to) = @_;
$self->{compchunk} = $cchunk;
# forget our backup length/digest. this handle information
# to the stored chunk should be asked of our composite
# chunk in the future, when it's done populating.
$self->{backdigest} = undef;
$self->{backlength} = undef;
$self->forget_chunkref;
$self->{compfrom} = $from;
$self->{compto} = $to;
}
lib/Brackup/StoredChunk.pm view on Meta::CPAN
return 0;
}
# the original length, pre-encryption
sub length {
my $self = shift;
return $self->{pchunk}->length;
}
# the length, either encrypted or not
sub backup_length {
my $self = shift;
return $self->{backlength} if defined $self->{backlength};
$self->_populate_lengthdigest;
return $self->{backlength};
}
# the digest, either encrypted or not
sub backup_digest {
my $self = shift;
return $self->{backdigest} if $self->{backdigest};
$self->_populate_lengthdigest;
return $self->{backdigest};
}
sub _populate_lengthdigest {
my $self = shift;
# Composite chunk version
if (my $cchunk = $self->{compchunk}) {
$self->{backlength} = $cchunk->backup_length;
$self->{backdigest} = $cchunk->digest;
return 1;
}
die "ASSERT: encrypted length or digest not set" if $self->encrypted;
# Unencrypted version
$self->{backdigest} = "sha1:" . io_sha1($self->{pchunk}->raw_chunkref);
$self->{backlength} = $self->{pchunk}->length; # length of raw data
return 1;
}
sub chunkref {
my $self = shift;
if ($self->{_chunkref}) {
$self->{_chunkref}->seek(0, SEEK_SET);
return $self->{_chunkref};
}
# encrypting case: chunkref gets set via set_encrypted_chunkref in Backup::backup
croak "ASSERT: encrypted but no chunkref set" if $self->encrypted;
# caller/consistency check:
Carp::confess("Can't access chunkref on lite StoredChunk instance (handle only)")
if $self->{lite};
# non-encrypting case
return $self->{_chunkref} = $self->{pchunk}->raw_chunkref;
}
lib/Brackup/StoredChunk.pm view on Meta::CPAN
# to the format used by the metafile
sub to_meta {
my $self = shift;
my @parts = ($self->{pchunk}->offset,
$self->{pchunk}->length);
if (my $range = $self->range_in_composite) {
push @parts, (
$range,
$self->backup_digest,
);
} else {
push @parts, (
$self->backup_length,
$self->backup_digest,
);
}
# if the inventory database is lost, it should be possible to
# recover the inventory database from the *.brackup files.
# if a file only has on chunk, the digest(raw) -> digest(enc)
# can be inferred from the file's digest, then the stored
# chunk's digest. but if we have multiple chunks, we need
# to store each chunk's raw digest as well in the chunk
# list. we could do this all the time, but considering
lib/Brackup/StoredChunk.pm view on Meta::CPAN
sub inventory_value {
my $self = shift;
# when this chunk was stored as part of a composite chunk, the instructions
# are of form:
# sha1:deadbeef 0-50
# which means download "sha1:deadbeef", then the contents will be in from
# byte offset 0 to byte offset 50 (length of 50).
if (my $range = $self->range_in_composite) {
return join(" ",
$self->backup_digest,
$self->backup_length,
$range);
}
# else, the historical format:
# sha1:deadbeef <length>
return join(" ", $self->backup_digest, $self->backup_length);
}
1;
lib/Brackup/Target.pm view on Meta::CPAN
sub new {
my ($class, $confsec) = @_;
my $self = bless {}, $class;
$self->{name} = $confsec->name;
$self->{name} =~ s/^TARGET://
or die "No target found matching " . $confsec->name;
die "Target name must be only a-z, A-Z, 0-9, and _."
unless $self->{name} =~ /^\w+/;
$self->{keep_backups} = $confsec->value("keep_backups");
$self->{inv_db} =
Brackup::InventoryDatabase->new($confsec->value("inventorydb_file") ||
$confsec->value("inventory_db") ||
"$ENV{HOME}/.brackup-target-$self->{name}.invdb",
$confsec);
return $self;
}
sub name {
my $self = shift;
return $self->{name};
}
# return hashref of key/value pairs you want returned to you during a restore
# you should include anything you need to restore.
# keys must match /^\w+$/
sub backup_header {
return {}
}
# returns bool
sub has_chunk {
my ($self, $chunk) = @_;
die "ERROR: has_chunk not implemented in sub-class $self";
}
# returns true on success, or returns false or dies otherwise.
lib/Brackup/Target.pm view on Meta::CPAN
sub stored_chunk_from_inventory {
my ($self, $pchunk) = @_;
my $key = $pchunk->inventory_key;
my $db = $self->inventory_db;
my $invval = $db->get($key)
or return undef;
return Brackup::StoredChunk->new_from_inventory_value($pchunk, $invval);
}
# return a list of TargetBackupStatInfo objects representing the
# stored backup metafiles on this target.
sub backups {
my ($self) = @_;
die "ERROR: backups method not implemented in sub-class $self";
}
# downloads the given backup name to the current directory (with
# *.brackup extension)
sub get_backup {
my ($self, $name) = @_;
die "ERROR: get_backup method not implemented in sub-class $self";
}
# deletes the given backup from this target
sub delete_backup {
my ($self, $name) = @_;
die "ERROR: delete_backup method not implemented in sub-class $self";
}
# removes old metafiles from this target
sub prune {
my ($self, %opt) = @_;
my $keep_backups = $opt{keep_backups} || $self->{keep_backups}
or die "ERROR: keep_backups option not set\n";
die "ERROR: keep_backups option must be at least 1\n"
unless $keep_backups > 0;
# select backups to delete
my (%backups, @backups_to_delete) = ();
foreach my $backup_name (map {$_->filename} $self->backups) {
$backup_name =~ /^(.+)-\d+$/;
$backups{$1} ||= [];
push @{ $backups{$1} }, $backup_name;
}
foreach my $source (keys %backups) {
next if $opt{source} && $source ne $opt{source};
my @b = reverse sort @{ $backups{$source} };
push @backups_to_delete, splice(@b, ($keep_backups > $#b+1) ? $#b+1 : $keep_backups);
}
warn ($opt{dryrun} ? "Pruning:\n" : "Pruned:\n") if $opt{verbose};
foreach my $backup_name (@backups_to_delete) {
warn " $backup_name\n" if $opt{verbose};
$self->delete_backup($backup_name) unless $opt{dryrun};
}
return scalar @backups_to_delete;
}
# removes orphaned chunks in the target
sub gc {
my ($self, %opt) = @_;
# get all chunks and then loop through metafiles to detect
#Â referenced ones
my %chunks = map {$_ => 1} $self->chunks;
my $total_chunks = scalar keys %chunks;
my $tempfile = +(tempfile())[1];
my @backups = $self->backups;
BACKUP: foreach my $i (0 .. $#backups) {
my $backup = $backups[$i];
warn sprintf "Collating chunks from backup %s [%d/%d]\n",
$backup->filename, $i+1, scalar(@backups)
if $opt{verbose};
$self->get_backup($backup->filename, $tempfile);
my $decrypted_backup = new Brackup::DecryptedFile(filename => $tempfile);
my $parser = Brackup::Metafile->open($decrypted_backup->name);
$parser->readline; # skip header
ITEM: while (my $it = $parser->readline) {
next ITEM unless $it->{Chunks};
my @item_chunks = map { (split /;/)[3] } grep { $_ } split(/\s+/, $it->{Chunks} || "");
delete $chunks{$_} for (@item_chunks);
}
}
my @orphaned_chunks = keys %chunks;
# report orphaned chunks
lib/Brackup/Target.pm view on Meta::CPAN
}
1;
__END__
=head1 NAME
Brackup::Target - describes the destination for a backup
=head1 EXAMPLE
In your ~/.brackup.conf file:
[TARGET:amazon]
type = Amazon
aws_access_key_id = ...
aws_secret_access_key = ....
lib/Brackup/Target.pm view on Meta::CPAN
B<Filesystem> -- see L<Brackup::Target::Filesystem> for configuration details
B<Ftp> -- see L<Brackup::Target::Ftp> for configuration details
B<Sftp> -- see L<Brackup::Target::Sftp> for configuration details
B<Amazon> -- see L<Brackup::Target::Amazon> for configuration details
B<Amazon> -- see L<Brackup::Target::CloudFiles> for configuration details
=item B<keep_backups>
The default number of recent backups to keep when running I<brackup-target prune>.
=item B<inventorydb_file>
The location of the L<Brackup::InventoryDatabase> inventory database file for
this target e.g.
[TARGET:amazon]
type = Amazon
aws_access_key_id = ...
aws_secret_access_key = ...
lib/Brackup/Target/Amazon.pm view on Meta::CPAN
use Net::Amazon::S3 0.42;
use DateTime::Format::ISO8601;
# fields in object:
# s3 -- Net::Amazon::S3
# access_key_id
# sec_access_key_id
# prefix
# location
# chunk_bucket : $self->{prefix} . "-chunks";
# backup_bucket : $self->{prefix} . "-backups";
# backup_prefix : added to the front of backup names when stored
#
sub new {
my ($class, $confsec) = @_;
my $self = $class->SUPER::new($confsec);
$self->{access_key_id} = $confsec->value("aws_access_key_id")
or die "No 'aws_access_key_id'";
$self->{sec_access_key_id} = $confsec->value("aws_secret_access_key")
or die "No 'aws_secret_access_key'";
$self->{prefix} = $confsec->value("aws_prefix") || $self->{access_key_id};
$self->{location} = $confsec->value("aws_location") || undef;
$self->{backup_prefix} = $confsec->value("backup_prefix") || undef;
$self->_common_s3_init;
my $s3 = $self->{s3};
my $buckets = $s3->buckets or die "Failed to get bucket list";
unless (grep { $_->{bucket} eq $self->{chunk_bucket} } @{ $buckets->{buckets} }) {
$s3->add_bucket({ bucket => $self->{chunk_bucket}, location_constraint => $self->{location} })
or die "Chunk bucket creation failed\n";
}
unless (grep { $_->{bucket} eq $self->{backup_bucket} } @{ $buckets->{buckets} }) {
$s3->add_bucket({ bucket => $self->{backup_bucket}, location_constraint => $self->{location} })
or die "Backup bucket creation failed\n";
}
return $self;
}
sub _common_s3_init {
my $self = shift;
$self->{chunk_bucket} = $self->{prefix} . "-chunks";
$self->{backup_bucket} = $self->{prefix} . "-backups";
$self->{s3} = Net::Amazon::S3->new({
aws_access_key_id => $self->{access_key_id},
aws_secret_access_key => $self->{sec_access_key_id},
retry => 1,
});
}
# ghetto
sub _prompt {
my ($q) = @_;
print "$q";
my $ans = <STDIN>;
$ans =~ s/^\s+//;
$ans =~ s/\s+$//;
return $ans;
}
# Location and backup_prefix aren't required for restores, so they're omitted here
sub backup_header {
my ($self) = @_;
return {
"AWSAccessKeyID" => $self->{access_key_id},
"AWSPrefix" => $self->{prefix},
};
}
# Location and backup_prefix aren't required for restores, so they're omitted here
sub new_from_backup_header {
my ($class, $header, $confsec) = @_;
my $accesskey = ($ENV{'AWS_KEY'} ||
$ENV{'AWS_ACCESS_KEY_ID'} ||
$header->{AWSAccessKeyID} ||
$confsec->value('aws_access_key_id') ||
_prompt("Your Amazon AWS access key? "))
or die "Need your Amazon access key.\n";
my $sec_accesskey = ($ENV{'AWS_SEC_KEY'} ||
$ENV{'AWS_ACCESS_KEY_SECRET'} ||
lib/Brackup/Target/Amazon.pm view on Meta::CPAN
my $self = bless {}, $class;
$self->{access_key_id} = $accesskey;
$self->{sec_access_key_id} = $sec_accesskey;
$self->{prefix} = $prefix || $self->{access_key_id};
$self->_common_s3_init;
return $self;
}
sub has_chunk {
my ($self, $chunk) = @_;
my $dig = $chunk->backup_digest; # "sha1:sdfsdf" format scalar
my $res = eval { $self->{s3}->head_key({ bucket => $self->{chunk_bucket}, key => $dig }); };
return 0 unless $res;
return 0 if $@ && $@ =~ /key not found/;
return 0 unless $res->{content_type} eq "x-danga/brackup-chunk";
return 1;
}
sub load_chunk {
my ($self, $dig) = @_;
my $bucket = $self->{s3}->bucket($self->{chunk_bucket});
my $val = $bucket->get_key($dig)
or return 0;
return \ $val->{value};
}
sub store_chunk {
my ($self, $chunk) = @_;
my $dig = $chunk->backup_digest;
my $fh = $chunk->chunkref;
my $chunkref = do { local $/; <$fh> };
my $try = sub {
eval {
$self->{s3}->add_key({
bucket => $self->{chunk_bucket},
key => $dig,
value => $chunkref,
content_type => 'x-danga/brackup-chunk',
lib/Brackup/Target/Amazon.pm view on Meta::CPAN
}
# returns a list of names of all chunks
sub chunks {
my $self = shift;
my $chunks = $self->{s3}->list_bucket_all({ bucket => $self->{chunk_bucket} });
return map { $_->{key} } @{ $chunks->{keys} };
}
sub store_backup_meta {
my ($self, $name, $fh, $meta) = @_;
$name = $self->{backup_prefix} . "-" . $name if defined $self->{backup_prefix};
eval {
my $bucket = $self->{s3}->bucket($self->{backup_bucket});
$bucket->add_key_filename(
$name,
$meta->{filename},
{ content_type => 'x-danga/brackup-meta' },
);
};
}
sub backups {
my $self = shift;
my @ret;
my $backups = $self->{s3}->list_bucket_all({ bucket => $self->{backup_bucket} });
foreach my $backup (@{ $backups->{keys} }) {
my $iso8601 = DateTime::Format::ISO8601->parse_datetime( $backup->{last_modified} );
push @ret, Brackup::TargetBackupStatInfo->new($self, $backup->{key},
time => $iso8601->epoch,
size => $backup->{size});
}
return @ret;
}
sub get_backup {
my $self = shift;
my ($name, $output_file) = @_;
my $bucket = $self->{s3}->bucket($self->{backup_bucket});
my $val = $bucket->get_key($name)
or return 0;
$output_file ||= "$name.brackup";
open(my $out, ">$output_file") or die "Failed to open $output_file: $!\n";
my $outv = syswrite($out, $val->{value});
die "download/write error" unless $outv == do { use bytes; length $val->{value} };
close $out;
return 1;
}
sub delete_backup {
my $self = shift;
my $name = shift;
my $bucket = $self->{s3}->bucket($self->{backup_bucket});
return $bucket->delete_key($name);
}
sub chunkpath {
my $self = shift;
my $dig = shift;
return $dig;
}
lib/Brackup/Target/Amazon.pm view on Meta::CPAN
return 0 unless $res;
return 0 if $@ && $@ =~ /key not found/;
return 0 unless $res->{content_type} eq "x-danga/brackup-chunk";
return $res->{content_length};
}
1;
=head1 NAME
Brackup::Target::Amazon - backup to Amazon's S3 service
=head1 EXAMPLE
In your ~/.brackup.conf file:
[TARGET:amazon]
type = Amazon
aws_access_key_id = ...
aws_secret_access_key = ....
aws_prefix = ....
backup_prefix = ....
=head1 CONFIG OPTIONS
All options may be omitted unless specified.
=over
=item B<type>
I<(Mandatory.)> Must be "B<Amazon>".
lib/Brackup/Target/Amazon.pm view on Meta::CPAN
=item B<aws_access_key_id>
I<(Mandatory.)> Your Amazon Web Services access key id.
=item B<aws_secret_access_key>
I<(Mandatory.)> Your Amazon Web Services secret password for the above access key. (not your Amazon password)
=item B<aws_prefix>
If you want to setup multiple backup targets on a single Amazon account you can
use different prefixes. This string is used to name the S3 buckets created by
Brackup. If not specified it defaults to the AWS access key id.
=item B<aws_location>
Sets the location constraint of the new buckets. If left unspecified, the
default S3 datacenter location will be used. Otherwise, you can set it
to 'EU' for an AWS European data center - note that costs are different.
This has only effect when your backup environment is initialized in S3 (i.e.
when buckets are created). If you want to move an existing backup environment
to another datacenter location, you have to delete its buckets before or create
a new one by specifing a different I<aws_prefix>.
=item B<backup_prefix>
When storing the backup metadata file to S3, the string specified here will
be prefixed onto the backup name. This is useful if you are collecting
backups from several hosts into a single Amazon S3 account but need to
be able to differentiate them; set your prefix to be the hostname
of each system, for example.
=back
=head1 SEE ALSO
L<Brackup::Target>
L<Net::Amazon::S3> -- required module to use Brackup::Target::Amazon
lib/Brackup/Target/CloudFiles.pm view on Meta::CPAN
use Carp qw(croak);
eval { require Net::Mosso::CloudFiles } or die "You need the Net::Mosso::CloudFiles module installed to use the CloudFiles target in brackup. Please install this module first.\n\n";
# fields in object:
# cf -- Net::Mosso::CloudFiles
# username
# apiKey
# chunkContainer : $self->{username} . "-chunks";
# backupContainer : $self->{username} . "-backups";
#
sub new {
my ($class, $confsec) = @_;
my $self = $class->SUPER::new($confsec);
$self->{username} = $confsec->value("cf_username")
or die "No 'cf_username'";
$self->{apiKey} = $confsec->value("cf_api_key")
or die "No 'cf_api_key'";
$self->_common_cf_init;
return $self;
}
sub _common_cf_init {
my $self = shift;
$self->{chunkContainerName} = $self->{username} . "-chunks";
$self->{backupContainerName} = $self->{username} . "-backups";
$self->{cf} = Net::Mosso::CloudFiles->new(
user => $self->{username},
key => $self->{apiKey}
);
#createContainer makes the object and returns it, or returns it
#if it already exists
$self->{chunkContainer} =
$self->{cf}->create_container(name => $self->{chunkContainerName})
or die "Failed to get chunk container";
$self->{backupContainer} =
$self->{cf}->create_container(name => $self->{backupContainerName})
or die "Failed to get backup container";
}
sub _prompt {
my ($q) = @_;
my $ans = <STDIN>;
$ans =~ s/^\s+//;
$ans =~ s/\s+$//;
return $ans;
}
sub new_from_backup_header {
my ($class, $header, $confsec) = @_;
my $username = ($ENV{'CF_USERNAME'} ||
$confsec->value('cf_username') ||
_prompt("Your CloudFiles username: "))
or die "Need your Cloud Files username.\n";
my $apiKey = ($ENV{'CF_API_KEY'} ||
$confsec->value('cf_api_key') ||
_prompt("Your CloudFiles api key: "))
lib/Brackup/Target/CloudFiles.pm view on Meta::CPAN
my $self = bless {}, $class;
$self->{username} = $username;
$self->{apiKey} = $apiKey;
$self->_common_cf_init;
return $self;
}
sub has_chunk {
my ($self, $chunk) = @_;
my $dig = $chunk->backup_digest; # "sha1:sdfsdf" format scalar
my $res = $self->{chunkContainer}->object(name => $dig);
return 0 unless $res;
#return 0 if $@ && $@ =~ /key not found/;
#TODO: check for content type?
#return 0 unless $res->{content_type} eq "x-danga/brackup-chunk";
return 1;
lib/Brackup/Target/CloudFiles.pm view on Meta::CPAN
sub load_chunk {
my ($self, $dig) = @_;
my $val = $self->{chunkContainer}->object(name => $dig)->get
or return 0;
return \ $val;
}
sub store_chunk {
my ($self, $chunk) = @_;
my $dig = $chunk->backup_digest;
my $chunkref = $chunk->chunkref;
my $content = do { local $/; <$chunkref> };
$self->{chunkContainer}->object(
name => $dig,
content_type => 'x-danga/brackup-chunk'
)->put($content);
return 1;
lib/Brackup/Target/CloudFiles.pm view on Meta::CPAN
sub chunks {
my $self = shift;
my @objectNames;
my @objects = $self->{chunkContainer}->objects->all;
foreach (@objects){ push @objectNames, $_->name;}
return @objectNames;
}
sub store_backup_meta {
my ($self, $name, $fh) = @_;
my $content = do { local $/; <$fh> };
$self->{backupContainer}->object(name => $name)->put($content);
return 1;
}
sub backups {
my $self = shift;
my @ret;
my @backups = $self->{backupContainer}->objects->all;
foreach my $backup (@backups) {
push @ret, Brackup::TargetBackupStatInfo->new(
$self, $backup->name,
time => str2time($backup->last_modified),
size => $backup->size);
}
return @ret;
}
sub get_backup {
my $self = shift;
my ($name, $output_file) = @_;
my $val = $self->{backupContainer}->object(name => $name)->get
or return 0;
$output_file ||= "$name.brackup";
open(my $out, ">$output_file") or die "Failed to open $output_file: $!\n";
my $outv = syswrite($out, $val);
die "download/write error" unless
$outv == do { use bytes; length $val };
close $out;
return 1;
}
sub delete_backup {
my $self = shift;
my $name = shift;
return $self->{backupContainer}->object(name => $name)->delete;
}
#############################################################
# These functions are for the brackup-verify-inventory script
#############################################################
sub chunkpath {
my $self = shift;
my $dig = shift;
lib/Brackup/Target/CloudFiles.pm view on Meta::CPAN
my $size = $obj->size;
return $size;
}
1;
=head1 NAME
Brackup::Target::CloudFiles - backup to Rackspace's CloudFiles Service
=head1 EXAMPLE
In your ~/.brackup.conf file:
[TARGET:cloudfiles]
type = CloudFiles
cf_username = ...
cf_api_key = ....
lib/Brackup/Target/Filebased.pm view on Meta::CPAN
}
return join("/", @parts) . "/$fulldig.chunk";
}
sub metapath {
my ($self, $name) = @_;
$name ||= '';
return "backups/$name";
}
1;
( run in 2.502 seconds using v1.01-cache-2.11-cpan-49f99fa48dc )