Brackup
view release on metacpan or search on metacpan
- actually strip non-ASCII characters (gary.richardson@gmail.com)
- smarts to filesystem target, noticed when using sshfs (slow filesystem).
does .partial files now, is smart about not overwriting existing chunk
that's there, etc.
- bradfitz: optional smart mp3 chunking. (start of file-aware chunking
strategies) with smart mp3 chunking, the ID3 tags are kept in separate
chunks, so future re-tagging of your music won't force iterative backups
to re-upload all the music bytes again... just the updated
metadata (tiny)
- Add a new option to the Amazon S3 target to force a prefix to be
added to the names of any saved backups.
It might be worth moving this up into Brackup itself at some point,
since it's probably useful for other network-based targets.
- Fix Restore.pm to use binmode for operating systems that care about
such things.
- clean up old, dead code in Amazon target (the old inventory db which
is now an official part of the core, and in the Target base class)
- retry PUTs to Amazon on failure, a few times, pausing in-between,
in case it was a transient error, as seems to happen occasionally
- halve number of stats when walking backup root
- cleanups, strictness
- don't upload meta files when in dry-run mode
- update amazon target support to work again, with the new inventory
database support (now separated from the old digest database)
- merge in the refactoring branch, in which a lot of long-standing
pet peeves in the design were rethought/redone.
- make decryption --use-agent and --batch, and help out if env not set
and gpg-agent probably not running
Overall, we now use Google code hosting:
http://code.google.com/p/brackup/
http://code.google.com/p/brackup/issues/list
If you submit a patch to the mailing list, please use this:
http://codereview.appspot.com/
It's a web-based code review tool, and it has command-line tools you
can download to automate the uploads. Send the codereview issue URL
to the mailing list.
doc/notes.txt view on Meta::CPAN
compressed digest, even if not encrypted as well.
Maybe we don't need per-chunk meta files:
-- can get it all from .brackup (meta)files on the server.
(TODO: abstract out parser for multiple users)
---
smart chunk-sizing on certain files w/ metadata and data separate:
like mp3 files and their id3. have a smart chunker that's the data
part vs. the id3 part, so updating id3 later doesn't reupload the
entire data part. :-)
doc/overview.txt view on Meta::CPAN
Files stored on Amazon/Google are of form:
-- meta files: backup_rootname-yyyymmddnn.meta, encrypted (YAML?) file mapping relative paths from backup directory root to the stat() information, original SHA1, and array of chunk keys (SHA1s of encrypted chunks) that comprise the file.
-- [sha1ofencryptedchunk].chunk -- content being <= ,say, 20MB chunk of encrypted data.
Then every night different hosts/laptops recurse directory trees,
consult a stat() cache (on, say, inode number, mtime, size, whatever)
and do SHA1 calculations on changed files, lookup rest from cache, and
build the metafile, upload any new chunks, encrypt the metafile,
upload the metafile.
Result:
-- I can restore any host from any point in time, with Amazon/Google
storing all my data, and only paying $0.15 cents/GB-month.
doc/todo.txt view on Meta::CPAN
---
Maybe we don't need per-chunk meta files:
-- can get it all from .brackup (meta)files on the server.
(TODO: abstract out parser for multiple users)
---
smart chunk-sizing on certain files w/ metadata and data separate:
like mp3 files and their id3. have a smart chunker that's the data
part vs. the id3 part, so updating id3 later doesn't reupload the
entire data part. :-)
lib/Brackup/Backup.pm view on Meta::CPAN
my $root = $self->{root};
my $target = $self->{target};
my $stats = Brackup::BackupStats->new;
my @gpg_rcpts = $self->{root}->gpg_rcpts;
my $n_kb = 0.0; # num: kb of all files in root
my $n_files = 0; # int: # of files in root
my $n_kb_done = 0.0; # num: kb of files already done with (uploaded or skipped)
# if we're pre-calculating the amount of data we'll
# actually need to upload, store it here.
my $n_files_up = 0;
my $n_kb_up = 0.0;
my $n_kb_up_need = 0.0; # by default, not calculated/used.
my $n_files_done = 0; # int
my @files; # Brackup::File objs
$self->debug("Discovering files in ", $root->path, "...\n");
$self->report_progress(0, "Discovering files in " . $root->path . "...");
$root->foreach_file(sub {
lib/Brackup/Backup.pm view on Meta::CPAN
if ($fn % 100 == 0) { warn "$fn / $n_files ...\n"; }
foreach my $pc ($f->chunks) {
if ($target->stored_chunk_from_inventory($pc)) {
$pc->forget_chunkref;
next;
}
$n_kb_up_need += $pc->length / 1024;
$pc->forget_chunkref;
}
}
warn "kb need to upload = $n_kb_up_need\n";
$stats->timestamp('Calc Needed');
}
my $chunk_iterator = Brackup::ChunkIterator->new(@files);
undef @files;
$stats->timestamp('Chunk Iterator');
my $gpg_iter;
my $gpg_pm; # gpg ProcessManager
lib/Brackup/Backup.pm view on Meta::CPAN
else {
print $metafh $cur_file->as_rfc822([ @stored_chunks ], $self) if $metafh;
}
$self->add_saved_file($cur_file, [ @stored_chunks ]) if $self->{savefiles};
$n_files_done++;
$n_kb_done += $cur_file->size / 1024;
$cur_file = undef;
};
my $show_status = sub {
# use either size of files in normal case, or if we pre-calculated
# the size-to-upload (by looking in inventory, then we'll show the
# more accurate percentage)
my $percdone = 100 * ($n_kb_up_need ?
($n_kb_up / $n_kb_up_need) :
($n_kb_done / $n_kb));
my $mb_remain = ($n_kb_up_need ?
($n_kb_up_need - $n_kb_up) :
($n_kb - $n_kb_done)) / 1024;
$self->debug(sprintf("* %-60s %d/%d (%0.02f%%; remain: %0.01f MB)",
$cur_file->path, $n_files_done, $n_files, $percdone,
lib/Brackup/InventoryDatabase.pm view on Meta::CPAN
=over
=item B<1) Exists in inventory database; not on target>
If a chunk exists in the inventory database, but not on the target, brackup
won't store it on the target, and you'll think a backup succeeded, but
it's not actually there.
=item B<2a) Exists on target; not in inventory database (without encryption)>
You re-upload it to the target, so you waste time & bandwidth, but no
extra disk space is wasted, and no chunks are orphaned. Actually,
chunks are un-orphaned, as the inventory database is now updated and
contains the chunk you just uploaded.
=item B<2b) Exists on target; not in inventory database (with encryption)>
When using encryption, each time a chunk is encrypted with gpg, the
contents are different. So if the inventory database says a given
chunk isn't already stored on the server, it will be re-encrypted and
stored (uploaded) again. You may or may not have an orphaned chunk on
the server, depending on whether or not it's referenced by any other
*.brackup meta files.
=back
For those reasons, it's somewhat important that your inventory
database be kept around and not deleted. If you're running brackup to
the same target from different computers, you might want to sync up
your inventory databases with each other, so you don't do unnecessary
uploads to the target.
Tools to rebuild your inventory database from the target's enumeration
of its chunks and the target's *.brackup metafiles isn't yet done, but
would be pretty easy. (this is a TODO item)
In any case, it's not tragic if you lose your inventory database... it
just means you'll need to upload more stuff and maybe waste some disk
space until you next run a 'L<brackup-target> gc' garbage collection,
which cleans up orphaned chunks. If you're feeling paranoid, it's safer
to delete your inventory database, tricking Brackup into thinking your
target is empty (even if it's not), rather than Brackup thinking your
target has something when it actually doesn't.
=head1 DETAILS
=head2 Storage type
lib/Brackup/Target/Amazon.pm view on Meta::CPAN
};
my $rv;
my $n_fails = 0;
while (!$rv && $n_fails < 5) {
$rv = $try->();
last if $rv;
# transient failure?
$n_fails++;
warn "Error uploading chunk $chunk [$@]... will do retry \#$n_fails in 5 seconds ...\n";
sleep 5;
}
unless ($rv) {
warn "Error uploading chunk again: " . $self->{s3}->errstr . "\n";
return 0;
}
return 1;
}
sub delete_chunk {
my ($self, $dig) = @_;
my $bucket = $self->{s3}->bucket($self->{chunk_bucket});
return $bucket->delete_key($dig);
}
lib/Brackup/Target/GoogleAppEngine.pm view on Meta::CPAN
return $self->_init;
}
sub _init {
my $self = shift;
$self->{url} =~ s!/$!!;
my $conn_cache = LWP::ConnCache->new(total_capacity => 10);
$self->{ua} = LWP::UserAgent->new(conn_cache => $conn_cache);
$self->{upload_urls} = [];
return $self;
}
sub _prompt {
my ($q) = @_;
print $q if $q;
my $ans = <STDIN>;
$ans =~ s/^\s+//;
$ans =~ s/\s+$//;
return $ans;
lib/Brackup/Target/GoogleAppEngine.pm view on Meta::CPAN
return 0;
}
sub _eurl {
my $a = defined $_[0] ? $_[0] : "";
$a =~ s/([^a-zA-Z0-9_\,\-.\/\\\: ])/uc sprintf("%%%02x",ord($1))/eg;
$a =~ tr/ /+/;
return $a;
}
sub _get_upload_url {
my $self = shift;
my $for_backup = shift || 0;
if (!$for_backup && @{$self->{upload_urls}}) {
my $url = shift @{$self->{upload_urls}};
die "Bogus URL: $url" unless $url =~ /^http/;
return $url;
}
my $count = $for_backup ? 1 : 10;
my $req = HTTP::Request->new("GET",
"$self->{url}/get_upload_urls?" .
"for_backup=$for_backup&" .
"count=$count&" .
"password=" . _eurl($self->{password}) . "&" .
"user_email=" . $self->{user_email});
my $res = $self->{ua}->request($req);
if ($res->is_success) {
$self->{upload_urls} = [ split(/\s*\n\s*/, $res->content) ];
} else {
die "Failed to get upload URLs: " . $res->status_line . "\n" . $res->content;
}
my $url = shift @{$self->{upload_urls}};
die "Bogus URL: $url" unless $url =~ /^http/;
return $url;
}
sub store_chunk {
my ($self, $chunk) = @_;
my $dig = $chunk->backup_digest;
my $blen = $chunk->backup_length;
my $chunkref = $chunk->chunkref;
my $upload_url = $self->_get_upload_url
or die;
my $filename = $dig;
$filename =~ s/:/_/;
$filename .= ".chunk";
print "Storing chunk: $dig\n";
my $content = do { local $/; <$chunkref> };
my $req = HTTP::Request::Common::POST($upload_url,
Content_Type => 'form-data',
Content => [
"password" => $self->{password},
"user_email" => $self->{user_email},
"algo_digest" => $dig,
"size" => $blen,
"file" => [ undef, $filename,
"Content-Type" => "x-danga/brackup-chunk",
Content => $content ]
]);
lib/Brackup/Target/GoogleAppEngine.pm view on Meta::CPAN
my $self = shift;
}
sub store_backup_meta {
my ($self, $name, $fh, $meta) = @_;
$meta ||= {};
print "Storing backup: $name\n";
my $upload_url = $self->_get_upload_url(1) # for backup
or die;
my $content = do { local $/; <$fh> };
my $req = HTTP::Request::Common::POST($upload_url,
Content_Type => 'form-data',
Content => [
"password" => $self->{password},
"user_email" => $self->{user_email},
"encrypted" => $meta->{is_encrypted} ? 1 : 0,
"title" => $name,
"file" => [ undef, $name,
"Content-Type" => "x-danga/brackup-backup",
Content => $content ]
]);
lib/Brackup/Target/GoogleAppEngine.pm view on Meta::CPAN
=over
=item B<type>
Must be "B<GoogleAppEngine>".
=item B<user_email>
Email address that you've logged into your brackup-gae-server instance
with and configured uploading.
=item B<password>
Your brackup-gae-server password. B<NOT> your Google account's password.
You should make a separate password just for this.
=item B<server_url>
URL to your brackup-gae-server instance.
lib/Brackup/Target/Riak.pm view on Meta::CPAN
my $sub = sub {
my $obj = $bucket->new_object($key, $data,
content_type => $content_type,
);
$obj->store;
};
my $obj = $self->_retry("storing $type $key", $sub);
unless ($obj->exists) {
warn "Error uploading chunk again: " . $obj->status . "\n";
return 0;
}
return 1;
}
sub store_chunk {
my ($self, $chunk) = @_;
my $dig = $chunk->backup_digest;
my $fh = $chunk->chunkref;
my $chunkref = do { local $/; <$fh> };
t/01-backup-ftp.t view on Meta::CPAN
# -*-perl-*-
#
# Backup test of ftp target - set $ENV{BRACKUP_TEST_FTP} to run
#
# By default, attempts to do anonymous uploads to localhost to a 'tmp'
# directory within your ftp root, so configure your ftp server appropriately,
# or set FTP_HOST, FTP_USER, and FTP_PASSWORD environment variables to modify.
#
# Note that unlike the equivalent Filesystem and Sftp tests, this one does not
# cleanup after itself, since in the default anonymous mode the owner of the
# uploaded files is likely to be different from the user running the test.
#
use strict;
use Test::More;
use Brackup::Test;
use FindBin qw($Bin);
use Brackup::Util qw(tempfile);
if ($ENV{BRACKUP_TEST_FTP}) {
t/03-composite-ftp.t view on Meta::CPAN
# -*-perl-*-
#
# Composite test of ftp target - set $ENV{BRACKUP_TEST_FTP} to run
#
# By default, attempts to do anonymous uploads to localhost to a 'tmp'
# directory within your ftp root, so configure your ftp server appropriately,
# or set FTP_HOST, FTP_USER, and FTP_PASSWORD environment variables to modify.
#
# Note that unlike the equivalent Filesystem and Sftp tests, this one does not
# cleanup after itself, since in the default anonymous mode the owner of the
# uploaded files is likely to be different from the user running the test.
#
use strict;
use Test::More;
use Brackup::Test;
use FindBin qw($Bin);
use Brackup::Util qw(tempfile);
############### Setup
( run in 3.170 seconds using v1.01-cache-2.11-cpan-b16cb0d3907 )