App-SimpleBackuper
view release on metacpan or search on metacpan
0.2.26 2024-07-27
- Reduced peak memory usage.
- Minor fix.
0.2.25 2021-03-27
- Use a static feature bundle (thanks to Graham Knop aka @haarg)
0.2.24 2021-02-28
- More verbosity about priorities in choosing files to delete: added user's priority and backup version score.
- Fix: first deleting heaviest files of one priority.
0.2.23 2021-02-13
- Version bump
0.2.22 2021-02-13
- Improve files version priorities.
- Fix versions dups.
- Fix crash while deleting not existent file while storage checking.
- Fix warning in test.
- Speedup initialize queue of blocks to delete.
- Speedup DB initialization.
- Not finished yet backups marks in stats as 'unfinished'.
- DB format v2: parts and blocks saved too for fastest DB launch.
0.2.21 2020-12-20
- Non-zero exit code if some files failed to backup.
- Report about deleted files for free up space.
0.2.20 2020-12-12
- Before backup storage fix instead of storage check (for more simplicity).
0.2.19 2020-12-02
- Fix: periodically db saving cancels part of backuped files.
- Fix: dirs in path to backuped files doesn't counts in backup counters.
0.2.18 2020-12-01
- Fix: non-ascii files masks.
- Save DB every 10 min.
- Zero compression for files with media extentions, because it's useless.
0.2.17 2020-11-26
- Reduced memory utilization at start.
- Undebug.
- Zero compression level in tests.
0.2.16 2020-11-21
- Fix backuping crash when dir from config doesn't exists.
0.2.15 2020-11-15
- Fix bug: infonity attempts ti read when reads more than 0 bytes and then read fails.
- Minor cosmetic fix for verbose mode.
0.2.14 2020-11-15
- Fix bug with output compression ratio of zero file.
0.2.13 2020-11-08
- Remove debug of OOM on openBSD: cause of OOM in Compress::Raw::Lzma & this compression method eats many of RAM.
0.2.12 2020-11-05
- Removed requirements included to perl core.
- Debug OOM for openbsd.
0.2.10 2020-10-31
- Minor docs fix.
- Minor fix in stats command.
- Progress info minor fix.
- Tests run in verbose mode.
0.2.9 2020-10-26
- Fix merging blocks of file parts.
0.2.8 2020-10-22
- Minor POD fix.
0.2.7 2020-10-21
- Fix case with ignoring some dirs to backup.
- Minor storage integrity report fix.
- Minor verbosity output fix.
- Minor POD improvement.
0.2.6 2020-10-18
- POD minor change.
0.2.5 2020-10-18
- Version bump.
0.2.4 2020-10-18
- Added info about installing required libs to docs.
- Fix some restoring bugs.
- Some kwalitee improvements.
0.2.1 2020-10-15
- Some kwalitee improvements.
0.2.0 2020-10-15
- Version bump.
0.1.1 2020-10-15
- Fix cpan module requirements.
- POD moved to .pod-file.
0.1 2020-10-12
- Fully rewrited app.
- Data encryption on host with backuped data.
- Data compression with LZMA algorythm.
- Data deduplication with subfile graduation (incremental backup).
- Ssh and local FS backups location.
- Different files priorities.
- Compact for RAM and for FS database.
bin/simple-backuper
Changes
cpanfile
cpanfile.snapshot
lib/App/SimpleBackuper.pm
lib/App/SimpleBackuper.pod
lib/App/SimpleBackuper/_BlockDelete.pm
lib/App/SimpleBackuper/_BlocksInfo.pm
lib/App/SimpleBackuper/_format.pm
lib/App/SimpleBackuper/_print_table.pm
lib/App/SimpleBackuper/Backup.pm
{
"abstract" : "Just simple backuper app with incremental compressed encrypted backups stored on remote ssh server",
"author" : [
"Dmitry Novozhilov <Dmitry@Novozhilov.ru>"
],
"dynamic_config" : 1,
"generated_by" : "ExtUtils::MakeMaker version 7.64, CPAN::Meta::Converter version 2.150010",
"license" : [
"gpl_3"
],
"meta-spec" : {
"url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec",
"test" : {
"requires" : {
"Test::Spec" : "0"
}
}
},
"release_status" : "stable",
"resources" : {
"repository" : {
"type" : "git",
"url" : "git://github.com/dmitry-novozhilov/simple-backuper.git",
"web" : "https://github.com/dmitry-novozhilov/simple-backuper"
}
},
"version" : "v0.2.26",
"x_serialization_backend" : "JSON::PP version 4.07"
}
---
abstract: 'Just simple backuper app with incremental compressed encrypted backups stored on remote ssh server'
author:
- 'Dmitry Novozhilov <Dmitry@Novozhilov.ru>'
build_requires:
ExtUtils::MakeMaker: '0'
Test::Spec: '0'
configure_requires:
ExtUtils::MakeMaker: '0'
dynamic_config: 1
generated_by: 'ExtUtils::MakeMaker version 7.64, CPAN::Meta::Converter version 2.150010'
license: gpl
requires:
Compress::Raw::Lzma: '0'
Const::Fast: '0'
Crypt::OpenSSL::RSA: '0'
Crypt::Rijndael: '0'
Net::SFTP::Foreign: '0'
Text::Glob: '0'
Try::Tiny: '0'
perl: '5.014'
resources:
repository: git://github.com/dmitry-novozhilov/simple-backuper.git
version: v0.2.26
x_serialization_backend: 'CPAN::Meta::YAML version 0.018'
Makefile.PL view on Meta::CPAN
use warnings;
use ExtUtils::MakeMaker;
use ExtUtils::Manifest qw(mkmanifest);
die "OS unsupported\n" if $^O eq "MSWin32";
mkmanifest();
WriteMakefile(
NAME => 'App::SimpleBackuper',
ABSTRACT => 'Just simple backuper app with incremental compressed encrypted backups stored on remote ssh server',
AUTHOR => 'Dmitry Novozhilov <Dmitry@Novozhilov.ru>',
VERSION_FROM => 'lib/App/SimpleBackuper.pm',
LICENSE => 'gpl_3',
test => {TESTS => 't/*.t t/*/*.t'},
TEST_REQUIRES => {'Test::Spec' => 0},
EXE_FILES => ['bin/simple-backuper'],
MIN_PERL_VERSION=> 5.014,
PREREQ_PM => {
'Crypt::Rijndael' => 0,
'Crypt::OpenSSL::RSA' => 0,
'Compress::Raw::Lzma' => 0,
'Text::Glob' => 0,
'Try::Tiny' => 0,
'Net::SFTP::Foreign' => 0,
'Const::Fast' => 0,
},
META_MERGE => {
'meta-spec' => { version => 2 },
resources => {
repository => {
type => 'git',
url => 'git://github.com/dmitry-novozhilov/simple-backuper.git',
web => 'https://github.com/dmitry-novozhilov/simple-backuper',
},
},
},
);
[](https://metacpan.org/release/App-SimpleBackuper)
# What's this
**Simple-backuper** is a simple tool for backuping files and restoring it from backups.
# Benefits
- Simplicity and transparency. Few lib files and one short script. Most programmers can understand it.
- Efficient use of disk space (incremental backup):
- Automatic deduplication of parts of files (most modified files differ only partially).
- All files will be compressed with archivator (compression level may be configured).
- Incremental backup format doesn't require initial data snapshot.
- Security:
- All files will be encrypted with AES256 + RSA4096.
- Encryption doing before data send to storage host.
- For backuping you don't need to keep private RSA key accessible to this program. It needs only for restoring.
- Thus, even with the backup data, no one can get the source files from them. And also no one can fake a backup.
- You can specify different priorities for any files.
- For recover your backup you need only: access to storage host, your crypting keys and some backup options.
- You can backup to local directory or to remote sftp server.
- Requires on backuper host: perl and some perl libs.
- Requires on SFTP storage host: disk space only.
# Installing
You can install simple-backuper from CPAN (perl packages repository) or directly from github.
## From CPAN
`cpan install App::SimpleBackuper`
## From GitHub
- `git clone https://github.com/dmitry-novozhilov/simple-backuper.git`
- `cd simple-backuper`
- `perl Makefile.pl`
- `make`
- `sudo make install`
Required libraries you can install from your distro package manager:
`apt install libcrypt-rijndael-perl libcrypt-openssl-rsa-perl libcompress-raw-lzma-perl libdigest-sha-perl libtext-glob-perl libtry-tiny-perl libnet-sftp-foreign-perl libconst-fast-perl libmime-base64-perl libjson-pp-perl`
# Configuring
You need a configuration file. By default simple-backuper trying to read ~/.simple-backuper/config, but you can use other path.
In this case you need specify --cfg option on all simple-backuper run.
This file is json with comments allowed. It can be like this:
```javascript
{
"db": "~/.simple-backuper/db", // This database file changes every new backup. ~/.simple-backuper/db - is a default value.
"compression_level": 9, // LZMA algorythm supports levels 1 to 9
"public_key": "~/.simple-backuper/key.pub", // This key using with "backup" command.
// For restore-db command you need to use private key of this public key.
// Creating new pair of keys:
// Private (for restoring): openssl genrsa -out ~/.simple-backuper/key 4096
// Public (for backuping): openssl rsa -in ~/.simple-backuper/key -pubout > ~/.simple-backuper/key.pub
// Keep the private key as your most valuable asset. Copy it to a safe place.
// It is desirable not in the backup storage, otherwise it will make it possible to use the backup data for someone other than you.
"storage": "/mnt/backups", // Use "host:path" or "user@host:path" for remote SFTP storage.
// All transfered data already encrypted.
// If you choose SFTP, make sure that this SFTP server works without a password.
// This can be configured with ~/.ssh/ config and ssh key-based authorization.
"space_limit": "100G", // Maximum of disc space on storage.
// While this limit has been reached, simple-backuper deletes the oldest and lowest priority file.
// K means kilobytes, M - megabytes, G - gygabytes, T - terabytes.
"files": { // Files globs with it's priorityes.
"~": 5,
"~/.gnupg": 50, // The higher the priority, the less likely it is to delete these files.
"~/.bash_history": 0, // Zero priority prohibits backup. Use it for exceptions.
"~/.cache": 0,
"~/.local/share/Trash": 0,
"~/.mozilla/firefox/*/Cache": 0,
"~/.thumbnails": 0,
}
}
```
# First (initial) backup
After configuring you need to try backuping to check for it works:
`simple-backuper backup --backup-name initial --verbose`
The initial backup will take a long time. It takes me more than a day.
The next backups will take much less time. Because usually only a small fraction of the files are changed.
# Scheduled backups
You can add to crontab next command:
```
0 0 * * * simple-backuper backup --backup-name `date -Idate`
```
It creates backup named as date every day.
# Logging
Simple backuper is so simple that it does not log itself. You can write logs from STDOUT & STDERR:
```
0 0 * * * simple-backuper backup --backup-name `date -Idate` 2>&1 >> simple-backuper.log
```
# Recovering
1. The first thing you need is a database file. If you have it, move to next step. Otherwise you can restore it from your backup storage:
`simple-backuper restore-db --storage YOUR_STORAGE --priv-key KEY`
YOUR_STORAGE - is your `storage` option from config. For example `my_ssh_backup_host:/path/to/backup/`.
KEY - is path to your private key!
2. Chose backup and files by exploring you storage by commands like `simple-backuper info`, `simple-backuper info /home`,..
3. Try to dry run of restoring files: `simple-backuper restore --path CHOSED_PATH --backup-name CHOSED_BACKUP --storage YOUR_STORAGE --destination TARGET_DIR`
CHOSED_PATH - is path in backup to restoring files.
CHOSED_BACKUP - is what version of your files must be restored.
YOUR_STORAGE - is your `storage` option from config. For example `my_ssh_backup_host:/path/to/backup/`.
TARGET_DIR - is dir for restored files.
4. If all ok, run restoring files with same command and `--write` argument!
bin/simple-backuper view on Meta::CPAN
# libnet-sftp-foreign-perl
sub usage {
say foreach @_;
print foreach <DATA>;
exit -1;
}
GetOptions(
\my %options,
'cfg=s', 'db=s', 'backup-name=s', 'path=s', 'storage=s', 'destination=s', 'priv-key=s', 'write', 'verbose', 'quiet'
) or usage();
my $command = shift;
$options{cfg} //= '~/.simple-backuper/config' if $command and grep {$command eq $_} qw(backup storage-check storage-fix stats);
my %state = (profile => {total => - Time::HiRes::time});
if($options{cfg}) {
$options{cfg} =~ s/^~/(getpwuid($>))[7]/e;
open(my $h, "<", $options{cfg}) or usage("Can't read config '$options{cfg}': $!");
my $config;
try {
$config = JSON::PP->new->utf8->relaxed(1)->decode(join('', <$h>));
} catch {
bin/simple-backuper view on Meta::CPAN
$mask =~ s/^~([^\/]*)/(getpwuid($1 ? getpwnam($1) : $<))[7]/e;
$mask =~ s/\/$//;
Encode::_utf8_off($mask);
$files_rules{ $mask } = $priority;
}
$options{files} = \%files_rules;
}
}
{
$options{db} ||= '~/.simple-backuper/db';
$options{db} =~ s/^~/(getpwuid($>))[7]/e;
if(! -e $options{db} and $command and grep {$command eq $_} qw(backup storage-check storage-fix stats)) {
print "Initializing new database...\t";
my $db_file = App::SimpleBackuper::RegularFile->new($options{db}, \%options);
$db_file->set_write_mode();
$db_file->data_ref( App::SimpleBackuper::DB->new()->dump() );
$db_file->compress();
$db_file->write();
print "done.\n";
}
if(-e $options{db}) {
bin/simple-backuper view on Meta::CPAN
print "decompressing...\t" if $options{verbose};
$db_file->decompress();
print "init...\t" if $options{verbose};
$state{profile}->{load_db} -= Time::HiRes::time();
$state{db} = App::SimpleBackuper::DB->new($db_file->data_ref);
$state{profile}->{load_db} += Time::HiRes::time();
print "done.\n" if $options{verbose};
}
}
if($options{storage} and grep {$command eq $_} qw(backup restore-db restore storage-check storage-fix)) {
if($options{storage} =~ /^[^:]+:/) {
$state{storage} = App::SimpleBackuper::StorageSFTP->new( $options{storage} );
}
else {
$state{storage} = App::SimpleBackuper::StorageLocal->new( $options{storage} );
}
}
if(! $command) {
usage("Please specify a command");
}
elsif($command eq 'storage-check') {
App::SimpleBackuper::StorageCheck(\%options, \%state);
}
elsif($command eq 'storage-fix') {
App::SimpleBackuper::StorageCheck(\%options, \%state, 1);
}
elsif($command eq 'backup') {
exists $options{$_} or usage("Option --$_ is required for command backup") foreach qw(cfg backup-name);
App::SimpleBackuper::StorageCheck(\%options, \%state, 1);
App::SimpleBackuper::Backup(\%options, \%state);
$state{profile}->{total} += Time::HiRes::time;
if(! $options{quiet}) {
printf "%s time spend: math - %s (crypt: %s, hash: %s, compress: %s), fs - %s, storage - %s\n",
fmt_time($state{profile}->{total}),
fmt_time($state{profile}->{math}),
fmt_time($state{profile}->{math_encrypt}),
fmt_time($state{profile}->{math_hash}),
fmt_time($state{profile}->{math_compress}),
fmt_time($state{profile}->{fs}),
fmt_time($state{profile}->{storage});
}
if($state{fails}) {
print "Some files failed to backup:\n";
while(my($error, $list) = each %{ $state{fails} }) {
print "\t$error:\n";
print "\t\t$_\n" foreach @$list;
}
}
if(! $options{quiet}) {
if($state{longest_files}) {
print "Top ".@{ $state{longest_files} }." longest files:\n";
printf "% 10s\t%s\n", fmt_time($_->{time}), $_->{path} foreach @{ $state{longest_files} };
bin/simple-backuper view on Meta::CPAN
}
}
App::SimpleBackuper::StorageCheck(\%options, \%state);
exit -1 if $state{fails};
}
elsif($command eq 'info') {
use Fcntl ':mode'; # For S_ISDIR & same
($state{db} and $state{db}->{backups} and @{ $state{db}->{backups} })
or usage("Database file is not exists or empty. May be your backups database is located in backup storage. You can restore in here with 'restore-db' command");
my $result = App::SimpleBackuper::Info(\%options, \%state);
if($result->{error}) {
if($result->{error} eq 'NOT_FOUND') {
print "Path $options{path} wasn't found in backups.\n";
} else {
print "Unknown error $result->{error}.\n";
}
exit -1;
}
if(@{ $result->{subfiles} }) {
print "Subfiles:\n";
App::SimpleBackuper::_print_table([
['name', 'oldest backup', 'newest backup'],
map {[ $_->{name}, $_->{oldest_backup}, $_->{newest_backup} ]} @{ $result->{subfiles} } ],
1,
);
print "\n";
} else {
print "No subfiles\n";
}
if(@{ $result->{versions} }) {
print "Backuped versions of this file:\n";
App::SimpleBackuper::_print_table([
['rights', 'owner', 'group', 'size', 'mtime', 'backups'],
map {[
(
S_ISDIR($_->{mode}) ? 'd' :
S_ISLNK($_->{mode}) ? 'l' :
S_ISREG($_->{mode}) ? '-' :
'?'
)
.((S_IRUSR & $_->{mode}) ? 'r' : '-')
.((S_IWUSR & $_->{mode}) ? 'w' : '-')
.((S_IXUSR & $_->{mode}) ? 'x' : '-')
bin/simple-backuper view on Meta::CPAN
.((S_IWGRP & $_->{mode}) ? 'w' : '-')
.((S_IXGRP & $_->{mode}) ? 'x' : '-')
.((S_IROTH & $_->{mode}) ? 'r' : '-')
.((S_IWOTH & $_->{mode}) ? 'w' : '-')
.((S_IXOTH & $_->{mode}) ? 'x' : '-')
,
$_->{user},
$_->{group},
$_->{size},
$_->{mtime},
join(', ', @{ $_->{backups} }),
]} @{ $result->{versions} }
],
1,
);
} else {
print "This files has no backuped versions";
print " (but it's subfiles has)" if @{ $result->{subfiles} };
print "\n";
}
}
elsif($command eq 'restore') {
($state{db}->{backups} and @{ $state{db}->{backups} })
or usage("Database file is not exists or empty. May be your backups database is located in backup storage. You can restore in here with 'restore-db' command");
$options{$_} or usage("Required option --$_ doesn't specified") foreach qw(path destination storage backup-name);
my $result = App::SimpleBackuper::Info(\%options, \%state);
if(! grep {$options{'backup-name'} eq $_} map {@{ $_->{backups} }} @{ $result->{versions} }) {
usage(qq|Backup named '$options{"backup-name"}' of path '$options{path}' was not found|);
}
App::SimpleBackuper::Restore(\%options, \%state);
}
elsif($command eq 'restore-db') {
! $state{db} or ! @{ $state{db}->{backups} }
or usage(qq|Database already exists and contains |.@{ $state{db}->{backups} }.qq| backups. |
.qq|If you want to rewrite database file with restored one, please delete current database file $options{db}|);
$options{'priv-key'} or usage("Required option --priv-key doesn't specified");
$options{storage} or usage("Required option --storage doesn't specified");
open(my $h, '<', $options{'priv-key'}) or usage(qq|Can't read private key file '$options{"priv-key"}': $!|);
$state{rsa} = Crypt::OpenSSL::RSA->new_private_key( join('', <$h>) );
close($h);
bin/simple-backuper view on Meta::CPAN
}
# TODO: ÑтатиÑтика: кол-во бекапов, кол-во бекапов без единого удалённого файла, % файлов в каждом бекапе
elsif($command eq 'stats') {
App::SimpleBackuper::_print_table([
['name', 'max files cnt', 'current files cnt', '%'],
map {[
$_->{name},
($_->{is_done} ? $_->{max_files_cnt} : '? (unfinished)'),
$_->{files_cnt},
($_->{is_done} ? int($_->{files_cnt} / $_->{max_files_cnt} * 100).'%' : ''),
]} map {$state{db}->{backups}->unpack($_)} @{ $state{db}->{backups} }
]);
}
elsif($command) {
usage("Unknown command $command");
}
__DATA__
Usage: simple-backuper <COMMAND> [OPTIONS]
COMMANDS:
backup - creates a new backup. Required options: --backup-name. Possible: --cfg.
info - prints info of files in backup. Possible options: --db, --path, --cfg
restore - restores files from backup. Required options: --path, --backup-name, --storage, --destination.
Possible options: --db, --write.
restore-db - fetch from storage & decrypt database. Required options: --storage --priv-key
storage-check - check for existents all data on storage with local database. Possible option: --cfg
storage-fix - fix local database for loosen data and remove unknown extra files from storage. Possible option: --cfg
stats - statistics about files count in backups. Possible options: --cfg, --db
OPTIONS:
--cfg %path% - path to config file (see below). (default is ~/.simple-backuper/config)
--db %path% - path to db file. (by default using config value if possible, or ~/.simple-backuper/db otherwise)
This file need for any operations. It copied to backup storage dir each creating backup time.
--backup-name %name% - name of creating, listing or restoring backup.
--path %path% - %path% of files in backup. (default: /)
--priv-key %path% - %path% to private key file for decryption.
--storage %path% - %path% to storage dir. Remote SFTP path must begins with '%host%:'.
--destination %path% - destination path to restore files.
--write - Without this option restoring use dry run.
For EXAMPLES see README.md
CONFIG file must be a json-file like this:
{
"db": "~/.simple-backuper/db",
// This database file changes every new backup. ~/.simple-backuper/db - is a default value.
"compression_level": 9,
// LZMA algorythm supports levels 1 to 9
"public_key": "~/.simple-backuper/key.pub",
// This key using with "backup" command.
// For restore-db command you need to use private key of this public key.
// Creating new pair of keys:
// Private (for restoring): openssl genrsa -out ~/.simple-backuper/key 4096
// Public (for backuping): openssl rsa -in ~/.simple-backuper/key -pubout > ~/.simple-backuper/key.pub
// Keep the private key as your most valuable asset. Copy it to a safe place.
// It is desirable not in the backup storage, otherwise it will make it possible to use the backup data for someone other than you.
"storage": "/mnt/backups",
// Use "host:path" or "user@host:path" for remote SFTP storage.
// All transfered data already encrypted.
// If you choose SFTP, make sure that this SFTP server works without a password.
// This can be configured with ~/.ssh/ config and ssh key-based authorization.
"space_limit": "100G",
// Maximum of disc space on storage.
// While this limit has been reached, simple-backuper deletes the oldest and lowest priority file.
// K means kilobytes, M - megabytes, G - gygabytes, T - terabytes.
"files": { // Files globs with it's priorityes.
"~": 5,
"~/.gnupg": 50, // The higher the priority, the less likely it is to delete these files.
"~/.bash_history": 0, // Zero priority prohibits backup. Use it for exceptions.
"~/.cache": 0,
"~/.local/share/Trash": 0,
"~/.mozilla/firefox/*/Cache": 0,
"~/.thumbnails": 0,
}
}
lib/App/SimpleBackuper.pod view on Meta::CPAN
=encoding utf8
=head1 NAME
App::SimpleBackuper - is a simple tool for backuping files and restoring it from backups.
=head1 Benefits
=over
=item * Simplicity and transparency. Few lib files and one short script. Most programmers can understand it.
=item * Efficient use of disk space (incremental backup):
=over
=item * Automatic deduplication of parts of files (most modified files differ only partially).
=item * All files will be compressed with archivator (compression level may be configured).
=item * Incremental backup format doesn't require initial data snapshot.
=back
=item * Security:
=over
=item * All files will be encrypted with AES256 + RSA4096.
=item * Encryption doing before data send to storage host.
=item * For backuping you don't need to keep private RSA key accessible to this program. It needs only for restoring.
=item * Thus, even with the backup data, no one can get the source files from them. And also no one can fake a backup.
=back
=item * You can specify different priorities for any files.
=item * For recover your backup you need only: access to storage host, your crypting keys and some backup options.
=item * You can backup to local directory or to remote sftp server.
=item * Requires on backuper host: perl and some perl libs.
=item * Requires on SFTP storage host: disk space only.
=back
=head1 Installing
You can install simple-backuper from CPAN (perl packages repository) or directly from github.
=head2 From CPAN
C<cpan install App::SimpleBackuper>
=head2 From GitHub
=over
=item * C<git clone https://github.com/dmitry-novozhilov/simple-backuper.git>
=item * C<cd simple-backuper>
=item * C<perl Makefile.pl>
=item * C<make>
=item * C<sudo make install>
Required libraries you can install from your distro package manager:
C<apt install libcrypt-rijndael-perl libcrypt-openssl-rsa-perl libcompress-raw-lzma-perl libdigest-sha-perl libtext-glob-perl libtry-tiny-perl libnet-sftp-foreign-perl libconst-fast-perl libmime-base64-perl libjson-pp-perl>
=back
=head1 Configuring
You need a configuration file. By default simple-backuper trying to read ~/.simple-backuper/config, but you can use other path.
In this case you need specify --cfg option on all simple-backuper run.
This file is json with comments allowed. It can be like this:
{
"db": "~/.simple-backuper/db",
// This database file changes every new backup. ~/.simple-backuper/db - is a default value.
"compression_level": 9,
// LZMA algorythm supports levels 1 to 9
"public_key": "~/.simple-backuper/key.pub",
// This key using with "backup" command.
// For restore-db command you need to use private key of this public key.
// Creating new pair of keys:
// Private (for restoring): openssl genrsa -out ~/.simple-backuper/key 4096
// Public (for backuping): openssl rsa -in ~/.simple-backuper/key -pubout > ~/.simple-backuper/key.pub
// Keep the private key as your most valuable asset. Copy it to a safe place.
// It is desirable not in the backup storage, otherwise it will make it possible to use the backup data for someone other than you.
"storage": "/mnt/backups",
// Use "host:path" or "user@host:path" for remote SFTP storage.
// All transfered data already encrypted.
// If you choose SFTP, make sure that this SFTP server works without a password.
// This can be configured with ~/.ssh/ config and ssh key-based authorization.
"space_limit": "100G",
// Maximum of disc space on storage.
// While this limit has been reached, simple-backuper deletes the oldest and lowest priority file.
// K means kilobytes, M - megabytes, G - gygabytes, T - terabytes.
"files": { // Files globs with it's priorityes.
"~": 5,
"~/.gnupg": 50, // The higher the priority, the less likely it is to delete these files.
"~/.bash_history": 0, // Zero priority prohibits backup. Use it for exceptions.
"~/.cache": 0,
"~/.local/share/Trash": 0,
"~/.mozilla/firefox/*/Cache": 0,
"~/.thumbnails": 0,
}
}
=head1 First (initial) backup
After configuring you need to try backuping to check for it works:
C<simple-backuper backup --backup-name initial --verbose>
The initial backup will take a long time. It takes me more than a day.
The next backups will take much less time. Because usually only a small fraction of the files are changed.
=head1 Scheduled backups
You can add to crontab next command:
C<
0 0 * * * simple-backuper backup --backup-name `date -Idate`
>
It creates backup named as date every day.
=head1 Logging
Simple backuper is so simple that it does not log itself. You can write logs from STDOUT & STDERR:
C<<
0 0 * * * simple-backuper backup --backup-name `date -Idate` 2E<gt>&1 E<gt>E<gt> simple-backuper.log
>>
=head1 Recovering
=over
=item 1. The first thing you need is a database file. If you have it, move to next step. Otherwise you can restore it from your backup storage:
C<simple-backuper restore-db --storage YOUR_STORAGE --priv-key KEY>
=over
=item
YOUR_STORAGE - is your C<storage> option from config. For example C<my_ssh_backup_host:/path/to/backup/>.
=item
KEY - is path to your private key!
=back
=item 2. Chose backup and files by exploring you storage by commands like C<simple-backuper info>, C<simple-backuper info /home>,..
=item 3. Try to dry run of restoring files: C<simple-backuper restore --path CHOSED_PATH --backup-name CHOSED_BACKUP --storage YOUR_STORAGE --destination TARGET_DIR>
=over
=item
CHOSED_PATH - is path in backup to restoring files.
=item
CHOSED_BACKUP - is what version of your files must be restored.
=item
YOUR_STORAGE - is your C<storage> option from config. For example C<my_ssh_backup_host:/path/to/backup/>.
=item
TARGET_DIR - is dir for restored files.
=back
=item 4. If all ok, run restoring files with same command and C<--write> argument!
=back
=head1 AUTHOR
Dmitriy Novozhilov <Dmitry@Novozhilov.ru>
=head1 LICENSE
L<GPL v3|https://github.com/dmitry-novozhilov/simple-backuper/blob/master/LICENSE>
lib/App/SimpleBackuper/Backup.pm view on Meta::CPAN
my $block_id = shift @$size_basket;
shift @$prio_basket if ! @$size_basket;
shift @{ $state->{blocks2delete_prio2size2chunks} } if ! @$prio_basket;
return $block_id;
}
sub Backup {
my($options, $state) = @_;
my($backups, $files, $parts, $blocks) = @{ $state->{db} }{qw(backups files parts blocks)};
die "Backup '$options->{\"backup-name\"}' already exists" if grep { $backups->unpack($_)->{name} eq $options->{'backup-name'} } @$backups;
$state->{$_} = 0 foreach qw(last_backup_id last_file_id last_block_id bytes_processed bytes_in_last_backup total_weight);
print "Preparing to backup: " if $options->{verbose};
$state->{profile}->{init_ids} = - time();
foreach (@$backups) {
my $id = $backups->unpack($_)->{id};
$state->{last_backup_id} = $id if ! $state->{last_backup_id} or $state->{last_backup_id} < $id;
}
#print "last backup id $state->{last_backup_id}, ";
foreach (@$files) {
my $file = $files->unpack($_);
$state->{last_file_id} = $file->{id} if ! $state->{last_file_id} or $state->{last_file_id} < $file->{id};
if($file->{versions} and @{ $file->{versions} } and $file->{versions}->[-1]->{backup_id_max} == $state->{last_backup_id}) {
$state->{bytes_in_last_backup} += $file->{versions}->[-1]->{size};
}
}
#print "last file id $state->{last_file_id}, ";
foreach (@$blocks) {
my $id = $blocks->unpack($_)->{id};
$state->{last_block_id} = $id if ! $state->{last_block_id} or $state->{last_block_id} < $id;
}
#print "last block id $state->{last_block_id}, ";
$state->{profile}->{init_ids} += time;
print "total weight " if $options->{verbose};
for(my $q = 0; $q <= $#$parts; $q++) {
$state->{total_weight} += $parts->unpack($parts->[ $q ])->{size};
}
print fmt_weight($state->{total_weight}).", " if $options->{verbose};
my $cur_backup = {name => $options->{'backup-name'}, id => ++$state->{last_backup_id}, files_cnt => 0, max_files_cnt => 0};
$backups->upsert({ id => $cur_backup->{id} }, $cur_backup);
{
$state->{blocks_info} = _BlocksInfo($options, $state);
$state->{blocks2delete_prio2size2chunks} = {};
foreach my $block (@$blocks) {
my $block_id = $blocks->unpack($block)->{id};
next if ! $block_id; # What's this?
my $block_info = $state->{blocks_info}->{ $block_id };
push @{ $state->{blocks2delete_prio2size2chunks}->{ $block_info->[0] }->{ $block_info->[1] } }, $block_id;
lib/App/SimpleBackuper/Backup.pm view on Meta::CPAN
my @cur_path;
foreach my $path_node (@path) {
push @cur_path, $path_node;
my $file = $files->find_by_parent_id_name($file_id, $path_node);
$file //= {
parent_id => $file_id,
id => ++$state->{last_file_id},
name => $path_node,
versions => [ {
backup_id_min => $state->{last_backup_id},
backup_id_max => 0,
uid => 0,
gid => 0,
size => 0,
mode => 0,
mtime => 0,
block_id => 0,
symlink_to => undef,
parts => [],
} ],
};
lib/App/SimpleBackuper/Backup.pm view on Meta::CPAN
if(time - $last_db_save > $SAVE_DB_PERIOD) {
App::SimpleBackuper::BackupDB($options, $state);
$last_db_save = time;
}
}
while(my($full_path, $dir2upd) = each %dirs2upd) {
print "Updating dir $full_path..." if $options->{verbose};
my $file = $files->find_by_parent_id_name($dir2upd->{parent_id}, $dir2upd->{filename});
my @stat = lstat($full_path);
if(@stat and $file->{versions}->[-1]->{backup_id_max} != $state->{last_backup_id}) {
my($uid, $gid) =_proc_uid_gid($stat[4], $stat[5], $state->{db}->{uids_gids});
if($file->{versions}->[-1]->{backup_id_max} == $state->{last_backup_id} - 1) {
$file->{versions}->[-1] = {
%{ $file->{versions}->[-1] },
backup_id_max => $state->{last_backup_id},
uid => $uid,
gid => $gid,
size => $stat[7],
mode => $stat[2],
mtime => $stat[9],
block_id => 0,
symlink_to => undef,
parts => [],
};
} else {
push @{ $file->{versions} }, {
backup_id_min => $state->{last_backup_id},
backup_id_max => $state->{last_backup_id},
uid => $uid,
gid => $gid,
size => $stat[7],
mode => $stat[2],
mtime => $stat[9],
block_id => 0,
symlink_to => undef,
parts => [],
}
}
$files->upsert({ id => $file->{id}, parent_id => $file->{parent_id} }, $file);
my $backup = $backups->find_row({ id => $state->{last_backup_id} });
$backup->{files_cnt}++;
$backup->{max_files_cnt}++;
$backups->upsert({ id => $backup->{id} }, $backup );
}
print "OK\n" if $options->{verbose};
}
my $backup = $backups->find_row({ id => $state->{last_backup_id} });
$backup->{is_done} = 1;
$backups->upsert({ id => $backup->{id} }, $backup );
App::SimpleBackuper::BackupDB($options, $state);
_print_progress($state) if ! $options->{quiet};
}
sub _print_progress {
print "Progress: ";
if($_[0]->{bytes_in_last_backup}) {
printf "processed %s of %s in last backup, ", fmt_weight($_[0]->{bytes_processed}), fmt_weight($_[0]->{bytes_in_last_backup});
}
printf "total backups weight %s.\n", fmt_weight($_[0]->{total_weight});
}
use Text::Glob qw(match_glob);
use Fcntl ':mode'; # For S_ISDIR & same
use App::SimpleBackuper::RegularFile;
sub _file_proc {
my($task, $options, $state) = @_;
confess "No task" if ! $task;
lib/App/SimpleBackuper/Backup.pm view on Meta::CPAN
$state->{profile}->{fs_lstat} += time;
if(! @stat) {
print ". Not exists\n" if $options->{verbose};
return;
}
else {
printf ", stat: %s:%s %o %s modified at %s", scalar getpwuid($stat[4]), scalar getgrgid($stat[5]), $stat[2], fmt_weight($stat[7]), fmt_datetime($stat[9]) if $options->{verbose};
}
my($backups, $blocks, $files, $parts, $uids_gids) = @{ $state->{db} }{qw(backups blocks files parts uids_gids)};
my($uid, $gid) = _proc_uid_gid($stat[4], $stat[5], $uids_gids);
my($file); {
my($filename) = $task->[0] =~ /([^\/]+)\/?$/;
$file = $files->find_by_parent_id_name($task->[2], $filename);
if($file) {
print ", is old file #$file->{id}" if $options->{verbose};
if($file->{versions}->[-1]->{backup_id_max} == $state->{last_backup_id}) {
print ", is already backuped.\n" if $options->{verbose};
return;
}
} else {
$file = {
parent_id => $task->[2],
id => ++$state->{last_file_id},
name => $filename,
versions => [],
};
print ", is new file #$file->{id}" if $options->{verbose};
}
}
$state->{bytes_processed} += $file->{versions}->[-1]->{size} if @{ $file->{versions} };
my %version = (
backup_id_min => $state->{last_backup_id},
backup_id_max => $state->{last_backup_id},
uid => $uid,
gid => $gid,
size => $stat[7],
mode => $stat[2],
mtime => $stat[9],
block_id => undef,
symlink_to => undef,
parts => [],
);
lib/App/SimpleBackuper/Backup.pm view on Meta::CPAN
$state->{profile}->{fs} += time;
$state->{profile}->{fs_read} += time;
return if ! $reg_file;
if(@{ $file->{versions} } and $file->{versions}->[-1]->{mtime} == $version{mtime}) {
$version{parts} = $file->{versions}->[-1]->{parts}; # If mtime not changed then file not changed
$version{block_id} = $file->{versions}->[-1]->{block_id};
my $block = $blocks->find_row({ id => $version{block_id} });
confess "File has lost block #$version{block_id} in backup "
.$backups->find_row({ id => $version{backup_id_min} })->{name}
."..".$backups->find_row({ id => $version{backup_id_max} })->{name}
if ! $block;
$block->{last_backup_id} = $state->{last_backup_id};
$blocks->upsert({ id => $block->{id} }, $block);
print ", mtime is not changed.\n" if $options->{verbose};
} else {
print @{ $file->{versions} } ? ", mtime changed.\n" : "\n" if $options->{verbose};
my $part_number = 0;
my %block_ids;
while(1) {
$state->{profile}->{fs} -= time;
$state->{profile}->{fs_read} -= time;
lib/App/SimpleBackuper/Backup.pm view on Meta::CPAN
# Search for part with this hash
if(my $part = $parts->find_row({ hash => $part{hash} })) {
if($part->{block_id}) {
$block_ids{ $part->{block_id} }++;
}
$part{size} = $part->{size};
$part{aes_key} = $part->{aes_key};
$part{aes_iv} = $part->{aes_iv};
print "backuped earlier (".fmt_weight($read)." -> ".fmt_weight($part->{size}).");\n" if $options->{verbose};
} else {
print fmt_weight($read) if $options->{verbose};
$state->{profile}->{math} -= time;
$state->{profile}->{math_compress} -= time;
$file_time_spent -= time;
my $ratio = $reg_file->compress();
$file_time_spent += time;
$state->{profile}->{math} += time;
lib/App/SimpleBackuper/Backup.pm view on Meta::CPAN
parts_cnt => scalar @{ $version{parts} },
};
}
foreach my $part (@{ $version{parts} }) {
$part->{block_id} //= $block->{id};
$parts->upsert({ hash => $part->{hash} }, $part);
}
$block->{last_backup_id} = $state->{last_backup_id};
$blocks->upsert({ id => $block->{id} }, $block);
$version{block_id} = $block->{id};
}
}
else {
print ", skip not supported file type\n" if $options->{verbose};
return;
}
# If file version not changed, use old version with wider backup ids range
if( @{ $file->{versions} }
and (
$file->{versions}->[-1]->{backup_id_max} + 1 == $state->{last_backup_id}
or $file->{versions}->[-1]->{backup_id_max} == $state->{last_backup_id}
)
and $file->{versions}->[-1]->{uid} == $version{uid}
and $file->{versions}->[-1]->{gid} == $version{gid}
and $file->{versions}->[-1]->{size} == $version{size}
and $file->{versions}->[-1]->{mode} == $version{mode}
and $file->{versions}->[-1]->{mtime}== $version{mtime}
and (
defined $file->{versions}->[-1]->{symlink_to} == defined $version{symlink_to}
and (
! defined $version{symlink_to}
or $file->{versions}->[-1]->{symlink_to} eq $version{symlink_to}
)
)
and join(' ', map { $_->{hash} } @{ $file->{versions}->[-1]->{parts} }) eq join(' ', map { $_->{hash} } @{ $version{parts} })
) {
$file->{versions}->[-1]->{backup_id_max} = $state->{last_backup_id};
} else {
push @{ $file->{versions} }, \%version;
}
$files->upsert({ parent_id => $file->{parent_id}, id => $file->{id} }, $file );
my $backup = $backups->find_row({ id => $state->{last_backup_id} });
$backup->{files_cnt}++;
$backup->{max_files_cnt}++;
$backups->upsert({ id => $backup->{id} }, $backup );
$state->{longest_files} ||= [];
if( @{ $state->{longest_files} } < $SIZE_OF_TOP_FILES
or $state->{longest_files}->[-1]->{time} < $file_time_spent
) {
@{ $state->{longest_files} } = sort {$b->{time} <=> $a->{time}} (@{ $state->{longest_files} }, {time => $file_time_spent, path => $task->[0]});
splice @{ $state->{longest_files} }, $SIZE_OF_TOP_FILES;
}
lib/App/SimpleBackuper/Backup.pm view on Meta::CPAN
splice @{ $state->{heaviweightest_files} }, $SIZE_OF_TOP_FILES;
}
}
return @next;
}
sub _free_up_space {
my($options, $state, $protected_block_ids) = @_;
my($backups, $files, $blocks, $parts) = @{ $state->{db} }{qw(backups files blocks parts)};
my $deleted = 0;
while(1) {
my($block_id, @files) = _get_block_to_delete($state);
last if ! $block_id;
next if exists $protected_block_ids->{ $block_id };
my $block = $blocks->find_row({ id => $block_id });
next if ! $block;
next if $block->{last_backup_id} == $state->{last_backup_id};
$deleted += App::SimpleBackuper::_BlockDelete($options, $state, $block, $state->{blocks_info}->{ $block_id }->[2]);
last if $deleted;
}
die "Nothing to delete from storage for free space" if ! $deleted;
}
1;
lib/App/SimpleBackuper/DB.pm view on Meta::CPAN
sub _unpack_record {
my($self) = @_;
my $length = $self->_unpack_tmpl("J");
return $self->_unpack_tmpl("a$length");
}
sub new {
my($class, $dump_ref) = @_;
my $self = bless {
backups => App::SimpleBackuper::DB::BackupsTable->new(),
files => App::SimpleBackuper::DB::FilesTable->new(),
parts => App::SimpleBackuper::DB::PartsTable->new(),
blocks => App::SimpleBackuper::DB::BlocksTable->new(),
uids_gids => App::SimpleBackuper::DB::UidsGidsTable->new(),
} => $class;
if($dump_ref) {
$self->{dump} = $$dump_ref;
$self->{offset} = 0;
lib/App/SimpleBackuper/DB.pm view on Meta::CPAN
sub dump {
my($self) = @_;
my $dump_method = "dump_format_v$FORMAT_VERSION";
return $self->$dump_method();
}
sub parse_format_v2 {
my($self) = @_;
my($backups_cnt, $files_cnt, $parts_cnt, $blocks_cnt, $uids_gids_cnt) = $self->_unpack_tmpl("JJJJJ");
$self->{backups} = App::SimpleBackuper::DB::BackupsTable->new($backups_cnt);
$self->{backups} ->[$_ - 1] = $self->_unpack_record() for 1 .. $backups_cnt;
$self->{files} = App::SimpleBackuper::DB::FilesTable->new($files_cnt);
$self->{files} ->[$_ - 1] = $self->_unpack_record() for 1 .. $files_cnt;
$self->{parts} = App::SimpleBackuper::DB::PartsTable->new($parts_cnt);
$self->{parts} ->[$_ - 1] = $self->_unpack_record() for 1 .. $parts_cnt;
$self->{blocks} = App::SimpleBackuper::DB::BlocksTable->new($blocks_cnt);
$self->{blocks} ->[$_ - 1] = $self->_unpack_record() for 1 .. $blocks_cnt;
$self->{uids_gids} = App::SimpleBackuper::DB::UidsGidsTable->new($uids_gids_cnt);
$self->{uids_gids} ->[$_ - 1] = $self->_unpack_record() for 1 .. $uids_gids_cnt;
}
sub dump_format_v2 {
my($self) = @_;
return \ join('',
pack("JJJJJJ", $FORMAT_VERSION, map {scalar @{ $self->{$_} }} qw(backups files parts blocks uids_gids)),
map { pack("Ja".length($_), length($_), $_) } map {@{ $self->{$_} }} qw(backups files parts blocks uids_gids)
);
}
sub parse_format_v1 {
my($self) = @_;
my($backups_cnt, $files_cnt, $uids_gids_cnt) = $self->_unpack_tmpl("JJJ");
$self->{backups} = App::SimpleBackuper::DB::BackupsTable->new($backups_cnt);
foreach(my $q = 0; $q < $backups_cnt; $q++) {
# upgrade backups format
my $record = $self->_unpack_record();
$record = $self->{backups}->unpack_format_v1($record);
$record = $self->{backups}->pack($record);
$self->{backups}->[ $q ] = $record;
}
$self->{files} = App::SimpleBackuper::DB::FilesTable->new($files_cnt);
$self->{files} ->[$_ - 1] = $self->_unpack_record() for 1 .. $files_cnt;
$self->{uids_gids} = App::SimpleBackuper::DB::UidsGidsTable->new($uids_gids_cnt);
$self->{uids_gids} ->[$_ - 1] = $self->_unpack_record() for 1 .. $uids_gids_cnt;
delete $self->{ $_ } foreach qw(dump offset);
my %backups_files_cnt = map {$self->{backups}->unpack($_)->{id} => 0} @{ $self->{backups} };
for my $q (0 .. $#{ $self->{files} }) {
my $file = $self->{files}->unpack( $self->{files}->[ $q ] );
foreach my $version (@{ $file->{versions} }) {
foreach my $backup_id ( $version->{backup_id_min} .. $version->{backup_id_max} ) {
$backups_files_cnt{ $backup_id }++;
}
foreach my $part ( @{ $version->{parts} } ) {
$self->{parts}->upsert({hash => $part->{hash}}, {%$part, block_id => $version->{block_id}});
}
my $block = $self->{blocks}->find_row({ id => $version->{block_id} });
if(! $block) {
$self->{blocks}->upsert(
{id => $version->{block_id}},
{
id => $version->{block_id},
last_backup_id => $version->{backup_id_max},
parts_cnt => scalar @{ $version->{parts} },
}
);
} else {
$block->{last_backup_id} = $version->{backup_id_max} if $block->{last_backup_id} < $version->{backup_id_max};
$block->{parts_cnt} += @{ $version->{parts} };
$self->{blocks}->upsert({ id => $version->{block_id} }, $block);
}
}
}
while(my($backup_id, $files_cnt) = each %backups_files_cnt) {
my $backup = $self->{backups}->find_row({ id => $backup_id });
$backup->{files_cnt} = $files_cnt;
$self->{backups}->upsert({ id => $backup_id }, $backup );
}
}
sub dump_format_v1 {
my $self = shift;
return \ join('',
pack("JJJJ", $FORMAT_VERSION, scalar @{ $self->{backups} }, scalar @{ $self->{files} }, scalar @{ $self->{uids_gids} }),
map { pack("Ja".length($_), length($_), $_) } @{ $self->{backups} }, @{ $self->{files} }, @{ $self->{uids_gids} }
);
}
1;
lib/App/SimpleBackuper/DB/BlocksTable.pm view on Meta::CPAN
use warnings;
use feature ':5.14';
use parent qw(App::SimpleBackuper::DB::BaseTable);
sub pack {
my($self, $data) = @_;
my $p = $self->packer();
$p->pack(J => 1 => $data->{id});
if(exists $data->{last_backup_id}) {
$p->pack(J => 1 => $data->{last_backup_id});
if(exists $data->{parts_cnt}) {
$p->pack(J => 1 => $data->{parts_cnt});
}
}
return $p->data;
}
sub unpack {
my($self, $data) = @_;
my $p = $self->packer($data);
return {
id => $p->unpack(J => 1),
last_backup_id => $p->unpack(J => 1),
parts_cnt => $p->unpack(J => 1),
};
}
1;
lib/App/SimpleBackuper/DB/FilesTable.pm view on Meta::CPAN
use warnings;
use parent qw(App::SimpleBackuper::DB::BaseTable);
use Try::Tiny;
use Data::Dumper;
use App::SimpleBackuper::DB::PartsTable;
sub _pack_version {
my($version) = @_;
my $p = __PACKAGE__->packer()
->pack(J => 1 => $version->{backup_id_min})
->pack(J => 1 => $version->{backup_id_max})
->pack(J => 1 => $version->{uid})
->pack(J => 1 => $version->{gid})
->pack(J => 1 => $version->{size})
->pack(J => 1 => $version->{mode})
->pack(J => 1 => $version->{mtime})
->pack(J => 1 => $version->{block_id})
->pack(J => 1 => length($version->{symlink_to} // ''))
;
$p->pack(a => length($version->{symlink_to}) => $version->{symlink_to} // '') if $version->{symlink_to};
lib/App/SimpleBackuper/DB/FilesTable.pm view on Meta::CPAN
return $p->data;
}
sub _unpack_version {
my($version) = @_;
my $p = __PACKAGE__->packer($version);
my %version = (
backup_id_min => $p->unpack(J => 1),
backup_id_max => $p->unpack(J => 1),
uid => $p->unpack(J => 1),
gid => $p->unpack(J => 1),
size => $p->unpack(J => 1),
mode => $p->unpack(J => 1),
mtime => $p->unpack(J => 1),
block_id => $p->unpack(J => 1),
parts => [],
);
my $symlink_to_length = $p->unpack(J => 1);
lib/App/SimpleBackuper/Info.pm view on Meta::CPAN
use strict;
use warnings;
use Time::HiRes qw(time);
use App::SimpleBackuper::_print_table;
use App::SimpleBackuper::_format;
sub Info {
my($options, $state) = @_;
my($backups, $files, $uids_gids) = @{ $state->{db} }{qw(backups files uids_gids)};
my $parent_file;
my @path = split(/\//, $options->{path} // '/', -1);
pop @path if @path and $path[-1] eq '';
my $file_id = 0;
$state->{profile}->{walk2path} -= time;
foreach my $path_node (@path) {
my $file = $files->find_by_parent_id_name($file_id, $path_node);
return {error => 'NOT_FOUND'} if ! $file;
$file_id = $file->{id};
$parent_file = $file;
}
$state->{profile}->{walk2path} += time;
my @versions;
foreach my $version (@{ $parent_file->{versions} }) {
my @backups = map {$backups->find_row({id => $_})} $version->{backup_id_min} .. $version->{backup_id_max};
@backups = map {$_->{name}} @backups;
my $user = $uids_gids->find_row({id => $version->{uid}});
$user = $user->{name};
my $group = $uids_gids->find_row({id => $version->{gid}});
$group = $group->{name};
push @versions, {
backups => \@backups,
user => $user,
group => $group,
size => fmt_weight($version->{size}),
mode => $version->{mode},
mtime => fmt_datetime($version->{mtime}),
};
}
my @files = map {@$_} $files->find_all({ parent_id => $parent_file->{id} });
@files = sort {$a->{name} cmp $b->{name}} @files;
my @subfiles;
foreach my $file (@files) {
my $oldest_backup = $backups->find_row({id => $file->{versions}->[-1]->{backup_id_max} });
$oldest_backup = $oldest_backup->{name};
my $newest_backup = $backups->find_row({id => $file->{versions}->[0]->{backup_id_max} });
$newest_backup = $newest_backup->{name};
push @subfiles, {
name => ($file->{name} eq '' ? '/' : $file->{name}),
oldest_backup => $oldest_backup // '-',
newest_backup => $newest_backup // '-',
};
}
return {versions => \@versions, subfiles => \@subfiles};
}
1;
lib/App/SimpleBackuper/Restore.pm view on Meta::CPAN
package App::SimpleBackuper;
use strict;
use warnings;
use Fcntl ':mode'; # For S_ISDIR & same
use App::SimpleBackuper::_format;
sub Restore {
my($options, $state) = @_;
my($backup) = grep {$_->{name} eq $options->{'backup-name'}}
map {$state->{db}->{backups}->unpack($_)}
@{ $state->{db}->{backups} }
;
die qq|Backup $options->{'backup-name'} was not found in database| if ! $backup;
$state->{backup_id} = $backup->{id};
my @path = split(/\//, $options->{path}, -1);
pop @path if @path and $path[-1] eq '';
my $file = {id => 0};
my @cur_path;
foreach my $path_node (@path) {
push @cur_path, $path_node;
$file = $state->{db}->{files}->find_by_parent_id_name($file->{id}, $path_node);
return {error => 'NOT_FOUND'} if ! $file;
}
$options->{destination} =~ s/\/$//g;
_proc_file($options, $state, $file, join('/', @cur_path) || '/', $options->{destination});
print "Backup '$options->{'backup-name'}' was successful restored.\n" if ! $options->{quiet};
return {};
}
sub _restore_part {
my($options, $reg_file, $storage, $part, $number) = @_;
$reg_file->data_ref(\$storage->get(fmt_hex2base64($part->{hash}))->[0]);
print "fetched, " if $options->{verbose};
$reg_file->decrypt($part->{aes_key}, $part->{aes_iv});
print "decrypted, " if $options->{verbose};
my $ratio = $reg_file->decompress();
printf "decompressed (x%d)", $ratio if $options->{verbose};
$reg_file->write($number);
print " and restored.\n" if $options->{verbose};
}
sub _proc_file {
my($options, $state, $file, $backup_path, $fs_path) = @_;
print "$backup_path -> $fs_path\n" if $options->{verbose};
my($version) = grep {$_->{backup_id_min} <= $state->{backup_id} and $_->{backup_id_max} >= $state->{backup_id}}
@{ $file->{versions} };
if(! $version) {
print "\tnot exists in this backup.\n" if $options->{verbose};
return;
}
my @stat = lstat($fs_path);
my($fs_user, $fs_group);
if(@stat) {
$fs_user = getpwuid($stat[4]);
$fs_group = getpwuid($stat[5]);
}
if(S_ISDIR $version->{mode}) {
my $need2mkdir;
if(@stat) {
if(! S_ISDIR $stat[2]) {
print "\tin backup it's dir but on FS it's not.\n" if $options->{verbose};
unlink $fs_path if $options->{write};
$need2mkdir = 1;
}
} else {
$need2mkdir = 1;
}
if($need2mkdir) {
mkdir($fs_path, $version->{mode}) or die "Can't mkdir $fs_path: $!" if $options->{write};
$fs_user = scalar getpwuid $<;
$fs_group = scalar getgrgid $(;
$stat[2] = $version->{mode};
}
}
elsif(S_ISLNK $version->{mode}) {
my $need2link;
if(@stat) {
if(S_ISLNK $stat[2]) {
my $symlink_to = readlink($fs_path) // die "Can't read symlink $fs_path: $!";
if($symlink_to ne $version->{symlink_to}) {
print "\tin backup this symlink refers to $version->{symlink_to} but on FS - to $symlink_to.\n" if $options->{verbose};
unlink $fs_path if $options->{write};
$need2link = 1;
}
} else {
print "\tin backup it's a symlink but on FS it's not.\n" if $options->{verbose};
unlink $fs_path if $options->{write};
$need2link = 1;
}
} else {
$need2link = 1;
}
if($need2link) {
if($options->{write}) {
symlink($version->{symlink_to}, $fs_path) or die "Can't make symlink $fs_path -> $version->{symlink_to}: $!";
lib/App/SimpleBackuper/Restore.pm view on Meta::CPAN
$fs_group = scalar getgrgid $(;
}
}
elsif(S_ISREG $version->{mode}) {
my $need2rewrite_whole_file;
if(@stat) {
if(S_ISREG $stat[2]) {
my $reg_file = App::SimpleBackuper::RegularFile->new($fs_path, $options);
my $file_writer;
if($stat[7] != $version->{size}) {
print "\tin backup it's file with size ".fmt_weight($version->{size}).", but on FS - ".fmt_weight($version->{size}).".\n" if $options->{verbose};
$reg_file->truncate($version->{size}) if $options->{write};
}
for my $part_number (0 .. $#{ $version->{parts} }) {
$reg_file->read($part_number);
my $fs_hash = $reg_file->hash;
if($fs_hash eq $version->{parts}->[ $part_number ]->{hash}) {
print "\tpart #$part_number hash is ".fmt_hex2base64($fs_hash)." (OK)\n" if $options->{verbose};
}
else {
print "\tpart#$part_number in backup has hash ".fmt_hex2base64($version->{parts}->[ $part_number ]->{hash}).", but on FS - ".fmt_hex2base64($fs_hash).": " if $options->{verbose};
if($options->{write}) {
_restore_part($options, $reg_file, $state->{storage}, $version->{parts}->[ $part_number ], $part_number);
} else {
print "\twill be restored\n" if $options->{verbose};
}
}
}
} else {
print "\tin backup it's a regular file, but on FS it's not.\n" if $options->{verbose};
$need2rewrite_whole_file = 1;
unlink $fs_path if $options->{write};
}
} else {
$need2rewrite_whole_file = 1;
}
if($need2rewrite_whole_file) {
if($options->{write}) {
my $reg_file = App::SimpleBackuper::RegularFile->new($fs_path, $options, $state);
lib/App/SimpleBackuper/Restore.pm view on Meta::CPAN
} else {
print "\tfile will be restored.\n" if $options->{verbose};
}
$fs_user = scalar getpwuid $<;
$fs_group = scalar getgrgid $(;
}
}
if((! @stat or $stat[2] != $version->{mode}) and ! S_ISLNK $version->{mode}) {
printf "\tin backup it has mode %o but on FS - %o.\n", $version->{mode}, $stat[2] // 0 if $options->{verbose};
if($options->{write}) {
chmod($version->{mode}, $fs_path) or die sprintf("Can't chmod %s to %o: %s", $fs_path, $version->{mode}, $!);
}
}
my($db_user) = map {$_->{name}}
grep {$_->{id} == $version->{uid}}
map { $state->{db}->{uids_gids}->unpack($_) }
@{ $state->{db}->{uids_gids} }
;
my($db_group) = map {$_->{name}}
grep {$_->{id} == $version->{gid}}
map { $state->{db}->{uids_gids}->unpack($_) }
@{ $state->{db}->{uids_gids} }
;
if(($fs_user ne $db_user or $fs_group ne $db_group) and ! S_ISLNK $version->{mode}) {
print "\tin backup it owned by $db_user:$db_group but on FS - by $fs_user:$fs_group.\n" if $options->{verbose};
chown scalar(getpwnam $db_user), scalar getgrnam($db_group), $fs_path if $options->{write};
}
if(S_ISDIR $version->{mode}) {
foreach my $subfile (sort {$a->{name} cmp $b->{name}} map {@$_} $state->{db}->{files}->find_all({parent_id => $file->{id}})) {
_proc_file($options, $state, $subfile, $backup_path.'/'.$subfile->{name}, $fs_path.'/'.$subfile->{name});
}
}
}
1;
lib/App/SimpleBackuper/StorageCheck.pm view on Meta::CPAN
sub StorageCheck {
my($options, $state, $fix) = @_;
my %fails;
print "Fetching files listing from storage...\t" if $options->{verbose};
my $listing = $state->{storage}->listing();
print keys(%$listing)." files done\n" if $options->{verbose};
if(@{ $state->{db}->{backups} } and (! exists $listing->{db} or ! exists $listing->{'db.key'})) {
if($fix) {
App::SimpleBackuper::BackupDB($options, $state);
} else {
push @{ $fails{'Storage lost file'} }, grep {! exists $listing->{ $_ }} qw(db db.key);
}
}
delete @$listing{qw(db db.key)};
my %blocks2delete;
lib/App/SimpleBackuper/StorageCheck.pm view on Meta::CPAN
$fails{'Storage has unknown extra file'} = [ keys %$listing ];
}
}
if(%fails) {
print "Storage data was corrupted:\n";
while(my($error, $list) = each %fails) {
print "\t$error (".@$list."):\n";
print "\t\t$_\n" foreach @$list;
}
print "Please run `simple-backuper storage-fix` to sync database about storage state.\n";
print "But the lost data will remain lost.\n";
exit -2;
} else {
print "Storage checking done.\n" if $options->{verbose};
}
}
1;
lib/App/SimpleBackuper/_BlockDelete.pm view on Meta::CPAN
use strict;
use warnings;
use App::SimpleBackuper::_format;
sub _BlockDelete {
my($options, $state, $block, $block_files) = @_;
if($options->{verbose}) {
my $block_info = $state->{blocks_info}->{ $block->{id} };
print "\tDeleting block # $block->{id} (score $block_info->[0] = backup_id_score $block_info->[3] * prio $block_info->[4], size => ".fmt_weight($block_info->[1])."):\n";
}
my($backups, $files, $blocks, $parts) = @{ $state->{db} }{qw(backups files blocks parts)};
my %parts2delete;
# Delete all from block
$state->{profile}->{db_delete_all_from_block} -= time;
while(@$block_files) {
my $parent_id = shift @$block_files;
my $id = shift @$block_files;
my $full_path = shift @$block_files;
lib/App/SimpleBackuper/_BlockDelete.pm view on Meta::CPAN
my $found;
foreach my $version ( @{ $file->{versions} } ) {
next if $version->{block_id} != $block->{id};
$found = 1;
if($options->{verbose}) {
print "\t\tDeleting $full_path from ".
(
$version->{backup_id_min} == $version->{backup_id_max}
? "backup ".$backups->find_row({ id => $version->{backup_id_max} })->{name}
: "backups ".$backups->find_row({ id => $version->{backup_id_min} })->{name}
."..".$backups->find_row({ id => $version->{backup_id_max} })->{name}
)."\n";
}
foreach my $part ( @{ $version->{parts} } ) {
$parts2delete{ $part->{hash} } = $part;
$state->{deletions_stats}->{bytes} += $part->{size};
}
$state->{deletions_stats}->{versions}++;
foreach my $backup_id ( $version->{backup_id_min} .. $version->{backup_id_max} ) {
my $backup = $backups->find_row({ id => $backup_id });
next if ! $backup;
$backup->{files_cnt}--;
if( $backup->{files_cnt} ) {
$backups->upsert({ id => $backup_id }, $backup);
} else {
$backups->delete({ id => $backup_id });
}
}
}
if($found) {
$state->{deletions_stats}->{files}++;
# Delete version
@{ $file->{versions} } = grep {$_->{block_id} != $block->{id}} @{ $file->{versions} };
lib/App/SimpleBackuper/_BlocksInfo.pm view on Meta::CPAN
sub _BlocksInfo($$;$$$$);
sub _BlocksInfo($$;$$$$) {
my($options, $state, $block_info, $parent_id, $path, $priority) = @_;
$block_info //= {};
$parent_id //= 0;
$path //= '/';
$priority //= 0;
my($oldest_backup_id) = $state->{db}->{backups}->unpack($state->{db}->{backups}->[0])->{id};
my $subfiles = $state->{db}->{files}->find_all({parent_id => $parent_id});
foreach my $file ( @$subfiles ) {
my $full_path = ($path eq '/' ? $path : "$path/").$file->{name};
my $prio = $priority;
while(my($mask, $p) = each %{ $options->{files} }) {
$prio = $p if match_glob( $mask, $full_path );
}
_BlocksInfo($options, $state, $block_info, $file->{id}, $full_path, $prio);
my %file_added2block;
foreach my $version ( @{ $file->{versions} } ) {
next if ! $version->{block_id};
my $backup_id_score = $version->{backup_id_max} - $oldest_backup_id + 1;
$block_info->{ $version->{block_id} } ||= [0, 0, [], 0, 0];
if($block_info->{ $version->{block_id} }->[0] < $backup_id_score * $prio) {
$block_info->{ $version->{block_id} }->[0] = $backup_id_score * $prio;
$block_info->{ $version->{block_id} }->[3] = $backup_id_score;
$block_info->{ $version->{block_id} }->[4] = $prio;
}
foreach my $part (@{ $version->{parts} }) {
$block_info->{ $version->{block_id} }->[1] += $part->{size};
}
if(! $file_added2block{ $version->{block_id} }) {
push @{ $block_info->{ $version->{block_id} }->[2] }, $file->{parent_id}, $file->{id}, $full_path;
}
$file_added2block{ $version->{block_id} } = 1;
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Compress/Raw/Lzma/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/Compress/Raw/Lzma.pm
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Compress/Raw/Lzma/Lzma.so
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Compress/Raw/Lzma/autosplit.ix
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Const/Fast/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/Const/Fast.pm
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/Guess/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/Crypt/OpenSSL/Guess.pm
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/RSA/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/Crypt/OpenSSL/RSA.pm
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/RSA/RSA.so
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/RSA/autosplit.ix
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/RSA/get_key_parameters.al
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/RSA/import_random_seed.al
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/RSA/new_key_from_parameters.al
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/RSA/new_public_key.al
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/Random/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/Crypt/OpenSSL/Random.pm
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/OpenSSL/Random/Random.so
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/Rijndael/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/Crypt/Rijndael.pm
/home/me/dev/simple-backuper/local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Crypt/Rijndael/Rijndael.so
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Data/Dump/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/perl/local/lib/perl5/Data/Dump.pm
/home/me/dev/simple-backuper/perl/local/lib/perl5/Data/Dump/FilterContext.pm
/home/me/dev/simple-backuper/perl/local/lib/perl5/Data/Dump/Filtered.pm
/home/me/dev/simple-backuper/perl/local/lib/perl5/Data/Dump/Trace.pm
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Devel/GlobalPhase/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/perl/local/lib/perl5/Devel/GlobalPhase.pm
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Dist/CheckConflicts/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/perl/local/lib/perl5/Dist/CheckConflicts.pm
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/ExtUtils/Config/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/ExtUtils/Config.pm
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/ExtUtils/Helpers/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/ExtUtils/Helpers.pm
/home/me/dev/simple-backuper/local/lib/perl5/ExtUtils/Helpers/Unix.pm
/home/me/dev/simple-backuper/local/lib/perl5/ExtUtils/Helpers/VMS.pm
/home/me/dev/simple-backuper/local/lib/perl5/ExtUtils/Helpers/Windows.pm
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/ExtUtils/InstallPaths/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/lib/perl5/ExtUtils/InstallPaths.pm
local/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Module/Build/.packlist view on Meta::CPAN
/home/me/dev/simple-backuper/local/bin/config_data
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/API.pod
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Authoring.pod
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Base.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Bundling.pod
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Compat.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Config.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/ConfigData.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Cookbook.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Dumper.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Notes.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/PPMMaker.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/Default.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/MacOS.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/Unix.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/VMS.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/VOS.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/Windows.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/aix.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/cygwin.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/darwin.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/Platform/os2.pm
/home/me/dev/simple-backuper/local/lib/perl5/Module/Build/PodParser.pm
( run in 1.664 second using v1.01-cache-2.11-cpan-49f99fa48dc )