App-MtAws
view release on metacpan or search on metacpan
return 8; # default 8;
}
my $build = Module::Build->new(
module_name => 'App::MtAws',
author => 'Victor Efimov <vs@vs-dev.com>',
dist_author => 'Victor Efimov <vs@vs-dev.com>',
dynamic_config => 1,
recursive_test_files=>1,
dist_abstract=>'mt-aws/glacier - Perl Multithreaded Multipart sync to Amazon Glacier',
license =>'gpl3',
# this is a special version of 'mtglacier' script with different shebang and without FindBin
# http://blogs.perl.org/users/randy_stauner/2011/09/shebangs-with-perlbrew-aliases-and-eumm-and-without-locallib.html
script_files => ['bin/mtglacier'], # script_files directive since M::B 0.18
meta_merge => {
resources => {
repository => 'https://github.com/vsespb/mt-aws-glacier',
bugtracker => 'https://github.com/vsespb/mt-aws-glacier/issues',
However if your OS/filesystem does not work with such dates, anything except correct date in journal file
is not guaranteed.
Before this fix, such dates could result in lost of filenames and modification time in journal (filename replaced
with random token) when restoring inventory (you are affected if you uploaded file on 64bit system with date
after Y2038, but then restored on 32bit system).
* Documentation: Also, note about Y2038 added to "Limitationss" section.
* CSV inventory parsing - making it 30% slower, but more consistent with what Amazon documented about its format
https://forums.aws.amazon.com/thread.jspa?threadID=141807&tstart=0
* Cosmetic changes to docs
* CPAN install - on some systems like ARM, some NAS, 32bit OSes decrease number of concurrent tests during install.
Might help preventing out-of-memory problems (but makes test slower).
### 2014-01-14 v1.112
* PPA package for Ubuntu 14.04 added to Ubuntu PPA (+ fixes in metadata of Debian/Ubuntu packages)
* Workaround: 31 December 2013 Amazon introduced extension to inventory retrieval API: now one can request for just
a part of inventory. This, however, can break mt-aws-glacier behaviour in rare circumstances i.e. when you use 3rd
party app to request a part of inventory and then run mt-aws-glacier to get full inventory, mt-aws-glacier can
download partial inventory instead of full. (details here https://forums.aws.amazon.com/thread.jspa?threadID=143107).
Releasing workaround now - mt-aws-glacier now tried to check if this is a full inventory (it's still possible for the
bug to appear very-very rare circumstances i.e. if 3rd party app will request for part inventory with limit set and
then request for continuation without any limits).
Also, now all inventory jobs raised by mt-aws-glacier have special marker. In the future versions bug will be fully
fixed as all non-mtglacier jobs will be disabled.
* Fixed several brittle tests introduced in v1.110, preventing mtglacier from install via CPAN on some systems:
- systems with perl 5.18.1, 5.18.2 and stock version of Digest::SHA
Currently upload-file behaviour changed (see above in ChangeLog) so mtglacier not affected. In previous versions this
would result in wrong relative filenames in journal and Amazon glacier metadata
(precisely, those filenames are without path prefix, as if they would be in current directory, otherwise filename
part is correct).
* Workaround: Digest::SHA perl module prior to version 5.62 calculates SHA256 wrong on 32bit machines, when data
size is more than 2^29 bytes. Now mtglacier throws an error if --partsize >= 512Mb and machine is 32bit and digest-sha
version is below 5.62. Commands which don't use --partsize are unaffected.
* Fixed: Amazon CSV format parsing: Amazon escapes doublequote with backslash but.. does not escape backslash itself.
https://forums.aws.amazon.com/thread.jspa?threadID=141807
This format is undocumented and broken by design. Fixing parser now to parse this.
this bug was not affecting any real use of mtglacier as mtglacier does not use backslashes in metadata and ignores
foreign metadata.
* Cosmetic changes to process manager code
### 2013-12-14 v1.103
* Fixed: issue #48 download-inventory was crashing if there was a request for inventory retrieval in CSV format
issued by 3rd party application. mt-aws-glacier was not supporting CSV and thus crashing.
{
"abstract" : "mt-aws/glacier - Perl Multithreaded Multipart sync to Amazon Glacier",
"author" : [
"Victor Efimov <vs@vs-dev.com>"
],
"dynamic_config" : 1,
"generated_by" : "Module::Build version 0.4205",
"license" : [
"gpl_3"
],
"meta-spec" : {
"url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec",
---
abstract: 'mt-aws/glacier - Perl Multithreaded Multipart sync to Amazon Glacier'
author:
- 'Victor Efimov <vs@vs-dev.com>'
build_requires:
B: 0
Data::Dumper: 0
DateTime: 0
HTTP::Daemon: 1.36
Module::Build: 0.36
Scalar::Util: 1.11
TAP::Harness: 0
mt-aws-glacier
==============
Perl Multithreaded multipart sync to Amazon Glacier service.
## Intro
Amazon Glacier is an archive/backup service with very low storage price. However with some caveats in usage and archive retrieval prices.
[Read more about Amazon Glacier][amazon glacier]
*mt-aws-glacier* is a client application for Amazon Glacier, written in Perl programming language, for *nix.
[amazon glacier]:http://aws.amazon.com/glacier/
* [Minimum Amazon Glacier permissions](#minimum-amazon-glacier-permissions)
## Features
* Does not use any existing Amazon Glacier library, so can be flexible in implementing advanced features
* Amazon Glacier Multipart upload
* Multi-segment download (using HTTP Range header)
* Multithreaded upload/download
* Multipart+Multithreaded download/upload
* Multithreaded archive retrieval, deletion and download
* TreeHash validation while downloading
* Tracking of all uploaded files with a local journal file (opened for write in append mode only)
* Checking integrity of local files using journal
* Ability to limit number of archives to retrieve
* File selection options for all commands (using flexible rules with wildcard support)
* Full synchronization to Amazon Glacier - new file uploaded, modified files can be replaced, deletions can be propogated
* File name and modification times are stored as Glacier metadata ([metadata format for developers][mt-aws-glacier Amazon Glacier meta-data format specification])
* Ability to re-create journal file from Amazon Glacier metadata
* Full UTF-8 support (and full single-byte encoding support for *BSD systems)
* Multipart/multithreaded upload from STDIN
* User selectable HTTPS support. Currently defaults to plaintext HTTP
* Vault creation and deletion
* STS/IAM security tokens support
[mt-aws-glacier Amazon Glacier meta-data format specification]:https://github.com/vsespb/mt-aws-glacier/blob/master/lib/App/MtAws/MetaData.pm
## Important bugs/missing features
* Only multipart upload implemented, no plain upload
* Mac OS X filesystem treated as case-sensitive
lib/App/MtAws.pm view on Meta::CPAN
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
=head1 NAME
mt-aws-glacier - Perl Multithreaded Multipart sync to Amazon Glacier
=head1 SYNOPSIS
More info in README.md or L<https://github.com/vsespb/mt-aws-glacier> or L<http://mt-aws.com/>
=cut
package App::MtAws;
lib/App/MtAws/Glacier/Inventory/CSV.pm view on Meta::CPAN
bless $self, $class;
$self;
}
sub _parse
{
my ($self) = @_;
# Text::CSV with below options does not seem to work for our case
# ( { binary => 1 , allow_whitespace => 1, quote_char => '"', allow_loose_quotes => 1, escape_char => "\\", auto_diag=>1} )
# because Amazon CSV is buggy https://forums.aws.amazon.com/thread.jspa?threadID=141807&tstart=0
my $re = undef;
my @fields;
my @records;
while (${$self->{rawdata}} =~ /^(.*?)\r?$/gsm) {
my $line = $1;
if(!defined $re) {
@fields = split /,/, $line;
for (@fields) {
s/^\"//;
( run in 0.415 second using v1.01-cache-2.11-cpan-3cd7ad12f66 )