App-MtAws
view release on metacpan or search on metacpan
### 2013-07-10 v0.972 beta
* FSM potential bug fixed, can lead to possible instability
* FSM bug fixed - with --concurrency=1 it was not deleting old archives after upload replacements
* Enhancement: --detect now accepts new value "always-positive"
### 2013-07-09 v0.971 beta
* Enhancement: sync command now can synchronize modified files or propogate deletions (backward compatibility
with previous command line API preserved) - see documentation for sync command (--new, --replace-modified,
--delete-removed, --detect options)
* Compatibility: journal format modified. only backward compatibility preserved, no forward compatibility (i.e. if you
use this version once and modify journal, you can't downgrade)
* check-local-hash output minor changes
* Now compatible with ancient versions of LWP::UserAgent (5.803+ ) in HTTP mode, such versions shipped with
CentOS 5.x (or maybe even 6.x). Note that that does not mean that ancient versions of LWP are stable ;)
* Some effort to manage mtglacier version information, print version each run. Report version in HTTP
UserAgent string.
* Show some progress when reading large local directory listings
* Retry with throttling 408/500 HTTP requests. 408 can mean timeout/bad bandwidth. 500 means Amazon not ready yet.
* Some old unused code removed
* Option --max-number-of-files now works with "sync" and limits number of files to synchronize
* More verbose output
* For certain commands like Sync read only not-yet-synced filenames into memory (saves memory when working
with huge filesets)
* Use Journal module to handle journals and file listings + unit test the module
### 2012-08-28: v0.7 beta
* Amazon Glacier Multipart upload
* Multi-segment download (using HTTP Range header)
* Multithreaded upload/download
* Multipart+Multithreaded download/upload
* Multithreaded archive retrieval, deletion and download
* TreeHash validation while downloading
* Tracking of all uploaded files with a local journal file (opened for write in append mode only)
* Checking integrity of local files using journal
* Ability to limit number of archives to retrieve
* File selection options for all commands (using flexible rules with wildcard support)
* Full synchronization to Amazon Glacier - new file uploaded, modified files can be replaced, deletions can be propogated
* File name and modification times are stored as Glacier metadata ([metadata format for developers][mt-aws-glacier Amazon Glacier meta-data format specification])
* Ability to re-create journal file from Amazon Glacier metadata
* Full UTF-8 support (and full single-byte encoding support for *BSD systems)
* Multipart/multithreaded upload from STDIN
* User selectable HTTPS support. Currently defaults to plaintext HTTP
* Vault creation and deletion
* STS/IAM security tokens support
[mt-aws-glacier Amazon Glacier meta-data format specification]:https://github.com/vsespb/mt-aws-glacier/blob/master/lib/App/MtAws/MetaData.pm
Journal is needed to restore backup, and we can expect that if you need to restore a backup, that means that you lost your filesystem, together with Journal.
However Journal also needed to perform *new backups* (`sync` command), to determine which files are already in Glacier and which are not. And also to checking local file integrity (`check-local-hash` command).
Actually, usually you perform new backups every day. And you restore backups (and loose your filesystem) very rare.
So fast (local) journal is essential to perform new backups fast and cheap (important for users who backups thousands or millions of files).
And if you lost your journal, you can restore it from Amazon Glacier (see `retrieve-inventory` command). Also it's recommended to backup your journal
to another backup system (Amazon S3 ? Dropbox ?) with another tool, because retrieving inventory from Amazon Glacier is pretty slow.
Also some users might want to backup *same* files from *multiple* different locations. They will need *synchronization* solution for journal files.
Anyway I think problem of putting Journals into cloud can be automated and solved with 3 lines bash script..
#### How to maintain a relation between my journal files and my vaults?
1. You can name journal with same name as your vault. Example: Vault name is `Photos`. Journal file name is `Photos.journal`. Or `eu-west-1-Photos.journal`
2. (Almost) Any command line option can be used in config file, so you can create `myphotos.cfg` with following content:
key=YOURKEY
+ **How PATTERNS work**
+ 1) If the pattern starts with a '/' then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the final
component of the filename.
`/tmp/myfile` - matches only `/tmp/myfile`. But `tmp/myfile` - matches `/tmp/myfile` and `/home/john/tmp/myfile`
+ 2) If the pattern ends with a '/' then it will only match a directory and all files/subdirectories inside this directory. It won't match regular file.
Note that if directory is empty, it won't be synchronized to Amazon Glacier, as it does not support directories
`log/` - matches only directory `log`, but not a file `log`
+ 3) If pattern does not end with a '/', it won't match directory (directories are not supported by Amazon Glacier, so it makes no sense to match a directory
without subdirectories). However if, in future versions, we find a way to store empty directories in Amazon Glacier, this behavior may change.
`log` - matches only file `log`, but not a directory `log` nor files inside it
+ 4) if the pattern contains a '/' (not counting a trailing '/') then it is matched against the full pathname, including any leading directories.
Otherwise it is matched only against the final component of the filename.
lib/App/MtAws/Filter.pm view on Meta::CPAN
Note: You can use spaces in PATTERNs here
(III)
PATTERN:
1) if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the final
component of the filename.
2) if the pattern ends with a / then it will only match a directory and all files/subdirectories inside this directory. It won't match regular file.
Note that if directory is empty, it won't be synchronized to Amazon Glacier, as it does not support directories
3) if pattern does not end with a '/', it won't match directory (directories are not supported by Amazon Glacier, so it has no sense to match a directory
without subdirectories). However if, in future versions we find a way to store empty directories in glacier, this behaviour could change.
4) Wildcard '*' matches any path component, but it stops at slashes.
5) Wildcard '**' matches anything, including slashes.
6) When wildcard '**' meant to be a separated path component (i.e. surrounded with slashes/beginning of line/end of line), it matches 0 or more subdirectories
7) Wildcard '?' matches any character except a slash (/).
8) if the pattern contains a / (not counting a trailing /) then it is matched against the full pathname, including any leading directories.
Otherwise it is matched only against the final component of the filename.
9) if PATTERN is empty, it matches anything.
10) If PATTERN is started with '!' it only match when rest of pattern (i.e. without '!') does not match.
( run in 0.817 second using v1.01-cache-2.11-cpan-0d8aa00de5b )