App-DistSync
view release on metacpan or search on metacpan
{
"abstract" : "Utility for synchronizing distribution mirrors",
"author" : [
"Serz Minus (Sergey Lepenkov) <abalama@cpan.org>"
],
"dynamic_config" : 1,
"generated_by" : "ExtUtils::MakeMaker version 7.70, CPAN::Meta::Converter version 2.150010",
"license" : [
"perl_5"
],
"meta-spec" : {
"url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec",
---
abstract: 'Utility for synchronizing distribution mirrors'
author:
- 'Serz Minus (Sergey Lepenkov) <abalama@cpan.org>'
build_requires:
ExtUtils::MakeMaker: '6.6'
Test::More: '0.94'
configure_requires:
ExtUtils::MakeMaker: '0'
dynamic_config: 1
generated_by: 'ExtUtils::MakeMaker version 7.70, CPAN::Meta::Converter version 2.150010'
license: perl
[//]: # ( README.md Fri 05 Dec 2025 16:41:07 MSK )
# App::DistSync
**App::DistSync** is a ready-to-use solution for synchronizing two or more web resources containing static data. The project has proven itself in scenarios involving distribution mirrors, software repositories, and collections of photos, videos, audi...
## FEATURES
- Directory and file replication between multiple resources
- Simple file addition workflow — just copy files into the resource directory
- No complex configuration required: behavior is controlled through command-line options and descriptor files
- Dynamic addition of new resources (mirrors)
- Installation via `RPM`, `APT`, `CPAN`, or manually with `make install`
## REQUIREMENTS
```
## RESOURCE DIRECTORY STRUCTURE
The resource directory is the root of the mirror and contains all files and descriptors. A single server may host multiple resources with different URLs.
Some descriptor files **must not** be edited manually — they are marked accordingly.
### META
*NOT EDITABLE.* A YAML file containing metadata about the resource, including the last synchronization time. Also provides system information required by other mirrors.
### MANIFEST
*NOT EDITABLE.* Generated automatically on each run. Describes the current directory structure. After sync completion, it is regenerated and the `mtime` field in `META` is updated.
Format:
```
DIRNAME/FILENAME MTIME SIZE MTIME_AS_STRING
```
Format:
```
DIRNAME/FILENAME COMMENT
```
### MANIFEST.DEL
Editable descriptor listing files that must be deleted after a specified offset (`DTIME`). `DTIME` is relative to the modification time of `MANIFEST.DEL` itself (default: `+3d`).
The file is **not synchronized**, but remote mirrors download it and delete the listed files locally. After processing, the file is cleared and recreated.
Format:
```
DIRNAME/FILENAME DTIME
```
### MIRRORS
Editable descriptor listing the URLs of all mirrors. The current resource **must** be included in this list.
Format:
```
URL COMMENT
```
### MANIFEST.LOCK
*NOT EDITABLE.* Contains the PID of the process performing synchronization. Prevents concurrent `distsync` runs across mirrors.
### MANIFEST.TEMP
*NOT EDITABLE.* Temporary file holding downloaded data. May persist between runs.
### README
Optional, local-only informational file. Not synchronized. If you need it to sync, use `README.md` instead.
## GETTING STARTED
After initialization, edit the `MIRRORS` file and add the new mirror’s URL. Then upload the updated file to any existing mirror so others can discover the new resource.
Run the first synchronization:
```bash
distsync -D /var/www/foo.localhost sync -d
```
`-d` enables progress output; `-dv` adds more verbose diagnostics.
### Important option
`-D DATADIR` — Specifies the local resource directory where synchronized files are stored.
Help:
```bash
distsync --help
man distsync
```
## PRODUCTION USE
Once mirrors can reach the newly created resource, it may be scheduled for automatic, periodic synchronization. The most common method is cron.
Example crontab entry:
```cron
37 * * * * /usr/bin/distsync -D /var/www/foo.localhost >/var/log/distsync.log 2>&1
```
All logs and errors are written to `/var/log/distsync.log`.
### Adding files
Copy files into the resource directory. They will be synchronized automatically.
### Deleting files
Add filenames to `MANIFEST.DEL`. The system will handle deletion. You may remove files from disk manually afterwards.
### Updating files
Replace the file with a newer version. Synchronization will propagate the change.
âš **Do NOT rename or move files or directories.**
Missing items will be recreated during synchronization, leading to divergence or duplication across mirrors.
bin/distsync view on Meta::CPAN
#!/usr/bin/perl -w
use strict;
use warnings;
use utf8;
use feature qw/say/;
=encoding utf8
=head1 NAME
distsync - launcher of synchronization via App::DistSync
=head1 SYNOPSIS
distsync [options] [commands]
distsync -D /var/www/dist init
distsync [-dv] -D /var/www/dist status
distsync [-d] -D /var/www/dist [-T TIMEOUT] [sync]
distsync [-d] [-T TIMEOUT] sync /var/www/dist
distsync [-dv] -D /var/www/dist manifest|mkmani
bin/distsync view on Meta::CPAN
=item B<-x PROXY_URL, --proxy=PROXY_URL>
Sets proxy URL (http/socks4/socks5)
For example: socks://user:pass@192.168.0.1:1080
=back
=head1 DESCRIPTION
Launcher of synchronization via App::DistSync
See C<README> file for details
=head1 AUTHOR
Serż Minus (Sergey Lepenkov) L<https://www.serzik.com> E<lt>abalama@cpan.orgE<gt>
=head1 COPYRIGHT
Copyright (C) 1998-2025 D&D Corporation. All Rights Reserved
lib/App/DistSync.pm view on Meta::CPAN
package App::DistSync;
use strict;
use warnings;
use utf8;
use feature qw/say/;
=encoding utf-8
=head1 NAME
App::DistSync - Utility for synchronizing distribution mirrors
=head1 SYNOPSIS
use App::DistSync;
my $ds = App::DistSync->new(
dir => "/var/www/www.example.com/dist",
pid => $$,
timeout => 60,
proxy => 'http://http.example.com:8001/',
);
$ds->init or die "Initialization error";
$ds->sync or die "Sync error";
=head1 DESCRIPTION
Utility for synchronizing distribution mirrors
=head1 METHODS
This module implements the following methods
=head2 new
my $ds = new App::DistSync(
dir => "/var/www/www.example.com/dist",
pid => $$,
lib/App/DistSync.pm view on Meta::CPAN
sub init { # Initialization
my $self = shift;
my $stamp = scalar(localtime($self->{started}));
my $status = 1;
# MANIFEST.SKIP
printf "%s... ", $self->{file_maniskip};
if (touch($self->{file_maniskip}) && (-e $self->{file_maniskip}) && -z $self->{file_maniskip}) {
my @content = (
"# Generated on $stamp",
"# List of files that should not be synchronized",
"#",
"# Format of file:",
"#",
"# dir1/dir2/.../dirn/foo.txt any comment, for example blah-blah-blah",
"# bar.txt any comment, for example blah-blah-blah",
"# baz.txt",
"# 'spaced dir1/foo.txt' any comment, for example blah-blah-blah",
"# 'spaced dir1/foo.txt' any comment, for example blah-blah-blah",
"# !!perl/regexp (?i-xsm:\\.bak\$) avoid all bak files",
"#",
lib/App/DistSync.pm view on Meta::CPAN
}
} else {
debug("Skipped. File %s not exists", MANIDEL);
}
}
# Adding files listed in MANIFEST.DEL to the exclusion list
for (keys %$dellist) {$skips{$_} = qrreconstruct($_)}
}
# Reading the MIRRORS file and deciding whether to synchronize or not
debug("Synchronization");
my $mirrors_mani = maniread($self->{file_mirrors}) // {}; # MIRRORS
my @mirrors = sort {$a cmp $b} keys %$mirrors_mani;
if (scalar(@mirrors)) {
foreach my $url (@mirrors) {
debug("RESOURCE \"%s\"", $url);
# Downloading the MANIFEST.LOCK file, skipping the mirror resource if this
# file was successfully downloaded from the resource
{
lib/App/DistSync.pm view on Meta::CPAN
if (-e $f) {
fdelete($f);
debug("> [DELETED] %s", $k);
} else {
debug("> [SKIPPED] %s (%s)", $k, $f);
}
}
}
#debug(Data::Dumper::Dumper(\%delete_list));
# Iterate through the synchronization list and download all files that
# are NOT present in the previously generated deletion list.
#debug(Data::Dumper::Dumper(\%sync_list));
{
debug("Downloading files");
my $total = 0; # Size
my $cnt = 0; # File number
my $all = scalar(keys %sync_list);
my $af = '[%0' . length("$all") . 'd/%0' . length("$all") . 'd] %s';
foreach my $k (sort {lc($a) cmp lc($b)} keys %sync_list) { $cnt++;
debug($af, $cnt, $all, $k);
( run in 0.770 second using v1.01-cache-2.11-cpan-ff066701436 )