App-FileSync
view release on metacpan or search on metacpan
host name is not the name you want, and allow that name on the server
instead. An id is not a login - the ssh key and the account do the
authenticating - and it does not divide the repo: every client of a repo
syncs the whole tree.
ssh runs with `BatchMode=yes`, so the client's key must already be in the
server account's `authorized_keys`, or loaded in `ssh-agent`: cron cannot
answer a password prompt.
### From Cron
```
# crontab -e, on each client
* * * * * ionice -c3 /usr/local/bin/filesync sync --repo documents --quiet >/dev/null
```
The summary line is dropped so cron does not mail on every successful run;
diagnostics still go to standard error, so a failure is mailed. Every attempt,
successful or not, is also appended to `~/.filesync/logs/documents.jsonl`, and
`filesync log --repo documents --exit 1` lists the ones that failed.
### On Windows
Windows clients use a scheduled task rather than cron. With the script in
`C:\filesync` and Strawberry Perl installed, the same job is set up from a
command prompt:
```
REM on the client, once
C:\Strawberry\perl\bin\perl.exe C:\filesync\bin\filesync init ^
--repo documents --root C:\Users\foo\Documents --server foo@srv ^
--server-command /home/foo/bin/filesync-server-home
REM then create the task; it runs every minute as the current user
schtasks /Create /SC MINUTE /MO 1 /TN FileSync /F ^
/TR "C:\Strawberry\perl\bin\perl.exe C:\filesync\bin\filesync sync --repo documents --quiet"
```
The task runs only while that user is logged on, which suits a Documents
folder; its run log is still written under `%USERPROFILE%\.filesync\logs`, so
`filesync log --repo documents --warnings` works there too.
### A Root of /srv/filesync Instead
To keep the master outside a home directory, create the default root once for
the account the clients log in to. Then no `--root` is needed on the server
commands, and no `--server-command` on the client:
```
# on the server, as an administrator
sudo mkdir -p /srv/filesync
sudo chown -R foo /srv/filesync
```
`filesync-admin repo create documents` works as it is, and the master is
`/srv/filesync/master/documents`.
### Backup and Restore
The master directory is an ordinary directory tree, so it can be mirrored to a
backup directory from the server account's crontab. Make the backup directory
once - rsync creates the last component only - then let cron keep it in step:
```
# on the server, once
mkdir -p /home/foo/backups/documents
# crontab -e, on the server
* * * * * ionice -c3 rsync -a --delete --exclude '.filesync-tmp-*' \
/home/foo/filesync/master/documents/ /home/foo/backups/documents/
```
Filesync writes each file to a temporary name and renames it into place, so the
mirror never catches a half-written upload; the exclusion keeps the temporary
files themselves out of the backup. `--delete` makes the copy a mirror: a
deletion that reaches the master also removes the backed-up file.
To restore the master from that backup, stop the client jobs (comment out their
cron lines, or disable the scheduled task) and the server's backup job, so the
backup job cannot copy a half-restored tree back into the backup. Let any
running sync finish, then mirror the backup back:
```
rsync -a --delete --checksum \
/home/foo/backups/documents/ /home/foo/filesync/master/documents/
```
`--checksum` compares the contents, so a restored file is not skipped just
because its size and modification time happen to match the newer copy. This is
a rollback rather than a merge: files created after the backup was taken
disappear from the master, and the next sync propagates that to the clients, so
copy anything you need to keep out of the tree first. The server
would rehash the restored files on the next session anyway, but
`filesync-admin reindex` rebuilds the index immediately, before any client
reconnects; clients then pull the restored versions, merging edits they have not
synced as usual.
Before re-enabling the jobs, check that the copy is exact with a read-only
comparison against the backup:
```
rsync -a --delete --checksum --dry-run --itemize-changes \
/home/foo/backups/documents/ /home/foo/filesync/master/documents/
```
No output and exit 0 mean the master matches the backup. Any line beginning
`>f` names a path that still differs, and `*deleting` names one the
rollback should have removed. The index still describes the pre-rollback tree,
so `filesync-admin verify` may report drift at this point; that is expected.
Rebuild it and check it:
```
filesync-admin --root ~/filesync reindex documents
filesync-admin --root ~/filesync verify documents --deep
```
`reindex` hashes every file in the master and replaces the index atomically; it
refuses to run while a sync session holds the repo lock, and it names any paths
it could not index. `clean` and exit 0 from `verify` confirm that the master
and its index agree. Re-enable the jobs once that check passes.
To restore just one file or directory, copy that path back instead of mirroring
the whole tree, with the jobs paused as above so nothing changes the tree while
the copy runs:
```
rsync -a --checksum /home/foo/backups/documents/notes/plan.txt \
/home/foo/filesync/master/documents/notes/
rsync -a --checksum /home/foo/backups/documents/photos/ \
/home/foo/filesync/master/documents/photos/
```
The trailing slash on the second source copies the directory's contents; without
it rsync would put the directory itself inside the destination. Add `--delete`
only when that directory should match the backup exactly, since it removes files
created after the backup was taken. Everything outside the restored paths is
left alone; refresh the index with `reindex` as above, or let the next session
do it, and clients pull the restored versions. Check the restored paths with the
matching `rsync --dry-run --itemize-changes` comparison before re-enabling the
jobs; no output means they match the backup.
Adjust `/usr/local/bin/filesync` to where the script was installed, because
cron's default PATH rarely includes a per-user Perl bin directory.
## Limitations
These are the design weaknesses a reviewer should weigh before relying on the
tool. None of them loses data silently; most cost time or space.
- **One master.** The topology is a star around one server. The replica
is a pull-only mirror, not a failover: if the server is lost, clients stop
until an administrator points them at a rebuilt master.
- **Every run exchanges the whole manifest.** The client sends one entry
per file and the server answers with the whole plan as a single JSON line, so
a run costs O(files) in time, memory and bandwidth even when nothing changed.
A tree of a million files means a plan message of the order of a hundred
megabytes. A change journal or Merkle summary would remove that, at the cost
of the "everything is inspectable with cat" property.
- **No line-length cap on the wire.** Because of the previous point, a
control message cannot be bounded without breaking large trees. The peer is
authenticated by ssh and pinned by the forced command, so this is a resource
risk from a misbehaving client, not from the network.
- **Lock-step protocol.** One request is in flight at a time and each
256 KiB chunk waits for its acknowledgement, so throughput is bound by the
round-trip time on a high-latency link.
- **Clock-dependent tie-breaks.** When both sides changed a file, the
newer mtime wins (the master wins ties within 2 seconds). A client with a
wrong clock picks the wrong winner; the loser is still kept as a conflict copy.
- **No fsync.** Renames are atomic between processes but not durable
across a power cut; see ["LIMITATIONS" in App::FileSync::IO](https://metacpan.org/pod/App%3A%3AFileSync%3A%3AIO#LIMITATIONS).
- **Pure-Perl deltas are bounded.** Only the first 8 MiB after the
unchanged prefix are scanned for moved blocks; later changes go as literals.
See [App::FileSync::Delta](https://metacpan.org/pod/App%3A%3AFileSync%3A%3ADelta).
- **What is not synced.** Symlinks, devices, sockets, extended attributes,
ACLs and ownership. Permission bits are carried only between POSIX systems.
- **Names.** Unicode normalisation is not reconciled, so the same name
in NFC (Linux) and NFD (older macOS) is two different files. A
case-insensitive client cannot hold two master paths that differ only in case.
- **Encapsulation costs.** Private methods use [Sub::Private](https://metacpan.org/pod/Sub%3A%3APrivate) in
`enforce` mode, which costs about 1.4 microseconds per call (measured on
5.40). Helpers that run per file or per JSON value (the merge comparisons,
the JSON byte downgrade, the rolling checksum) are therefore lexical code
references: private by scope, with no wrapper. Both Sub::Private and
[Sub::Protected](https://metacpan.org/pod/Sub%3A%3AProtected) are bypassed while `HARNESS_ACTIVE` is set.
- **Validation costs.** [Params::Validate::Strict](https://metacpan.org/pod/Params%3A%3AValidate%3A%3AStrict) costs about 11
microseconds per call, so it guards the public entry points that run a few
times per command (constructors, admin and client operations). Functions that
run per path or per protocol frame (["validate\_relpath"](#validate_relpath), the protocol
methods, the scanner) check their arguments by hand.
- **Errors carry locations.** Library errors are raised with `croak` and
so end with `at FILE line N.`; the commands strip that before printing, and
`FILESYNC_TRACE=1` turns every error into a full stack trace instead.
- **English only.** The catalogue mechanism supports other languages,
plural forms and gender, but only the English catalogue ships.
## See Also
- [App::FileSync::I18N](https://metacpan.org/pod/App%3A%3AFileSync%3A%3AI18N) - the message catalogue
- [Test Dashboard](https://nigelhorne.github.io/App-FileSync/coverage/)
( run in 1.323 second using v1.01-cache-2.11-cpan-99fa8243a42 )