App-FileSync

 view release on metacpan or  search on metacpan

website/html/how-it-works.html  view on Meta::CPAN


	<div class="toc">
		<p>On this page</p>
		<ul>
			<li><a href="#session">One session</a></li>
			<li><a href="#transfer">Transfer and deltas</a></li>
			<li><a href="#conflicts">Conflicts</a></li>
			<li><a href="#deletions">Deletions and trash</a></li>
			<li><a href="#snapshots">Replica snapshots</a></li>
			<li><a href="#layout">On-disk layout</a></li>
			<li><a href="#locking">Locking and concurrency</a></li>
			<li><a href="#security">Security model</a></li>
		</ul>
	</div>

	<h2 id="session">One session</h2>

	<p>Every run is a short-lived process. In outline:</p>

	<ol class="steps">
		<li>
			<h3>The client checks its root and takes its lock</h3>
			<p>The sync root must contain the <code>.filesync-root</code> marker that
			<code>filesync init</code> wrote, so an unmounted drive is reported rather
			than treated as a tree of deletions. A local lock keeps two runs of the
			same repo apart.</p>
		</li>
		<li>
			<h3>It opens the transport</h3>
			<p>Over ssh, with <code>BatchMode=yes</code> and a forced command pinned in
			<code>authorized_keys</code>. The pinned command carries the repo, the
			client id and the role, so the session itself never chooses the server
			root. The <code>local</code> transport runs the same protocol over a pipe
			for testing and single-machine use.</p>
		</li>
		<li>
			<h3>It scans the tree and sends a manifest</h3>
			<p>The walk is sorted, so manifests and logs are reproducible. A cache of
			size and mtime decides which files need hashing. Ignore patterns are
			applied on both sides. The manifest has one entry per file, and it is sent
			whole: there is no change journal.</p>
		</li>
		<li>
			<h3>The server plans against its index</h3>
			<p>Holding the repo lock, the server compares the client's manifest with
			the master index and with that client's base, and answers with one plan:
			what to pull, what to push, what to delete, and which paths conflict.</p>
		</li>
		<li>
			<h3>Both sides move the files the plan names</h3>
			<p>Pulls are requested in chunks and pushes are sent in chunks; each
			256&nbsp;KiB chunk waits for its acknowledgement, so one request is in
			flight at a time. Files of at least 64&nbsp;KiB are sent as block deltas
			when the other side already holds an older copy.</p>
		</li>
		<li>
			<h3>Transfers are verified, then committed</h3>
			<p>An upload is checked against its SHA-256 at the end and only then
			renamed into the master. A pull is checked against the hash in the plan; a
			mismatch fails that pull and the next run retries. Temporary names carry
			a <code>.filesync-tmp-</code> prefix, which the backup example excludes.</p>
		</li>
		<li>
			<h3>State is written and the run is logged</h3>
			<p>The server rewrites the index and the client's base and appends to the
			repo journal. The client updates its cache, prunes its local trash unless
			<code>--no-prune</code> was given, and appends to
			<code>~/.filesync/logs/REPO.jsonl</code> whatever the outcome. The summary
			reports pulled, pushed, deleted and conflicted counts, the payload bytes
			in each direction, and how many files moved as a delta.</p>
		</li>
	</ol>

	<h2 id="transfer">Transfer and deltas</h2>

	<p>Each control message is one JSON object on one line. A message that
	announces a payload carries <code>len</code>, the bytes on the wire, and
	<code>z</code> when the payload was deflated; the payload follows the line
	directly. Only payload bytes are counted in the transfer report; headers and
	control messages are excluded.</p>

	<p>Deflation is used when it makes the frame smaller. For a file the other side
	already has an older copy of, the sender builds a signature of the receiver's
	blocks and sends copy instructions plus literals, rsync-style. The rolling
	scan runs in pure Perl, so it covers at most the first 8&nbsp;MiB after the
	unchanged prefix; later changes go as literals. Offsets and lengths are 32-bit
	on the wire, so a delta cannot refer beyond 4&nbsp;GiB into the receiver's
	file; larger remainders fall back to whole-file transfer.</p>

	<p>Writing is atomic between processes: every file is written to a temporary
	name and renamed into place. Nothing is <code>fsync</code>ed, so a rename
	survives a crash as a name but not necessarily as durable data.</p>

	<h2 id="conflicts">Conflicts</h2>

	<p>When both sides changed a path, the newer mtime wins, and the master wins
	ties within two seconds. The loser is not discarded: it is kept beside the
	winner as a conflict copy named before the extension, for example</p>

<pre><code>report.conflict-laptop-1-20261022T101500Z.txt</code></pre>

	<p>The name is capped at 200 bytes so it fits common filesystems. The server
	keeps a conflict copy under the planned name whether or not the client names
	it, and refuses any other name; acknowledgements are accepted only for actions
	the client was told to apply.</p>

	<p>Because the tie-break uses mtimes, a client with a wrong clock can pick the
	wrong winner. The losing content is still preserved, so the cost is a merge,
	not lost data.</p>

	<h2 id="deletions">Deletions and trash</h2>

	<p>A deletion is a plan action like any other, but it is bounded: by default a
	session may delete at most 200 files or 10% of the known files, whichever
	limit is reached first. A tree that suddenly looks empty therefore stops the
	run instead of being propagated. <code>filesync sync --force</code> proceeds
	when that is really what you want.</p>

	<p>On the client, a removed file is moved into that repo's local trash, kept
	for 30 days by default. A second deletion of the same path on the same day is
	stored as <code>PATH.1</code>, then <code>PATH.2</code>, and so on. The server
	keeps its own trash, with the retention set per repo.</p>

<pre><code>filesync restore notes/a.txt --list
filesync restore notes/a.txt
filesync restore notes/a.txt --to /tmp/recovered</code></pre>

	<p>Restores copy rather than move, so the trash entry survives and the
	operation is repeatable. The destination must not already exist.</p>

	<h2 id="snapshots">Replica snapshots</h2>

	<p>A machine registered with <code>--role replica</code> may take a daily
	snapshot of its tree. Snapshots live in <code>.filesync-snapshots</code> inside
	the replica root and are made by hard link or by copy, chosen with
	<code>--snapshot-mode</code>. Fourteen are kept by default.</p>

	<p>A hard-linked snapshot shares the file with the live tree. The sync replaces
	files by rename, which leaves the snapshot's link intact, but any other tool
	that edits a replica file in place also changes every snapshot that links it.
	Use <code>snapshot_mode copy</code> if anything but filesync writes there.</p>

	<p>Snapshots guard against a bad sync, not against losing the disk: they sit on
	the same filesystem as the replica.</p>

	<h2 id="layout">On-disk layout</h2>

	<h3>On the client</h3>

	<div class="table-wrap">
	<table>
		<thead><tr><th>Path</th><th>Holds</th></tr></thead>
		<tbody>
			<tr><td><code>~/.filesync/config.json</code></td><td>The repo list, written by <code>filesync init</code>. <code>$FILESYNC_CONFIG</code> overrides the path, <code>$FILESYNC_HOME</code> the directory.</td></tr>
			<tr><td><code>~/.filesync/logs/REPO.jsonl</code></td><td>One JSON object per run attempt, successful or not.</td></tr>
			<tr><td><code>~/.filesync/</code></td><td>The cache, the per-repo lock and the local trash as well.</td></tr>
			<tr><td><code>ROOT/.filesync-root</code></td><td>The marker that proves the sync root is mounted and is the tree the client registered.</td></tr>
		</tbody>
	</table>
	</div>

	<h3>On the server</h3>

	<div class="table-wrap">
	<table>
		<thead><tr><th>Path</th><th>Holds</th></tr></thead>
		<tbody>
			<tr><td><code>ROOT/master/REPO/</code></td><td>The master tree, an ordinary directory.</td></tr>
			<tr><td><code>ROOT/.filesync/repos/REPO/config.json</code></td><td>The repo policy and the admitted clients.</td></tr>
			<tr><td><code>ROOT/.filesync/repos/REPO/</code></td><td>The index, the per-client bases, the server trash and the journal.</td></tr>
		</tbody>
	</table>
	</div>

	<p>The master is a normal tree, so backup is rsync and restore is rsync plus
	<code>filesync-admin reindex</code>. The index is a JSON file rewritten whole
	at the end of each session.</p>

	<h2 id="locking">Locking and concurrency</h2>

	<p>Both the client and the server take locks, and a refusal is not an error:
	<code>filesync</code> exits <strong>2</strong> so cron can tell "a run was
	already in progress" from "a run failed". The plan is computed once, from a
	scan taken while the repo lock is held; something else writing into the master
	during a session is not seen until the next one.</p>

	<p><code>flock</code> is advisory and, on NFS, may not lock across machines at
	all. Keep the server root and the client's <code>~/.filesync</code> on local
	disks.</p>

	<h2 id="security">Security model</h2>

	<ul>
		<li><strong>Authentication is ssh's.</strong> The client id is a name for
		bases, journals and conflict copies, not a credential. ssh runs with
		<code>BatchMode=yes</code>, so cron cannot sit at a password prompt.</li>
		<li><strong>The server pins the session.</strong> sshd starts
		<code>filesync-server</code> with <code>--repo</code>, <code>--client</code>
		and <code>--role</code> from <code>authorized_keys</code>. An ssh server of
		the form <code>-v@host</code>, or a program name starting with <code>-</code>,
		is refused rather than passed to ssh as an option.</li>
		<li><strong>Ids are plain.</strong> A repo or client name must match
		<code>[A-Za-z0-9][A-Za-z0-9._-]*</code> and be at most 64 characters, so
		<code>../x</code> cannot become a path.</li>
		<li><strong>Paths are validated on the wire.</strong> No absolute paths,
		drive letters, backslashes, NUL or control characters, empty or
		<code>.</code>/<code>..</code> components; at most 4096 bytes, and no
		component over 255. Symlinks are not followed.</li>
		<li><strong>Frames are bounded.</strong> A payload length must be a number no
		larger than 64&nbsp;MiB, and the receiver refuses a frame that does not match
		it. An upload larger than announced is refused and discarded.</li>
		<li><strong>Errors carry no file paths by default.</strong> Commands strip
		Perl's <code>at FILE line N</code> suffix; <code>FILESYNC_TRACE=1</code>
		turns every error into a full stack trace when debugging.</li>
	</ul>
</main>

<footer class="site-footer">
	<div class="wrap">
		<p>App::FileSync 0.002.0 &middot; Copyright 2026 Nigel Horne &middot; GPL-2.0</p>
		<p><a href="https://metacpan.org/dist/App-FileSync">CPAN</a> &middot;
		<a href="https://github.com/nigelhorne/App-FileSync">GitHub</a> &middot;
		<a href="https://github.com/nigelhorne/App-FileSync/issues">Issues</a> &middot;
		<a href="changelog.html">Changes</a></p>
	</div>
</footer>
</body>
</html>



( run in 2.116 seconds using v1.01-cache-2.11-cpan-99fa8243a42 )