view release on metacpan or  search on metacpan

script/uniq-files  view on Meta::CPAN

use Perinci::CmdLine::Any;

our $AUTHORITY = 'cpan:PERLANCAR'; # AUTHORITY
our $DATE = '2025-05-03'; # DATE
our $DIST = 'App-UniqFiles'; # DIST
our $VERSION = '0.144'; # VERSION

my $cmdline = Perinci::CmdLine::Any->new(
    url => "/File/FindUniq/uniq_files",
    program_name => "uniq-files",
    log => 1,
    read_config => 0,
    read_env => 0,
);

$cmdline->run;

# ABSTRACT: Report duplicate or unique files, optionally perform action on them
# PODNAME: uniq-files

__END__

=pod

=encoding UTF-8

=head1 NAME

uniq-files - Report duplicate or unique files, optionally perform action on them

=head1 VERSION

This document describes version 0.144 of uniq-files (from Perl distribution App-UniqFiles), released on 2025-05-03.

=head1 SYNOPSIS

B<uniq-files> B<L<--help|/"--help, -h, -?">> (or B<L<-h|/"--help, -h, -?">>, B<L<-?|/"--help, -h, -?">>)

B<uniq-files> B<L<--version|/"--version, -v">> (or B<L<-v|/"--version, -v">>)

B<uniq-files> [B<L<--algorithm|/"--algorithm=s">>=I<str>] [(B<L<--authoritative-dir|/"--authoritative-dirs-json=s">>=I<str>)+|B<L<--authoritative-dirs-json|/"--authoritative-dirs-json=s">>=I<json>|(B<L<-O|/"--authoritative-dirs-json=s">>=I<str>)+] [B...



See examples in the L</EXAMPLES> section.

=head1 DESCRIPTION

Given a list of filenames, will check each file's content (and/or size, and/or
only name) to decide whether the file is a duplicate of another.

There is a certain amount of flexibility on how duplicate is determined:
- when comparing content, various hashing algorithm is supported;
- when comparing size, a certain tolerance % is allowed;
- when comparing filename, munging can first be done.

There is flexibility on what to do with duplicate files:
- just print unique/duplicate files (and let other utilities down the pipe deal
  with them);
- move duplicates to some location;
- open the files first and prompt for action;
- let a Perl code process the files.

Interface is loosely based on the C<uniq> Unix command-line program.

=head1 OPTIONS

C<*> marks required options.

=head2 Main options

=over

=item B<--algorithm>=I<s>

What algorithm is used to compute the digest of the content.

The default is to use C<md5>. Some algorithms supported include C<crc32>, C<sha1>,
C<sha256>, as well as C<Digest> to use Perl L<Digest> which supports a lot of
other algorithms, e.g. C<SHA-1>, C<BLAKE2b>.

If set to '', 'none', or 'size', then digest will be set to file size. This
means uniqueness will be determined solely from file size. This can be quicker
but will generate a false positive when two files of the same size are deemed as
duplicate even though their content may be different.

If set to 'name' then only name comparison will be performed. This of course can
potentially generate lots of false positives, but in some cases you might want
to compare filename for uniqueness.


=item B<--authoritative-dir>=I<s@>, B<-O>

Denote director(y|ies) where authoritative/"Original" copies are found.

Can be specified multiple times.

=item B<--authoritative-dirs-json>=I<s>

Denote director(y|ies) where authoritative/"Original" copies are found (JSON-encoded).

See C<--authoritative-dir>.

=item B<--detail>, B<-l>

Show details (a.k.a. --show-digest, --show-size, --show-count).

=item B<--digest-args-json>=I<s>, B<-A>

See C<--digest-args>.

=item B<--digest-args>=I<s>

Some Digest algorithms require arguments, you can pass them here.


=item B<--exclude-empty-files>, B<-Z>

(No description)