App-cloc
view release on metacpan or search on metacpan
$script [options] --diff <set1> <set2>
Compute differences of physical lines of source code and comments
between any pairwise combination of directory names, archive
files or git commit hashes.
Example: cloc --diff Python-3.5.tar.xz python-3.6/
$script --help shows full documentation on the options.
http://$URL has numerous examples and more information.
";
my $usage = "
Usage: $script [options] <file(s)/dir(s)/git hash(es)> | <set 1> <set 2> | <report files>
Count, or compute differences of, physical lines of source code in the
given files (may be archives such as compressed tarballs or zip files,
or git commit hashes or branch names) and/or recursively below the
given directories.
${BB}Input Options${NN}
--extract-with=<cmd> This option is only needed if cloc is unable
to figure out how to extract the contents of
the input file(s) by itself.
Use <cmd> to extract binary archive files (e.g.:
.tar.gz, .zip, .Z). Use the literal '>FILE<' as
a stand-in for the actual file(s) to be
extracted. For example, to count lines of code
in the input files
gcc-4.2.tar.gz perl-5.8.8.tar.gz
on Unix use
--extract-with='gzip -dc >FILE< | tar xf -'
or, if you have GNU tar,
--extract-with='tar zxf >FILE<'
and on Windows use, for example:
--extract-with=\"\\\"c:\\Program Files\\WinZip\\WinZip32.exe\\\" -e -o >FILE< .\"
(if WinZip is installed there).
--list-file=<file> Take the list of file and/or directory names to
process from <file>, which has one file/directory
name per line. Only exact matches are counted;
relative path names will be resolved starting from
the directory where cloc is invoked.
See also --exclude-list-file.
--vcs=<VCS> Invoke a system call to <VCS> to obtain a list of
files to work on. If <VCS> is 'git', then will
invoke 'git ls-files' to get a file list and
'git submodule status' to get a list of submodules
whose contents will be ignored. See also --git
which accepts git commit hashes and branch names.
If <VCS> is 'svn' then will invoke 'svn list -R'.
The primary benefit is that cloc will then skip
files explicitly excluded by the versioning tool
in question, ie, those in .gitignore or have the
svn:ignore property.
Alternatively <VCS> may be any system command
that generates a list of files.
Note: cloc must be in a directory which can read
the files as they are returned by <VCS>. cloc will
not download files from remote repositories.
'svn list -R' may refer to a remote repository
to obtain file names (and therefore may require
authentication to the remote repository), but
the files themselves must be local.
--unicode Check binary files to see if they contain Unicode
expanded ASCII text. This causes performance to
drop noticeably.
${BB}Processing Options${NN}
--autoconf Count .in files (as processed by GNU autoconf) of
recognized languages. See also --no-autogen.
--by-file Report results for every source file encountered.
--by-file-by-lang Report results for every source file encountered
in addition to reporting by language.
--count-and-diff <set1> <set2>
First perform direct code counts of source file(s)
of <set1> and <set2> separately, then perform a diff
of these. Inputs may be pairs of files, directories,
or archives. If --out or --report-file is given,
three output files will be created, one for each
of the two counts and one for the diff. See also
--diff, --diff-alignment, --diff-timeout,
--ignore-case, --ignore-whitespace.
--diff <set1> <set2> Compute differences in code and comments between
source file(s) of <set1> and <set2>. The inputs
may be any mix of files, directories, archives,
or git commit hashes. Use --diff-alignment to
generate a list showing which file pairs where
compared. See also --count-and-diff, --diff-alignment,
--diff-timeout, --ignore-case, --ignore-whitespace.
--diff-timeout <N> Ignore files which take more than <N> seconds
to process. Default is 10 seconds.
(Large files with many repeated lines can cause
Algorithm::Diff::sdiff() to take hours.)
--follow-links [Unix only] Follow symbolic links to directories
(sym links to files are always followed).
--force-lang=<lang>[,<ext>]
Process all files that have a <ext> extension
with the counter for language <lang>. For
example, to count all .f files with the
Fortran 90 counter (which expects files to
end with .f90) instead of the default Fortran 77
counter, use
--force-lang=\"Fortran 90\",f
If <ext> is omitted, every file will be counted
with the <lang> counter. This option can be
specified multiple times (but that is only
useful when <ext> is given each time).
See also --script-lang, --lang-no-ext.
--force-lang-def=<file> Load language processing filters from <file>,
then use these filters instead of the built-in
filters. Note: languages which map to the same
file extension (for example:
MATLAB/Mathematica/Objective C/MUMPS/Mercury;
Pascal/PHP; Lisp/OpenCL; Lisp/Julia; Perl/Prolog)
will be ignored as these require additional
processing that is not expressed in language
definition files. Use --read-lang-def to define
new language filters without replacing built-in
filters (see also --write-lang-def).
--git Forces the inputs to be interpreted as git targets
(commit hashes, branch names, et cetera) if these
are not first identified as file or directory
names. This option overrides the --vcs=git logic
if this is given; in other words, --git gets its
to the same directory as the original files.
--read-binary-files Process binary files in addition to text files.
This is usually a bad idea and should only be
attempted with text files that have embedded
binary data.
--read-lang-def=<file> Load new language processing filters from <file>
and merge them with those already known to cloc.
If <file> defines a language cloc already knows
about, cloc's definition will take precedence.
Use --force-lang-def to over-ride cloc's
definitions (see also --write-lang-def ).
--script-lang=<lang>,<s> Process all files that invoke <s> as a #!
scripting language with the counter for language
<lang>. For example, files that begin with
#!/usr/local/bin/perl5.8.8
will be counted with the Perl counter by using
--script-lang=Perl,perl5.8.8
The language name is case insensitive but the
name of the script language executable, <s>,
must have the right case. This option can be
specified multiple times. See also --force-lang,
--lang-no-ext.
--sdir=<dir> Use <dir> as the scratch directory instead of
letting File::Temp chose the location. Files
written to this location are not removed at
the end of the run (as they are with File::Temp).
--skip-uniqueness Skip the file uniqueness check. This will give
a performance boost at the expense of counting
files with identical contents multiple times
(if such duplicates exist).
--stdin-name=<file> Give a file name to use to determine the language
for standard input. (Use - as the input name to
receive source code via STDIN.)
--strip-comments=<ext> For each file processed, write to the current
directory a version of the file which has blank
and commented lines removed (in-line comments
persist). The name of each stripped file is the
original file name with .<ext> appended to it.
It is written to the current directory unless
--original-dir is on.
--sum-reports Input arguments are report files previously
created with the --report-file option. Makes
a cumulative set of results containing the
sum of data from the individual report files.
--processes=NUM [Available only on systems with a recent version
of the Parallel::ForkManager module. Not
available on Windows.] Sets the maximum number of
cores that cloc uses. The default value of 0
disables multiprocessing.
--unix Override the operating system autodetection
logic and run in UNIX mode. See also
--windows, --show-os.
--use-sloccount If SLOCCount is installed, use its compiled
executables c_count, java_count, pascal_count,
php_count, and xml_count instead of cloc's
counters. SLOCCount's compiled counters are
substantially faster than cloc's and may give
a performance improvement when counting projects
with large files. However, these cloc-specific
features will not be available: --diff,
--count-and-diff, --strip-comments, --unicode.
--windows Override the operating system autodetection
logic and run in Microsoft Windows mode.
See also --unix, --show-os.
${BB}Filter Options${NN}
--exclude-dir=<D1>[,D2,] Exclude the given comma separated directories
D1, D2, D3, et cetera, from being scanned. For
example --exclude-dir=.cache,test will skip
all files and subdirectories that have /.cache/
or /test/ as their parent directory.
Directories named .bzr, .cvs, .hg, .git, .svn,
and .snapshot are always excluded.
This option only works with individual directory
names so including file path separators is not
allowed. Use --fullpath and --not-match-d=<regex>
to supply a regex matching multiple subdirectories.
--exclude-ext=<ext1>[,<ext2>[...]]
Do not count files having the given file name
extensions.
--exclude-lang=<L1>[,L2,] Exclude the given comma separated languages
L1, L2, L3, et cetera, from being counted.
--exclude-list-file=<file> Ignore files and/or directories whose names
appear in <file>. <file> should have one file
name per line. Only exact matches are ignored;
relative path names will be resolved starting from
the directory where cloc is invoked.
See also --list-file.
--fullpath Modifies the behavior of --match-f, --not-match-f,
and --not-match-d to include the file's path
in the regex, not just the file's basename.
(This does not expand each file to include its
absolute path, instead it uses as much of
the path as is passed in to cloc.)
Note: --match-d always looks at the full
path and therefore is unaffected by --fullpath.
--include-lang=<L1>[,L2,] Count only the given comma separated languages
L1, L2, L3, et cetera.
--match-d=<regex> Only count files in directories matching the Perl
regex. For example
--match-d='/(src|include)/'
only counts files in directories containing
/src/ or /include/. Unlike --not-match-d,
--match-f, and --not-match-f, --match-d always
compares the fully qualified path against the
regex.
--not-match-d=<regex> Count all files except those in directories
matching the Perl regex. Only the trailing
directory name is compared, for example, when
counting in /usr/local/lib, only 'lib' is
compared to the regex.
Add --fullpath to compare parent directories to
the regex.
Do not include file path separators at the
beginning or end of the regex.
--match-f=<regex> Only count files whose basenames match the Perl
regex. For example
--match-f='^[Ww]idget'
only counts files that start with Widget or widget.
Add --fullpath to include parent directories
in the regex instead of just the basename.
statements. Only valid with the --sql option.
--sql-project=<name> Use <name> as the project identifier for the
current run. Only valid with the --sql option.
--sql-style=<style> Write SQL statements in the given style instead
of the default SQLite format. Currently, the
only style option is Oracle.
--sum-one For plain text reports, show the SUM: output line
even if only one input file is processed.
--xml Write the results in XML.
--xsl=<file> Reference <file> as an XSL stylesheet within
the XML output. If <file> is 1 (numeric one),
writes a default stylesheet, cloc.xsl (or
cloc-diff.xsl if --diff is also given).
This switch forces --xml on.
--yaml Write the results in YAML.
";
# Help information for options not yet implemented:
# --inline Process comments that appear at the end
# of lines containing code.
# --html Create HTML files of each input file showing
# comment and code lines in different colors.
$| = 1; # flush STDOUT
my $start_time = get_time();
my (
$opt_categorized ,
$opt_found ,
@opt_force_lang ,
$opt_lang_no_ext ,
@opt_script_lang ,
$opt_count_diff ,
$opt_diff ,
$opt_diff_alignment ,
$opt_diff_timeout ,
$opt_html ,
$opt_ignored ,
$opt_counted ,
$opt_show_ext ,
$opt_show_lang ,
$opt_progress_rate ,
$opt_print_filter_stages ,
$opt_v ,
$opt_vcs ,
$opt_version ,
$opt_exclude_lang ,
$opt_exclude_list_file ,
$opt_exclude_dir ,
$opt_explain ,
$opt_include_lang ,
$opt_force_lang_def ,
$opt_read_lang_def ,
$opt_write_lang_def ,
$opt_strip_comments ,
$opt_original_dir ,
$opt_quiet ,
$opt_report_file ,
$opt_sdir ,
$opt_sum_reports ,
$opt_processes ,
$opt_unicode ,
$opt_no3 , # accept it but don't use it
$opt_3 ,
$opt_extract_with ,
$opt_by_file ,
$opt_by_file_by_lang ,
$opt_by_percent ,
$opt_xml ,
$opt_xsl ,
$opt_yaml ,
$opt_csv ,
$opt_csv_delimiter ,
$opt_fullpath ,
$opt_json ,
$opt_md ,
$opt_match_f ,
$opt_not_match_f ,
$opt_match_d ,
$opt_not_match_d ,
$opt_skip_uniqueness ,
$opt_list_file ,
$opt_help ,
$opt_skip_win_hidden ,
$opt_read_binary_files ,
$opt_sql ,
$opt_sql_append ,
$opt_sql_project ,
$opt_sql_style ,
$opt_inline ,
$opt_exclude_ext ,
$opt_ignore_whitespace ,
$opt_ignore_case ,
$opt_follow_links ,
$opt_autoconf ,
$opt_sum_one ,
$opt_stdin_name ,
$opt_force_on_windows ,
$opt_force_on_unix , # actually forces !$ON_WINDOWS
$opt_show_os ,
$opt_skip_archive ,
$opt_max_file_size , # in MB
$opt_use_sloccount ,
$opt_no_autogen ,
$opt_force_git ,
);
my $getopt_success = GetOptions(
"by_file|by-file" => \$opt_by_file ,
"by_file_by_lang|by-file-by-lang" => \$opt_by_file_by_lang ,
"categorized=s" => \$opt_categorized ,
"counted=s" => \$opt_counted ,
"include_lang|include-lang=s" => \$opt_include_lang ,
"exclude_lang|exclude-lang=s" => \$opt_exclude_lang ,
"exclude_dir|exclude-dir=s" => \$opt_exclude_dir ,
"exclude_list_file|exclude-list-file=s" => \$opt_exclude_list_file ,
"explain=s" => \$opt_explain ,
"extract_with|extract-with=s" => \$opt_extract_with ,
"found=s" => \$opt_found ,
"count_and_diff|count-and-diff" => \$opt_count_diff ,
"diff" => \$opt_diff ,
"diff-alignment|diff_alignment=s" => \$opt_diff_alignment ,
"diff-timeout|diff_timeout=i" => \$opt_diff_timeout ,
"html" => \$opt_html ,
"ignored=s" => \$opt_ignored ,
"quiet" => \$opt_quiet ,
"force_lang_def|force-lang-def=s" => \$opt_force_lang_def ,
"read_lang_def|read-lang-def=s" => \$opt_read_lang_def ,
"show_ext|show-ext:s" => \$opt_show_ext ,
"show_lang|show-lang:s" => \$opt_show_lang ,
"progress_rate|progress-rate=i" => \$opt_progress_rate ,
"print_filter_stages|print-filter-stages" => \$opt_print_filter_stages ,
"report_file|report-file=s" => \$opt_report_file ,
"out=s" => \$opt_report_file ,
"script_lang|script-lang=s" => \@opt_script_lang ,
"sdir=s" => \$opt_sdir ,
"skip_uniqueness|skip-uniqueness" => \$opt_skip_uniqueness ,
"strip_comments|strip-comments=s" => \$opt_strip_comments ,
"original_dir|original-dir" => \$opt_original_dir ,
"sum_reports|sum-reports" => \$opt_sum_reports ,
"processes=n" => \$opt_processes ,
"unicode" => \$opt_unicode ,
"no3" => \$opt_no3 , # ignored
"3" => \$opt_3 ,
"v|verbose:i" => \$opt_v ,
"vcs=s" => \$opt_vcs ,
"version" => \$opt_version ,
"write_lang_def|write-lang-def=s" => \$opt_write_lang_def ,
"xml" => \$opt_xml ,
"xsl=s" => \$opt_xsl ,
"force_lang|force-lang=s" => \@opt_force_lang ,
"lang_no_ext|lang-no-ext=s" => \$opt_lang_no_ext ,
"yaml" => \$opt_yaml ,
"csv" => \$opt_csv ,
"csv_delimeter|csv-delimiter=s" => \$opt_csv_delimiter ,
"json" => \$opt_json ,
"md" => \$opt_md ,
"fullpath" => \$opt_fullpath ,
"match_f|match-f=s" => \$opt_match_f ,
"not_match_f|not-match-f=s" => \$opt_not_match_f ,
"match_d|match-d=s" => \$opt_match_d ,
"not_match_d|not-match-d=s" => \$opt_not_match_d ,
"list_file|list-file=s" => \$opt_list_file ,
"help" => \$opt_help ,
"skip_win_hidden|skip-win-hidden" => \$opt_skip_win_hidden ,
"read_binary_files|read-binary-files" => \$opt_read_binary_files ,
"sql=s" => \$opt_sql ,
"sql_project|sql-project=s" => \$opt_sql_project ,
"sql_append|sql-append" => \$opt_sql_append ,
"sql_style|sql-style=s" => \$opt_sql_style ,
"inline" => \$opt_inline ,
"exclude_ext|exclude-ext=s" => \$opt_exclude_ext ,
"ignore_whitespace|ignore-whitespace" => \$opt_ignore_whitespace ,
"ignore_case|ignore-case" => \$opt_ignore_case ,
"follow_links|follow-links" => \$opt_follow_links ,
"autoconf" => \$opt_autoconf ,
"sum_one|sum-one" => \$opt_sum_one ,
"by_percent|by-percent=s" => \$opt_by_percent ,
"stdin_name|stdin-name=s" => \$opt_stdin_name ,
"windows" => \$opt_force_on_windows ,
"unix" => \$opt_force_on_unix ,
"show_os|show-os" => \$opt_show_os ,
"skip_archive|skip-archive=s" => \$opt_skip_archive ,
"max_file_size|max-file-size=i" => \$opt_max_file_size ,
"use_sloccount|use-sloccount" => \$opt_use_sloccount ,
"no_autogen|no-autogen" => \$opt_no_autogen ,
"git" => \$opt_force_git ,
);
$opt_by_file = 1 if defined $opt_by_file_by_lang;
my $CLOC_XSL = "cloc.xsl"; # created with --xsl
$CLOC_XSL = "cloc-diff.xsl" if $opt_diff;
die "\n" unless $getopt_success;
print $usage and exit if $opt_help;
my %Exclude_Language = ();
%Exclude_Language = map { $_ => 1 } split(/,/, $opt_exclude_lang)
if $opt_exclude_lang;
my %Exclude_Dir = ();
%Exclude_Dir = map { $_ => 1 } split(/,/, $opt_exclude_dir )
if $opt_exclude_dir ;
die unless exclude_dir_validates(\%Exclude_Dir);
my %Include_Language = ();
%Include_Language = map { $_ => 1 } split(/,/, $opt_include_lang)
if $opt_include_lang;
# Forcibly exclude .svn, .cvs, .hg, .git, .bzr directories. The contents of these
# directories often conflict with files of interest.
$opt_exclude_dir = 1;
$Exclude_Dir{".svn"} = 1;
$Exclude_Dir{".cvs"} = 1;
$Exclude_Dir{".hg"} = 1;
$Exclude_Dir{".git"} = 1;
$Exclude_Dir{".bzr"} = 1;
$Exclude_Dir{".snapshot"} = 1; # NetApp backups
$opt_count_diff = defined $opt_count_diff ? 1 : 0;
$opt_diff = 1 if $opt_diff_alignment;
$opt_exclude_ext = "" unless $opt_exclude_ext;
$opt_ignore_whitespace = 0 unless $opt_ignore_whitespace;
$opt_ignore_case = 0 unless $opt_ignore_case;
$opt_lang_no_ext = 0 unless $opt_lang_no_ext;
$opt_follow_links = 0 unless $opt_follow_links;
$opt_diff_timeout =10 unless $opt_diff_timeout;
$opt_csv = 1 if $opt_csv_delimiter;
$ON_WINDOWS = 1 if $opt_force_on_windows;
$ON_WINDOWS = 0 if $opt_force_on_unix;
$opt_max_file_size = 100 unless $opt_max_file_size;
my $HAVE_SLOCCOUNT_c_count = 0;
if (!$ON_WINDOWS and $opt_use_sloccount) {
# Only bother doing this kludgey test is user explicitly wants
# to use SLOCCount. Debian based systems will hang if just doing
# external_utility_exists("c_count")
# if c_count is in $PATH; c_count expects to have input.
$HAVE_SLOCCOUNT_c_count = external_utility_exists("c_count /bin/sh");
}
if ($opt_use_sloccount) {
if (!$HAVE_SLOCCOUNT_c_count) {
warn "c_count could not be found; ignoring --use-sloccount\n";
$opt_use_sloccount = 0;
} else {
warn "Using c_count, php_count, xml_count, pascal_count from SLOCCount\n";
warn "--diff is disabled with --use-sloccount\n" if $opt_diff;
warn "--count-and-diff is disabled with --use-sloccount\n" if $opt_count_diff;
warn "--unicode is disabled with --use-sloccount\n" if $opt_unicode;
warn "--strip-comments is disabled with --use-sloccount\n" if $opt_strip_comments;
$opt_diff = 0;
$opt_count_diff = undef;
$opt_unicode = 0;
$opt_strip_comments = 0;
}
}
$opt_vcs = 0 if $opt_force_git;
my @COUNT_DIFF_ARGV = undef;
my $COUNT_DIFF_report_file = undef;
if ($opt_count_diff) {
die "--count-and-diff requires two arguments; got ", scalar @ARGV, "\n"
if scalar @ARGV != 2;
# prefix with a dummy term so that $opt_count_diff is the
# index into @COUNT_DIFF_ARGV to work on at each pass
@COUNT_DIFF_ARGV = (undef, $ARGV[0],
$ARGV[1],
[$ARGV[0], $ARGV[1]]); # 3rd pass: diff them
$COUNT_DIFF_report_file = $opt_report_file if $opt_report_file;
}
# Options defaults:
$opt_quiet = 1 if ($opt_md or $opt_json) and !defined $opt_report_file;
$opt_progress_rate = 100 unless defined $opt_progress_rate;
$opt_progress_rate = 0 if defined $opt_quiet;
if (!defined $opt_v) {
$opt_v = 0;
} elsif (!$opt_v) {
$opt_v = 1;
}
if (defined $opt_xsl) {
$opt_xsl = $CLOC_XSL if $opt_xsl eq "1";
$opt_xml = 1;
}
my $skip_generate_report = 0;
$opt_sql_style = 0 unless defined $opt_sql_style;
$opt_sql = 0 unless $opt_sql_style or defined $opt_sql;
if ($opt_sql eq "-" || $opt_sql eq "1") { # stream SQL output to STDOUT
$opt_quiet = 1;
$skip_generate_report = 1;
$opt_by_file = 1;
$opt_sum_reports = 0;
$opt_progress_rate = 0;
} elsif ($opt_sql) { # write SQL output to a file
$opt_by_file = 1;
$skip_generate_report = 1;
$opt_sum_reports = 0;
}
if ($opt_sql_style) {
$opt_sql_style = lc $opt_sql_style;
if (!grep { lc $_ eq $opt_sql_style } qw ( Oracle )) {
die "'$opt_sql_style' is not a recognized SQL style.\n";
}
}
$opt_by_percent = '' unless defined $opt_by_percent;
if ($opt_by_percent and $opt_by_percent !~ m/^(c|cm|cb|cmb)$/i) {
die "--by-percent must be either 'c', 'cm', 'cb', or 'cmb'\n";
}
$opt_by_percent = lc $opt_by_percent;
if (defined $opt_vcs) {
if ($opt_vcs eq "git") {
$opt_vcs = "git ls-files";
foreach my $index (sort {$a <=> $b} keys %replacement_arg_list) {
push @{$ra_arg_list}, $replacement_arg_list{$index};
}
#print "ra_arg_list 2: @{$ra_arg_list}\n";
} # 1}}}
sub git_archive { # {{{1
# Invoke 'git archive' as a system command to create a tar file
# using the given argument(s).
my ($args, ) = @_;
print "-> git_archive($args)\n" if $opt_v > 2;
my ($Tarfh, $Tarfile) = tempfile(UNLINK => 1, SUFFIX => $ON_WINDOWS ? '.zip' : '.tar'); # delete on exit
my $cmd = "git archive -o $Tarfile $args";
print $cmd, "\n" if $opt_v;
system $cmd;
if (!-r $Tarfile or !-s $Tarfile) {
# not readable, or zero sized
die "Failed to create tarfile of files from git.";
}
print "<- git_archive()\n" if $opt_v > 2;
return $Tarfile
} # 1}}}
sub make_file_list { # {{{1
my ($ra_arg_list, # in file and/or directory names to examine
$rh_Err , # in hash of error codes
$raa_errors , # out errors encountered
$rh_ignored , # out files not recognized as computer languages
) = @_;
print "-> make_file_list(@{$ra_arg_list})\n" if $opt_v > 2;
my ($fh, $filename);
if ($opt_categorized) {
$filename = $opt_categorized;
$fh = new IO::File $filename, "+>"; # open for read/write
die "Unable to write to $filename: $!\n" unless defined $fh;
} elsif ($opt_sdir) {
# write to the user-defined scratch directory
$filename = $opt_sdir . '/cloc_file_list.txt';
$fh = new IO::File $filename, "+>"; # open for read/write
die "Unable to write to $filename: $!\n" unless defined $fh;
} else {
# let File::Temp create a suitable temporary file
($fh, $filename) = tempfile(UNLINK => 1); # delete file on exit
print "Using temp file list [$filename]\n" if $opt_v;
}
my @dir_list = ();
foreach my $file_or_dir (@{$ra_arg_list}) {
#print "make_file_list file_or_dir=$file_or_dir\n";
my $size_in_bytes = 0;
if (!-r $file_or_dir) {
push @{$raa_errors}, [$rh_Err->{'Unable to read'} , $file_or_dir];
next;
}
if (is_file($file_or_dir)) {
if (!(-s $file_or_dir)) { # 0 sized file, named pipe, socket
$rh_ignored->{$file_or_dir} = 'zero sized file';
next;
} elsif (-B $file_or_dir and !$opt_read_binary_files) {
# avoid binary files unless user insists on reading them
if ($opt_unicode) {
# only ignore if not a Unicode file w/trivial
# ASCII transliteration
if (!unicode_file($file_or_dir)) {
$rh_ignored->{$file_or_dir} = 'binary file';
next;
}
} else {
$rh_ignored->{$file_or_dir} = 'binary file';
next;
}
}
push @file_list, "$file_or_dir";
} elsif (is_dir($file_or_dir)) {
push @dir_list, $file_or_dir;
} else {
push @{$raa_errors}, [$rh_Err->{'Neither file nor directory'} , $file_or_dir];
$rh_ignored->{$file_or_dir} = 'not file, not directory';
}
}
# apply exclusion rules to file names passed in on the command line
my @new_file_list = ();
foreach my $File (@file_list) {
my ($volume, $directories, $filename) = File::Spec->splitpath( $File );
my $ignore_this_file = 0;
foreach my $Sub_Dir ( File::Spec->splitdir($directories) ) {
if ($Exclude_Dir{$Sub_Dir}) {
$Ignored{$Sub_Dir} = "($File) --exclude-dir=$Sub_Dir";
$ignore_this_file = 1;
last;
}
}
push @new_file_list, $File unless $ignore_this_file;
}
@file_list = @new_file_list;
foreach my $dir (@dir_list) {
#print "make_file_list dir=$dir Exclude_Dir{$dir}=$Exclude_Dir{$dir}\n";
# populates global variable @file_list
if ($Exclude_Dir{$dir}) {
$Ignored{$dir} = "--exclude-dir=$Exclude_Dir{$dir}";
next;
}
find({wanted => \&files ,
preprocess => \&find_preprocessor,
follow => $opt_follow_links }, $dir);
}
# there's a possibility of file duplication if user provided a list
# file or --vcs command that returns directory names; squash these
my %unique_file_list = map { $_ => 1 } @file_list;
@file_list = sort keys %unique_file_list;
$nFiles_Found = scalar @file_list;
printf "%8d text file%s.\n", plural_form($nFiles_Found) unless $opt_quiet;
write_file($opt_found, sort @file_list) if $opt_found;
my $nFiles_Categorized = 0;
foreach my $file (@file_list) {
printf "classifying $file\n" if $opt_v > 2;
my $basename = basename $file;
if ($Not_Code_Filename{$basename}) {
$rh_ignored->{$file} = "listed in " . '$' .
next if $F_or_D =~ /^\.{1,2}$/; # skip . and ..
if ($Exclude_Dir{$F_or_D}) {
$Ignored{$File::Find::name} = "--exclude-dir=$Exclude_Dir{$F_or_D}";
} else {
#printf " F_or_D=%-20s File::Find::name=%s\n", $F_or_D, $File::Find::name;
if ($opt_not_match_d) {
if ($opt_fullpath and $File::Find::name =~ m{$opt_not_match_d}) {
$Ignored{$File::Find::name} = "--not-match-d=$opt_not_match_d";
} elsif (!-d $F_or_D and basename($File::Find::name) =~ m{$opt_not_match_d}) {
$Ignored{$File::Find::name} = "--not-match-d (basename) =$opt_not_match_d";
} else {
push @ok, $F_or_D;
}
} else {
push @ok, $F_or_D;
}
}
}
print "<- find_preprocessor(@ok)\n" if $opt_v > 2;
return @ok;
} # 1}}}
sub files { # {{{1
# invoked by File::Find's find() Populates global variable @file_list.
# See also find_preprocessor() which prunes undesired directories.
my $Dir = fastcwd(); # not $File::Find::dir which just gives relative path
if ($opt_fullpath) {
# look at as much of the path as is known
if ($opt_match_f ) {
return unless $File::Find::name =~ m{$opt_match_f};
}
if ($opt_not_match_f) {
return if $File::Find::name =~ m{$opt_not_match_f};
}
} else {
# only look at the basename
if ($opt_match_f ) { return unless /$opt_match_f/; }
if ($opt_not_match_f) { return if /$opt_not_match_f/; }
}
if ($opt_match_d ) { return unless $Dir =~ m{$opt_match_d} }
my $nBytes = -s $_ ;
if (!$nBytes) {
$Ignored{$File::Find::name} = 'zero sized file';
printf "files(%s) zero size\n", $File::Find::name if $opt_v > 5;
}
return unless $nBytes ; # attempting other tests w/pipe or socket will hang
if ($nBytes > $opt_max_file_size*1024**2) {
$Ignored{$File::Find::name} = "file size of " .
$nBytes/1024**2 . " MB exceeds max file size of " .
"$opt_max_file_size MB";
printf "file(%s) exceeds $opt_max_file_size MB\n",
$File::Find::name if $opt_v > 5;
return;
}
my $is_dir = is_dir($_);
my $is_bin = -B $_ ;
printf "files(%s) size=%d is_dir=%d -B=%d\n",
$File::Find::name, $nBytes, $is_dir, $is_bin if $opt_v > 5;
$is_bin = 0 if $opt_unicode and unicode_file($_);
$is_bin = 0 if $opt_read_binary_files;
return if $is_dir or $is_bin;
++$nFiles_Found;
printf "%8d files\r", $nFiles_Found
unless (!$opt_progress_rate or ($nFiles_Found % $opt_progress_rate));
push @file_list, $File::Find::name;
} # 1}}}
sub archive_files { # {{{1
# invoked by File::Find's find() Populates global variable @binary_archive
foreach my $ext (keys %Known_Binary_Archives) {
push @binary_archive, $File::Find::name
if $File::Find::name =~ m{$ext$};
}
} # 1}}}
sub is_file { # {{{1
# portable method to test if item is a file
# (-f doesn't work in ActiveState Perl on Windows)
my $item = shift @_;
return (-f $item);
# Was:
####if ($ON_WINDOWS) {
#### my $mode = (stat $item)[2];
#### $mode = 0 unless $mode;
#### if ($mode & 0100000) { return 1; }
#### else { return 0; }
####} else {
#### return (-f $item); # works on Unix, Linux, CygWin, z/OS
####}
} # 1}}}
sub is_dir { # {{{1
my $item = shift @_;
return (-d $item); # should work everywhere now (July 2017)
# Was:
##### portable method to test if item is a directory
##### (-d doesn't work in older versions of ActiveState Perl on Windows)
####if ($ON_WINDOWS) {
#### my $mode = (stat $item)[2];
#### $mode = 0 unless $mode;
#### if ($mode & 0040000) { return 1; }
#### else { return 0; }
####} else {
#### return (-d $item); # works on Unix, Linux, CygWin, z/OS
####}
} # 1}}}
sub is_excluded { # {{{1
my ($file , # in
$excluded , # in hash of excluded directories
) = @_;
my($filename, $filepath, $suffix) = fileparse($file);
foreach my $path (sort keys %{$excluded}) {
return 1 if ($filepath =~ m{^$path/}i);
}
} # 1}}}
sub classify_file { # {{{1
my ($full_file , # in
$rh_Err , # in hash of error codes
$raa_errors , # out
binmode $IN;
my $MD5 = Digest::MD5->new->addfile($IN)->hexdigest;
#print "$F, $MD5\n";
push @{$file_hash{$MD5}}, $F;
} else {
# all files treated unique
push @{$file_hash{$F}}, $F;
}
$IN->close;
}
}
# Loop over file sets having identical MD5 sums. Within
# each set, pick the file that most resembles known source
# code.
my @unique = ();
for my $md5 (sort keys %file_hash) {
my $i_best = 0;
for (my $i = 1; $i < scalar(@{$file_hash{$md5}}); $i++) {
my $F = $file_hash{$md5}[$i];
my (@nul_a, %nul_h);
my $language = classify_file($F, $rh_Err,
# don't save these errors; pointless
\@nul_a, \%nul_h);
$i_best = $i if $language ne "(unknown)";
}
# keep the best one found and identify the rest as ignored
for (my $i = 0; $i < scalar(@{$file_hash{$md5}}); $i++) {
if ($i == $i_best) {
push @unique, $file_hash{$md5}[$i_best];
} else {
$rh_ignored->{$file_hash{$md5}[$i]} = "duplicate of " .
$file_hash{$md5}[$i_best];
}
}
}
print "<- different_files(@unique)\n" if $opt_v > 2;
return @unique;
} # 1}}}
sub call_counter { # {{{1
my ($file , # in
$language , # in
$ra_Errors, # out
) = @_;
# Logic: pass the file through the following filters:
# 1. remove blank lines
# 2. remove comments using each filter defined for this language
# (example: SQL has two, remove_starts_with(--) and
# remove_c_comments() )
# 3. compute comment lines as
# total lines - blank lines - lines left over after all
# comment filters have been applied
print "-> call_counter($file, $language)\n" if $opt_v > 2;
#print "call_counter: ", Dumper(@routines), "\n";
my @lines = ();
my $ascii = "";
if (-B $file and $opt_unicode) {
# was binary so must be unicode
$/ = undef;
my $IN = new IO::File $file, "r";
my $bin_text = <$IN>;
$IN->close;
$/ = "\n";
$ascii = unicode_to_ascii( $bin_text );
@lines = split("\n", $ascii );
foreach (@lines) { $_ = "$_\n"; }
} else {
# regular text file
@lines = read_file($file);
$ascii = join('', @lines);
}
my @original_lines = @lines;
my $total_lines = scalar @lines;
print_lines($file, "Original file:", \@lines) if $opt_print_filter_stages;
@lines = rm_blanks(\@lines, $language, \%EOL_Continuation_re); # remove blank lines
my $blank_lines = $total_lines - scalar @lines;
print_lines($file, "Blank lines removed:", \@lines)
if $opt_print_filter_stages;
@lines = rm_comments(\@lines, $language, $file,
\%EOL_Continuation_re, $ra_Errors);
my $comment_lines = $total_lines - $blank_lines - scalar @lines;
if ($opt_strip_comments) {
my $stripped_file = "";
if ($opt_original_dir) {
$stripped_file = $file . ".$opt_strip_comments";
} else {
$stripped_file = basename $file . ".$opt_strip_comments";
}
write_file($stripped_file, @lines);
}
if ($opt_html and !$opt_diff) {
chomp(@original_lines); # includes blank lines, comments
chomp(@lines); # no blank lines, no comments
my (@diff_L, @diff_R, %count);
# remove blank lines to get better quality diffs; count
# blank lines separately
my @original_lines_minus_white = ();
# however must keep track of how many blank lines were removed and
# where they were removed so that the HTML display can include it
my %blank_line = ();
my $insert_line = 0;
foreach (@original_lines) {
if (/^\s*$/) {
++$count{blank}{same};
++$blank_line{ $insert_line };
} else {
++$insert_line;
push @original_lines_minus_white, $_;
}
}
array_diff( $file , # in
\@original_lines_minus_white , # in
\@lines , # in
"comment" , # in
\@diff_L, \@diff_R, , # out
$ra_Errors); # in/out
my ($title , ) = @_;
print "-> html_header\n" if $opt_v > 2;
return
'<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="GENERATOR" content="cloc http://github.com/AlDanial/cloc">
' .
"
<!-- Created by $script v$VERSION -->
<title>$title</title>
" .
'
<style TYPE="text/css">
<!--
body {
color: black;
background-color: white;
font-family: monospace
}
.whitespace {
background-color: gray;
}
.comment {
color: gray;
font-style: italic;
}
.clinenum {
color: red;
}
.linenum {
color: green;
}
-->
</style>
</head>
<body>
<pre><tt>
';
print "<- html_header\n" if $opt_v > 2;
} # 1}}}
sub html_end { # {{{1
return
'</tt></pre>
</body>
</html>
';
} # 1}}}
sub die_unknown_lang { # {{{1
my ($lang, $option_name) = @_;
die "Unknown language '$lang' used with $option_name option. " .
"The command\n $script --show-lang\n" .
"will print all recognized languages. Language names are " .
"case sensitive.\n" ;
} # 1}}}
sub unicode_file { # {{{1
my $file = shift @_;
print "-> unicode_file($file)\n" if $opt_v > 2;
return 0 if (-s $file > 2_000_000);
# don't bother trying to test binary files bigger than 2 MB
my $IN = new IO::File $file, "r";
if (!defined $IN) {
warn "Unable to read $file; ignoring.\n";
return 0;
}
my @lines = <$IN>;
$IN->close;
if (unicode_to_ascii( join('', @lines) )) {
print "<- unicode_file()\n" if $opt_v > 2;
return 1;
} else {
print "<- unicode_file()\n" if $opt_v > 2;
return 0;
}
} # 1}}}
sub unicode_to_ascii { # {{{1
my $string = shift @_;
# A trivial attempt to convert UTF-16 little or big endian
# files into ASCII. These files exhibit the following byte
# sequence:
# byte 1: 255
# byte 2: 254
# byte 3: ord of ASCII character
# byte 4: 0
# byte 3+i: ord of ASCII character
# byte 4+i: 0
# or
# byte 1: 255
# byte 2: 254
# byte 3: 0
# byte 4: ord of ASCII character
# byte 3+i: 0
# byte 4+i: ord of ASCII character
my $length = length $string;
#print "length=$length\n";
return '' if $length <= 3;
my @unicode = split(//, $string);
# check the first 100 characters for big or little endian UTF-16 encoding
my $max_peek = $length < 200 ? $length : 200;
my @view_1 = ();
for (my $i = 2; $i < $max_peek; $i += 2) { push @view_1, $unicode[$i] }
my @view_2 = ();
for (my $i = 3; $i < $max_peek; $i += 2) { push @view_2, $unicode[$i] }
my $points_1 = 0;
foreach my $C (@view_1) {
++$points_1 if (32 <= ord($C) and ord($C) <= 127) or ord($C) == 13
or ord($C) == 10
or ord($C) == 9;
}
my $points_2 = 0;
foreach my $C (@view_2) {
++$points_2 if (32 <= ord($C) and ord($C) <= 127) or ord($C) == 13
or ord($C) == 10
or ord($C) == 9;
}
#print "points 1: $points_1\n";
#print "points 2: $points_2\n";
my $offset = undef;
if ($points_1 > 90) { $offset = 2; }
elsif ($points_2 > 90) { $offset = 3; }
else { return '' } # neither big or little endian UTF-16
my @ascii = ();
for (my $i = $offset; $i < $length; $i += 2) { push @ascii, $unicode[$i]; }
return join("", @ascii);
} # 1}}}
sub uncompress_archive_cmd { # {{{1
my ($archive_file, ) = @_;
# Wrap $archive_file in single or double quotes in the system
# commands below to avoid filename chicanery (including
# spaces in the names).
print "-> uncompress_archive_cmd($archive_file)\n" if $opt_v > 2;
my $extract_cmd = "";
my $missing = "";
if ($opt_extract_with) {
( $extract_cmd = $opt_extract_with ) =~ s/>FILE</$archive_file/g;
} elsif (basename($archive_file) eq "-" and !$ON_WINDOWS) {
$extract_cmd = "cat > -";
} elsif (($archive_file =~ /\.tar\.(gz|Z)$/ or
$archive_file =~ /\.tgz$/ ) and !$ON_WINDOWS) {
if (external_utility_exists("gzip --version")) {
if (external_utility_exists("tar --version")) {
$extract_cmd = "gzip -dc '$archive_file' | tar xf -";
} else {
$missing = "tar";
}
} else {
$missing = "gzip";
}
} elsif ($archive_file =~ /\.tar\.bz2$/ and !$ON_WINDOWS) {
if (external_utility_exists("bzip2 --help")) {
if (external_utility_exists("tar --version")) {
$extract_cmd = "bzip2 -dc '$archive_file' | tar xf -";
} else {
$missing = "tar";
}
} else {
$missing = "bzip2";
}
} elsif ($archive_file =~ /\.tar\.xz$/ and !$ON_WINDOWS) {
if (external_utility_exists("unxz --version")) {
if (external_utility_exists("tar --version")) {
$extract_cmd = "unxz -dc '$archive_file' | tar xf -";
} else {
$missing = "tar";
}
} else {
$missing = "bzip2";
}
} elsif ($archive_file =~ /\.tar$/ and !$ON_WINDOWS) {
$extract_cmd = "tar xf '$archive_file'";
} elsif ($archive_file =~ /\.src\.rpm$/i and !$ON_WINDOWS) {
if (external_utility_exists("cpio --version")) {
if (external_utility_exists("rpm2cpio")) {
$extract_cmd = "rpm2cpio '$archive_file' | cpio -i";
} else {
$missing = "rpm2cpio";
}
} else {
$missing = "bzip2";
}
} elsif ($archive_file =~ /\.zip$/i and !$ON_WINDOWS) {
To count standard input, use the special filename B<-> and either
B<--stdin-name=FILE> to tell cloc the name of the file being
piped in, or B<--force-lang=LANG> to apply the LANG counter to
all input.
=over 4
=item B<--extract-with=CMD>
This option is only needed if cloc is unable to figure out how to
extract the contents of the input file(s) by itself. Use CMD to
extract binary archive files (e.g.: .tar.gz, .zip, .Z). Use the
literal 'E<gt>FILEE<lt>' as a stand-in for the actual file(s) to be
extracted. For example, to count lines of code in the input files
gcc-4.2.tar.gz perl-5.8.8.tar.gz on Unix use:
--extract-with='gzip -dc >FILE< | tar xf -
or, if you have GNU tar:
--extract-with='tar zxf >FILE<'
and on Windows, use, for example:
--extract-with="\"c:\Program Files\WinZip\WinZip32.exe\" -e -o >FILE<
=item B<--list-file=FILE>
Take the list of file and/or directory names to
process from FILE, which has one file/directory
name per line. Only exact matches are counted;
relative path names will be resolved starting from
the directory where cloc is invoked.
See also
B<--exclude-list-file>.
=item B<--vcs=VCS>
Invoke a system call to VCS to obtain a list of
files to work on. If VCS is 'git', then will
invoke 'git ls-files' to get a file list and
'git submodule status' to get a list of submodules
whose contents will be ignored. See also --git
which accepts git commit hashes and branch names.
If VCS is 'svn' then will invoke 'svn list -R'.
The primary benefit is that cloc will then skip
files explicitly excluded by the versioning tool
in question, ie, those in .gitignore or have the
svn:ignore property.
Alternatively VCS may be any system command
that generates a list of files.
Note: cloc must be in a directory which can read
the files as they are returned by VCS. cloc will
not download files from remote repositories.
'svn list -R' may refer to a remote repository
to obtain file names (and therefore may require
authentication to the remote repository), but
the files themselves must be local.
=item B<--unicode>
Check binary files to see if they contain Unicode expanded ASCII text.
This causes performance to drop noticeably.
=back
=head2 Processing Options
=over 4
=item B<--autoconf>
Count .in files (as processed by GNU autoconf) of recognized languages.
See also B<--no-autogen>.
=item B<--by-file>
Report results for every source file encountered.
=item B<--by-file-by-lang>
Report results for every source file encountered in addition to
reporting by language.
=item B<--count-and-diff SET1 SET2>
First perform direct code counts of source file(s)
of SET1 and SET2 separately, then perform a diff
of these. Inputs may be pairs of files, directories,
or archives. If --out or --report-file is given, three output
files will be created, one for each of the two counts and
one for the diff. See also B<--diff>, B<--diff-alignment>,
B<--diff-timeout>, B<--ignore-case>, B<--ignore-whitespace>.
=item B<--diff SET1 SET2>
Compute differences in code and comments between source file(s) of
SET1 and SET2. The inputs may be pairs of files, directories, or
archives. Use B<--diff-alignment> to generate a list showing
which file pairs where compared. See also
B<--count-and-diff>, B<--diff-alignment>, B<--diff-timeout>,
B<--ignore-case>, B<--ignore-whitespace>.
=item B<--diff-timeout N>
Ignore files which take more than N seconds
to process. Default is 10 seconds.
(Large files with many repeated lines can cause
Algorithm::Diff::sdiff() to take hours.)
=item B<--follow-links>
[Unix only] Follow symbolic links to directories (sym links to files
are always followed).
=item B<--force-lang=LANG[,EXT]>
Process all files that have a EXT extension with the counter for
language LANG. For example, to count all .f files with the Fortran
90 counter (which expects files to end with .f90) instead of the
specified multiple times. See also B<--force-lang>.
=item B<--sdir=DIR>
Use DIR as the scratch directory instead of letting I<File::Temp> chose
the location. Files written to this location are not removed at the
end of the run (as they are with I<File::Temp>).
=item B<--skip-uniqueness>
Skip the file uniqueness check. This will give a performance boost at
the expense of counting files with identical contents multiple times
(if such duplicates exist).
=item B<--stdin-name=FILE>
Count lines streamed via I<STDIN> as if they came from a file named FILE.
=item B<--strip-comments=EXT>
For each file processed, write to the current directory a version of
the file which has blank and commented lines removed (in-line comments
persist). The name of each stripped file is the original file name with
C<.EXT> appended to it. It is written to the current directory unless
B<--original-dir> is on.
=item B<--original-dir>
Write the stripped files the same directory as the original files.
Only effective in combination with B<--strip-comments>.
=item B<--sum-reports>
Input arguments are report files previously created with the
B<--report-file> option. Makes a cumulative set of results containing
the sum of data from the individual report files.
=item B<--processes=NUM>
[Available only on systems with a recent version
of the Parallel::ForkManager module. Not
available on Windows.] Sets the maximum number of
cores that cloc uses. The default value of 0
disables multiprocessing.
=item B<--unix>
Over-ride the operating system detection logic and run in UNIX
mode. See also B<--windows>, B<--show-os>.
=item B<--use-sloccount>
If SLOCCount is installed, use its compiled
executables c_count, java_count, pascal_count,
php_count, and xml_count instead of cloc's
counters. SLOCCount's compiled counters are
substantially faster than cloc's and may give
a performance improvement when counting projects
with large files. However, these cloc-specific
features will not be available: B<--diff>,
B<--count-and-diff>, B<--strip-comments>, B<--unicode>.
=item B<--windows>
Over-ride the operating system detection logic and run in
Microsoft Windows mode. See also B<--unix>, B<--show-os>.
=back
=head2 Filter Options
=over 4
=item B<--exclude-dir=DIR1[,DIR2 ...]>
Exclude the given comma separated directories from being scanned. For
example:
--exclude-dir=.cache,test
will skip all files that match C</.cache/> or C</test/> as part of
their path. Directories named C<.bzr>, C<.cvs>, C<.hg>, C<.git>,
and C<.svn> are always excluded.
This option only works with individual directory
names so including file path separators is not
allowed. Use B<--fullpath> and B<--not-match-d=REGEX>
to supply a regex matching multiple subdirectories.
=item B<--exclude-ext=EXT1[,EXT2 ...]>
Do not count files having the given file name extensions.
=item B<--exclude-lang=L1[,L2 ...]>
Exclude the given comma separated languages from being counted.
=item B<--exclude-list-file=FILE>
Ignore files and/or directories whose names
appear in FILE. FILE should have one file
name per line. Only exact matches are ignored;
relative path names will be resolved starting from
the directory where cloc is invoked.
See also B<--list-file>.
=item B<--fullpath>
Modifies the behavior of B<--match-f> or
B<--not-match-f> to include the file's path
in the regex, not just the file's basename.
(This does not expand each file to include its
absolute path, instead it uses as much of
the path as is passed in to cloc.)
=item B<--include-lang=L1[,L2 ...]>
Count only the given comma separated languages
L1, L2, L3, et cetera.
=item B<--match-d=REGEX>
( run in 0.727 second using v1.01-cache-2.11-cpan-cdf2f3d4e48 )