File-SOPS
view release on metacpan or search on metacpan
lib/File/SOPS/Metadata.pm view on Meta::CPAN
# be written back as `mac_only_encrypted: false`.
$hash->{mac_only_encrypted} = JSON->true if $self->mac_only_encrypted;
return $hash;
}
sub update_lastmodified {
my ($self) = @_;
$self->lastmodified(strftime('%Y-%m-%dT%H:%M:%SZ', gmtime));
return $self;
}
sub add_age_recipient {
my ($self, %args) = @_;
my $recipient = $args{recipient} // croak "recipient required";
my $enc = $args{enc} // croak "enc required";
push @{$self->age}, {
recipient => $recipient,
enc => $enc,
};
return $self;
}
sub get_age_encrypted_keys {
my ($self) = @_;
return @{$self->age};
}
###############################################################################
# The two regex rules, and the dialect they are matched in
###############################################################################
# unencrypted_regex and encrypted_regex are matched HERE with Perl and in sops
# with Go's RE2, and those are not the same dialect. Two consequences, both
# measured against sops 3.13.3 -- docs/adr/0048, k161.
#
# 1. RE2's \w \W \d \D \s \S \b \B and its POSIX classes are ASCII-only for
# every subject. Perl's are Unicode-aware for any string carrying the UTF-8
# flag -- which is every non-ASCII key our parsers produce. So an
# `unencrypted_regex: ^\w+$` is a rule under which sops ENCRYPTS `café`,
# `密`, `n٣` and `a<NBSP>b` and this library left them BARE. That is why the
# patterns are compiled /a below: it is RE2's answer for these classes, and
# it makes the answer independent of a flag ADR 0003 forbids reading
# anywhere else in this distribution.
#
# 2. A pattern RE2 cannot COMPILE is not reported by sops. It matches with the
# error discarded, so the rule silently matches NOTHING: measured,
# `--encrypted-regex '(?=f)foo'` writes every value of the document in
# PLAINTEXT at exit 0, and `--unencrypted-regex '(?=f)foo'` encrypts every
# value. Neither is a classification that can be reproduced quietly, so the
# constructs RE2 rejects are refused here instead.
# The escapes RE2 accepts, meaning by them what Perl means once the pattern is
# compiled /a. Every OTHER alphanumeric escape is refused, because RE2 either
# rejects it -- \Z \K \G \R \h \H \V \N \X \C \c \e \o \g \k \u \l \U \L are
# all "invalid escape sequence", measured through a .sops.yaml path_regex,
# which is the one place sops reports a compile error -- or reads it as
# something else:
#
# \v the vertical TAB to Go, the vertical-whitespace CLASS to Perl.
# \Q a quoted literal run to RE2, and NOTHING to Perl: \Q and \E are
# double-quotish escapes, processed when a pattern is written out in
# source and not by the regex compiler, so a pattern arriving in a
# variable -- which is every pattern here -- keeps them as the literal
# letters Q and E. Measured: `\Qa.b\E` selects the key `a.b` at sops
# and the key `Qa.bE` here.
#
# \0 covers the octal escapes (\0, \00, \017); a leading 1..9 is a
# backreference here and an error there.
my %RE2_ESCAPE = map { $_ => 1 } qw(
A B D P S W
a b d f n p r s t w x z
0
);
# The regex flags RE2 has, plus the `-` that turns one off. Perl's a, d, l, u,
# n, p and x are all "invalid or unsupported Perl syntax" there; U is the
# reverse case -- RE2 has it and Perl does not -- and fails to compile here,
# which is _rule_qr's other croak.
my %RE2_FLAG = map { $_ => 1 } qw( i m s U - );
# The escapes both dialects accept and do not mean the same thing by. These
# are the ones a whitelist cannot catch: nothing fails, the rule just selects
# different keys on the two sides.
my %RE2_MEANS_OTHER = (
v => 'the vertical TAB to Go RE2 and the vertical-whitespace CLASS to '
. 'Perl',
Q => 'a quoted literal run to Go RE2 and nothing at all to Perl',
E => 'the end of a quoted literal run to Go RE2 and nothing at all to '
. 'Perl',
);
# What in this pattern, if any, the two dialects do not agree on. Returns the
# empty list when they do, and otherwise the construct and which KIND of
# disagreement it is:
#
# 'unsupported' -- RE2 cannot compile it, so the rule matches nothing there
# 'different' -- both compile it and read it as different things
#
# The check is deliberately whitelist-shaped for escapes and for (?...) groups:
# a construct nobody here has measured is refused rather than assumed
# harmless, because the harm runs one way (a key sops encrypts, left bare).
sub _re2_divergent_construct {
my ($pattern) = @_;
return () unless defined $pattern;
my @c = split //, $pattern;
my $i = 0;
my $in_class = 0;
my $class_pos = 0; # how far into the current [...] we are; a ] at 0 is
# the literal character, in both dialects
my $quantified = 0; # the token just read was a quantifier
while ($i < @c) {
my $ch = $c[$i];
lib/File/SOPS/Metadata.pm view on Meta::CPAN
# What follows a `(?`, given the rest of the pattern from the `(`.
sub _re2_divergent_group {
my ($rest) = @_;
return () if $rest =~ /\A\(\?:/; # (?:...) non-capturing
return () if $rest =~ /\A\(\?P</; # (?P<n>..) named
return ('a lookbehind ((?<=...) or (?<!...))', 'unsupported')
if $rest =~ /\A\(\?<[=!]/;
return () if $rest =~ /\A\(\?</; # (?<n>...) named, Go 1.22+
return ('a lookahead ((?=...) or (?!...))', 'unsupported')
if $rest =~ /\A\(\?[=!]/;
return ('an atomic group ((?>...))', 'unsupported')
if $rest =~ /\A\(\?>/;
return ('an inline comment ((?#...))', 'unsupported')
if $rest =~ /\A\(\?#/;
return ('a branch reset ((?|...))', 'unsupported')
if $rest =~ /\A\(\?\|/;
return ('embedded code ((?{...}))', 'unsupported')
if $rest =~ /\A\(\?\??\{/;
return ('a backreference ((?P=name))', 'unsupported')
if $rest =~ /\A\(\?P=/;
return ('a subpattern call or recursion', 'unsupported')
if $rest =~ /\A\(\?(?:R\)|[0-9]|[&+]|-[0-9]|P>)/;
return ("a named group in Perl's (?'name'...) spelling", 'unsupported')
if $rest =~ /\A\(\?'/;
return ('a flag reset ((?^...))', 'unsupported')
if $rest =~ /\A\(\?\^/;
if ($rest =~ /\A\(\?([a-zA-Z-]*)[:)]/) {
my ($bad) = grep { !$RE2_FLAG{$_} } split //, $1;
return defined $bad ? ("the regex flag (?$bad)", 'unsupported') : ();
}
return ('a (?...) group RE2 does not have', 'unsupported');
}
{
# Everything in this block is compiled with ASCII-only character classes,
# which is what RE2 gives the pattern on the other side. It is deliberately
# /a and not /aa: measured, Go folds `k` to U+212A KELVIN SIGN and `s` to
# U+017F LONG S under (?i), which /a keeps and /aa would break. What /a
# does not reach is Perl's FULL case folding -- `(?i)^ss$` matches U+00DF
# here and not there -- which is recorded as a limit in docs/adr/0048.
use re '/a';
# sops's own answer for a pattern RE2 cannot compile: it discards the
# compile error, so the rule matches NOTHING. Compiled once, in here with
# the real patterns, because that is the answer the read path uses.
my $MATCHES_NOTHING = qr/(?!)/;
# The one place a rule pattern is looked at. Returns the matcher to use,
# the KIND of disagreement if there is one, and the refusal that goes with
# it -- because the two paths need different halves of that answer:
#
# 'unsupported' RE2 cannot compile it, so sops matches nothing. That is
# reproducible, and $MATCHES_NOTHING reproduces it, so the
# READ path is given a matcher and no refusal is raised.
# 'different' both dialects compile it and read it apart (\v, \Q, \E).
# 'perl' RE2 compiles it and we cannot ((?U) is the only measured
# one; \C, \g and \k are rejected by BOTH).
#
# The last two have no answer to reproduce, so they are refused wherever
# they are met. The write path refuses all three. See docs/adr/0051.
sub _rule_verdict {
my ($field, $pattern) = @_;
my $shown = length($pattern) > 60
? substr($pattern, 0, 57) . '...'
: $pattern;
my ($construct, $kind) = _re2_divergent_construct($pattern);
return (
$kind eq 'unsupported' ? $MATCHES_NOTHING : undef,
$kind,
"Cannot use '$shown' as the $field: it uses $construct, "
. ($kind eq 'unsupported'
? "which Go RE2 does not support. sops does not report that "
. "-- it discards the compile error, so the rule silently "
. "matches NOTHING. Measured on sops 3.13.3: an "
. "encrypted_regex it cannot compile leaves every value of "
. "the document in PLAINTEXT at exit 0, and an "
. "unencrypted_regex it cannot compile encrypts every value."
: "which both dialects accept and read DIFFERENTLY, so the "
. "same rule selects different keys in the two "
. "implementations.")
. " Rewrite the pattern in constructs both dialects agree on. "
. "See docs/adr/0048."
) if defined $construct;
my $qr = eval { qr/$pattern/ };
return ($qr, undef, undef) if defined $qr;
return (undef, 'perl',
"Cannot use '$shown' as the $field: it is not a valid Perl "
. "regular expression (" . _regex_reason($@) . "). RE2 and Perl "
. "do not accept the same patterns in either direction -- '(?U)' "
. "compiles there and not here -- so a rule sops took can still "
. "be one this side cannot match with. See docs/adr/0048.");
}
sub _rule_qr {
my ($field, $pattern) = @_;
my ($qr, undef, $refusal) = _rule_verdict($field, $pattern);
# A pattern RE2 cannot compile arrives here WITH a matcher -- the one
# that matches nothing, which is what sops matches with. Every other
# refusal is one this side cannot reproduce, and it stands wherever
# the rule is asked, read path included. docs/adr/0051.
croak $refusal unless defined $qr;
return $qr;
}
}
sub _regex_reason {
my ($error) = @_;
my $reason = defined $error ? "$error" : '';
$reason =~ s/\s+at\s+\S+\s+line\s+\d+\.?\s*\z//;
$reason =~ s/\s+/ /g;
lib/File/SOPS/Metadata.pm view on Meta::CPAN
Regular expression: a value is encrypted only when some component of its path
matches it -- see L</should_encrypt_path>.
Defaults to C<undef>. Mutually exclusive with the other rules, see
L</Encryption rules are mutually exclusive>. Matched in Go RE2's dialect and
not in Perl's; a pattern RE2 cannot compile matches nothing, as it does at
sops, and one the two dialects read differently is refused rather than matched
-- see L</The regex rules are matched in RE2's dialect>.
=head2 The regex rules are matched in RE2's dialect
B<New in 0.003, and it changes which keys get encrypted.> L</unencrypted_regex>
and L</encrypted_regex> are matched here with Perl and in sops with Go's RE2,
and those are not the same dialect. Two things follow.
B<The character classes are ASCII-only.> RE2's C<\w>, C<\W>, C<\d>, C<\D>,
C<\s>, C<\S>, C<\b>, C<\B> and its POSIX classes reach ASCII and nothing
else, for every subject. Perl's are Unicode-aware for any string carrying the
UTF-8 flag -- which is every non-ASCII key our parsers produce. Measured against
sops 3.13.3 with one ordinary F<.sops.yaml>, one C<sops -e> and no hand editing
anywhere:
unencrypted_regex: '^\w+$' key: cafE<eacute>
sops encrypts the value
File::SOPS (0.002) left it READABLE
Twenty-nine such disagreements were measured over the class escapes, the
thirteen POSIX classes and both rule fields; twenty-two of them left a secret
readable that sops encrypts. The patterns are therefore compiled C</a>, which
is RE2's answer for these classes -- and, not incidentally, it takes the UTF-8
flag out of the answer, which ADR 0003 forbids reading everywhere else in this
distribution. It is C</a> and not C</aa> because RE2's C<(?i)> B<is>
Unicode-aware: Go folds C<k> to U+212A KELVIN SIGN and C<s> to U+017F LATIN
SMALL LETTER LONG S, which C</a> keeps and C</aa> would break.
An ASCII rule over ASCII keys does not move. Measured over the 15_960 decisions
this distribution's own test suite makes -- every rule pattern in it against
every key in it -- eighteen changed, all of them a non-ASCII key under
C<^\w+$>, and none of them an ASCII one.
B<A pattern the two dialects do not agree on is refused for writing.> sops does
not report a pattern RE2 cannot compile: it matches with the compile error
discarded, so the rule silently matches nothing. Measured --
C<< --encrypted-regex '(?=f)foo' >> writes B<every value of the document in
plaintext> at exit 0, under a C<sops> section that makes it look encrypted, and
C<< --unencrypted-regex '(?=f)foo' >> encrypts every value. Neither is a
document this library will produce, so writing under one of these dies:
=over 4
=item * lookahead and lookbehind, C<(?=)>, C<(?!)>, C<< (?<=) >> and C<< (?<!) >>
=item * backreferences, C<\1> and C<< (?P=name) >>
=item * atomic groups C<< (?>...) >> and possessive quantifiers C<*+>, C<++>,
C<?+>, C<{n,m}+>
=item * the escapes RE2 has no rule for: C<\Z>, C<\K>, C<\G>, C<\R>, C<\h>,
C<\H>, C<\V>, C<\N>, C<\X>, C<\C>, C<\c>, C<\e>, C<\o>, C<\g>, C<\k>,
C<\u>, C<\l>, C<\U>, C<\L>, and C<\b> inside a character class
=item * inline comments C<< (?#...) >>, branch resets C<< (?|...) >>, embedded
code, subpattern calls and recursion, the C<< (?'name'...) >> spelling of a
named group, the C<< (?^...) >> flag reset
=item * every regex flag RE2 does not have: C<x>, C<a>, C<d>, C<l>, C<u>, C<n>
and C<p>
=item * C<\v>, C<\Q> and C<\E>, which both dialects take and read
B<differently>: C<\v> is the vertical TAB to Go and the vertical-whitespace
class to Perl, and C<\Q>/C<\E> quote a literal run for RE2 while a Perl
pattern arriving in a variable keeps them as the letters C<Q> and C<E>
(measured: C<\Qa.b\E> selects the key C<a.b> at sops and the key C<Qa.bE>
here)
=back
Everything both dialects have is taken, C<(?i)>, C<(?m)>, C<(?s)>,
C<\p{...}>, C<[[:alpha:]]>, C<[[:^alpha:]]>, C<(?PE<lt>nameE<gt>)>, C<< (?<name>) >>
and the lazy quantifiers included. C<(?U)> is B<not> among them -- RE2 has that
flag and Perl does not, which is the next paragraph but one. The verdicts were read off RE2 itself, not
guessed: a F<.sops.yaml> C<path_regex> is the one place sops reports
I<error parsing regexp> rather than discarding it.
B<Reading such a document is a different question, and it gets a different
answer.> Since 0.003 the rule decides what a leaf B<is> on the way out as well
as on the way in, so this predicate is asked on the read path too. Where RE2
cannot compile the pattern, sops's answer -- the rule matches nothing -- is
reproducible, and it is reproduced: L<File::SOPS/decrypt>, L<File::SOPS/extract>
and L<File::SOPS/decrypt_file> read a document carrying such a rule exactly as
C<sops -d> reads it, at exit 0. What stops is B<writing>:
L<File::SOPS/encrypt>, L<File::SOPS/encrypt_file>,
L<File::SOPS/encrypt_in_place>, L<File::SOPS/rotate> and L<File::SOPS/edit> all
ask L</assert_rule_regexes_agree> once, before any leaf is walked. See
F<docs/adr/0051>.
A pattern B<Perl> cannot compile is refused everywhere, read path included,
with the reason: the dialects disagree in that direction as well, and
C<(?U)fo+> is a pattern sops takes and this side cannot. It is the only such
pattern measured -- C<\C>, C<\g> and C<\k> read like company for it and are
not, because RE2 rejects all three (measured on 3.13.3 through the
C<path_regex> oracle; F<docs/adr/0048> says otherwise and is wrong there).
=head3 Limits
Two measured disagreements survive, both recorded in F<docs/adr/0048>:
=over 4
=item * B<Full case folding.> Perl's C<(?i)> folds U+00DF to C<ss> and RE2's
does not, so C<< unencrypted_regex => '(?i)^ss$' >> leaves a key C<E<szlig>> readable
here and sops encrypts it. Perl has no flag for simple-only folding.
=item * B<C<$> before a trailing newline.> Perl's C<$> is C<< (?=\n?\z) >> and
RE2's is C<\z>, so C<^foo$> matches a key C<"foo\n"> here and not there.
C<\z> is in both dialects and says exactly what RE2's C<$> says.
=back
A C<\p{...}> naming a property Perl has and Go does not -- C<\p{Word}>,
C<\p{Alpha}>, C<\p{IsAlpha}> -- is a third: RE2 rejects it, so sops's rule
matches nothing, and this side cannot enumerate Go's table to tell that apart
from C<\p{Greek}>, which both accept.
A fourth arrived with the read path: a pattern B<neither> dialect can compile,
C<fo(> being the measured one. RE2 rejects it, so sops matches nothing and
reads the document at exit 0; here it reaches the Perl compile, which also
rejects it, and there is nothing to tell that apart from C<(?U)fo+> -- a
pattern RE2 B<does> compile and this side must not guess at. It is refused, so
a document carrying it is one sops reads and this library does not.
=head2 Encryption rules are mutually exclusive
L</unencrypted_suffix>, L</encrypted_suffix>, L</unencrypted_regex> and
L</encrypted_regex> -- together with C<unencrypted_comment_regex> and
C<encrypted_comment_regex>, which this distribution does not implement but
does recognise -- select which values get encrypted, and B<at most one of them
may be set>. Constructing a Metadata with two of them dies:
Cannot use more than one of unencrypted_suffix, encrypted_suffix, ...
in the same document (got unencrypted_suffix and encrypted_regex);
sops refuses such a file outright
That is not a house rule; it is the reference implementation's, which reports
the same conflict and refuses the document before decrypting anything.
Constructing the object is the earliest point at which the conflict can be
seen, so it is where it is reported.
The consequence worth knowing is the one on L</unencrypted_suffix>: its
C<_unencrypted> default has to stand down as soon as any other rule is set,
or every configured document would carry two rules and be unreadable.
=head2 mac_only_encrypted
When true, the MAC covers only the values that are actually encrypted; when
false (the default) it covers every value in the document, encrypted or not.
Both MAC implementations honour this, and a MAC computed with it on additionally
starts from a fixed 32-byte initialization block (C<MACOnlyEncryptedInitialization>
in the Go source), so the two settings can never produce the same digest for the
( run in 0.697 second using v1.01-cache-2.11-cpan-99fa8243a42 )