File-SOPS

 view release on metacpan or  search on metacpan

docs/adr/0048-a-rule-regex-is-matched-in-re2s-dialect-or-not-at-all.md  view on Meta::CPAN

The one place sops *does* report it is a `.sops.yaml` `path_regex`, which
fails with `error parsing regexp: invalid or unsupported Perl syntax: `(?=``
at exit 1. That path was used as an **oracle**: 137 patterns were put through
it to read RE2's own compile verdict rather than infer it. What it says RE2
rejects, and Perl accepts:

- lookahead `(?=`, `(?!`; lookbehind `(?<=`, `(?<!`
- backreferences `\1`..`\9` and `(?P=name)`
- atomic groups `(?>`; possessive quantifiers `*+`, `++`, `?+`, `{n,m}+`
- the escapes `\Z \K \G \R \h \H \V \N \X \c \e \o \u \l \U \L`, `\g`, `\k`,
  and `\b` **inside a character class** (where it is BACKSPACE, not a
  boundary)
- `(?#comment)`, `(?|branch reset)`, `(?{code})`, `(?R)`, `(?1)`, `(?&n)`,
  `(?'name'...)`, `(?^flags)`
- the flags RE2 has no letter for: `x`, `a`, `d`, `l`, `u`, `n`, `p`

The escape `\C` is refused by RE2 and refused by Perl (`\C no longer
supported in regex`), so it is not a divergence and belongs in the default
"both refuse" class rather than in either list.

docs/adr/0051-a-rule-re2-cannot-compile-matches-nothing-on-the-read-path-and-is-refused-on-the-write-path.md  view on Meta::CPAN

`unsupported` only ever meant "refuse" and is not harmless once it means "sops
matches nothing".

RE2's own verdict was read off the `.sops.yaml` `path_regex` oracle — the one
place sops reports `error parsing regexp` instead of discarding it — for
**every escape letter** and for every `(?...)` form the scanner names:

| | verdict |
|---|---|
| escapes the scanner accepts (`\A \B \D \P \S \W \a \b \d \f \n \p \r \s \t \w \x \z \0`) | RE2 **compiles** all of them |
| escapes the scanner calls `unsupported` (`\C \E \G \H \I \J \K \L \M \N \O \R \T \U \V \X \Y \Z \c \e \g \h \i \j \k \l \m \o \q \u \y`, `\1`, `\8`, `\b` in a class) | RE2 **rejects** all of them |
| `(?=` `(?!` `(?<=` `(?<!` `(?>` `*+` `(?#` `(?\|` `(?R` `(?1` `(?P=` `(?'n'` `(?^` `(?x` `(?a` | RE2 **rejects** all of them |
| `(?i)` `(?P<n>)` `(?<n>)` `(?i-s:)` `\p{L}` `\P{L}` `\x{41}` `\v` `\Qa.b\E` | RE2 **compiles** all of them |

**Every construct the scanner labels `unsupported` is measurably one RE2
refuses, and every construct RE2 accepts passes the scanner.** The label is
accurate, not merely conservative, so treating it as "sops matches nothing"
reproduces sops rather than guessing at it.

### A correction to ADR 0048

ADR 0048 says, of the other direction: *"what RE2 accepts and Perl rejects:
`(?U)`, `\C`, `\g`, `\k`"*. Measured through the same oracle:

| pattern | RE2 |
|---|---|
| `(?U)fo+` | **compiles** |
| `f\Coo` | rejects — `invalid escape sequence: \C` |
| `(f)\g{1}`, `(f)\g1` | rejects — `invalid escape sequence: \g` |
| `(?<n>f)\k<n>`, `(?<n>f)\k'n'` | rejects — `invalid escape sequence: \k` |

`(?U)` is the **only** measured pattern that RE2 compiles and Perl does not.
The other three are rejected by both, which puts them in the lenient half, not
beside `(?U)`. ADR 0048's list is wrong there and is corrected in the POD;
k175 carries the fix to the ADR itself.

(A lone `\E` is also RE2-rejected — `\Q..\E` compiles as a pair. It stays
refused regardless, as a `different` construct.)

docs/adr/0051-a-rule-re2-cannot-compile-matches-nothing-on-the-read-path-and-is-refused-on-the-write-path.md  view on Meta::CPAN

rejected it: it would refuse to *read* a document carrying such a rule, which
is the whole defect this ADR exists to remove. It is also what k166
explicitly ruled out.

## Notes

Measured against sops 3.13.3 (`/tmp/sops`) with `Crypt::Age` on this machine,
on 2026-08-22: the `(?=foo)` document through `sops -e`, `sops -d`,
`decrypt_file` and `extract` before any change; both rule fields through
`sops -e`/`sops -d`; 55 escape letters and 19 `(?...)` forms through the
`.sops.yaml` `path_regex` compile oracle; `(?U)`, `\C`, `\g{1}`, `\g1`,
`\k<n>`, `\k'n'`, `\Qa.b\E`, lone `\Q`, lone `\E` and `fo(` through the same
oracle in their realistic spellings. All fixtures are invented values; the age
keypair was generated for the run.

`t/66-a-rule-re2-cannot-compile-is-read-the-way-sops-reads-it.t` fails **27
assertions** in 6 of its 11 subtests against the tree before this change, with
the interop section executed against the binary rather than skipped.

lib/File/SOPS/Metadata.pm  view on Meta::CPAN

#
# 2. A pattern RE2 cannot COMPILE is not reported by sops. It matches with the
#    error discarded, so the rule silently matches NOTHING: measured,
#    `--encrypted-regex '(?=f)foo'` writes every value of the document in
#    PLAINTEXT at exit 0, and `--unencrypted-regex '(?=f)foo'` encrypts every
#    value. Neither is a classification that can be reproduced quietly, so the
#    constructs RE2 rejects are refused here instead.

# The escapes RE2 accepts, meaning by them what Perl means once the pattern is
# compiled /a. Every OTHER alphanumeric escape is refused, because RE2 either
# rejects it -- \Z \K \G \R \h \H \V \N \X \C \c \e \o \g \k \u \l \U \L are
# all "invalid escape sequence", measured through a .sops.yaml path_regex,
# which is the one place sops reports a compile error -- or reads it as
# something else:
#
#   \v   the vertical TAB to Go, the vertical-whitespace CLASS to Perl.
#   \Q   a quoted literal run to RE2, and NOTHING to Perl: \Q and \E are
#        double-quotish escapes, processed when a pattern is written out in
#        source and not by the regex compiler, so a pattern arriving in a
#        variable -- which is every pattern here -- keeps them as the literal
#        letters Q and E. Measured: `\Qa.b\E` selects the key `a.b` at sops

lib/File/SOPS/Metadata.pm  view on Meta::CPAN

    # The one place a rule pattern is looked at. Returns the matcher to use,
    # the KIND of disagreement if there is one, and the refusal that goes with
    # it -- because the two paths need different halves of that answer:
    #
    #   'unsupported'  RE2 cannot compile it, so sops matches nothing. That is
    #                  reproducible, and $MATCHES_NOTHING reproduces it, so the
    #                  READ path is given a matcher and no refusal is raised.
    #   'different'    both dialects compile it and read it apart (\v, \Q, \E).
    #   'perl'         RE2 compiles it and we cannot ((?U) is the only measured
    #                  one; \C, \g and \k are rejected by BOTH).
    #
    # The last two have no answer to reproduce, so they are refused wherever
    # they are met. The write path refuses all three. See docs/adr/0051.
    sub _rule_verdict {
        my ($field, $pattern) = @_;

        my $shown = length($pattern) > 60
            ? substr($pattern, 0, 57) . '...'
            : $pattern;

lib/File/SOPS/Metadata.pm  view on Meta::CPAN

=over 4

=item * lookahead and lookbehind, C<(?=)>, C<(?!)>, C<< (?<=) >> and C<< (?<!) >>

=item * backreferences, C<\1> and C<< (?P=name) >>

=item * atomic groups C<< (?>...) >> and possessive quantifiers C<*+>, C<++>,
C<?+>, C<{n,m}+>

=item * the escapes RE2 has no rule for: C<\Z>, C<\K>, C<\G>, C<\R>, C<\h>,
C<\H>, C<\V>, C<\N>, C<\X>, C<\C>, C<\c>, C<\e>, C<\o>, C<\g>, C<\k>,
C<\u>, C<\l>, C<\U>, C<\L>, and C<\b> inside a character class

=item * inline comments C<< (?#...) >>, branch resets C<< (?|...) >>, embedded
code, subpattern calls and recursion, the C<< (?'name'...) >> spelling of a
named group, the C<< (?^...) >> flag reset

=item * every regex flag RE2 does not have: C<x>, C<a>, C<d>, C<l>, C<u>, C<n>
and C<p>

=item * C<\v>, C<\Q> and C<\E>, which both dialects take and read

lib/File/SOPS/Metadata.pm  view on Meta::CPAN

and L<File::SOPS/decrypt_file> read a document carrying such a rule exactly as
C<sops -d> reads it, at exit 0. What stops is B<writing>:
L<File::SOPS/encrypt>, L<File::SOPS/encrypt_file>,
L<File::SOPS/encrypt_in_place>, L<File::SOPS/rotate> and L<File::SOPS/edit> all
ask L</assert_rule_regexes_agree> once, before any leaf is walked. See
F<docs/adr/0051>.

A pattern B<Perl> cannot compile is refused everywhere, read path included,
with the reason: the dialects disagree in that direction as well, and
C<(?U)fo+> is a pattern sops takes and this side cannot. It is the only such
pattern measured -- C<\C>, C<\g> and C<\k> read like company for it and are
not, because RE2 rejects all three (measured on 3.13.3 through the
C<path_regex> oracle; F<docs/adr/0048> says otherwise and is wrong there).

=head3 Limits

Two measured disagreements survive, both recorded in F<docs/adr/0048>:

=over 4

=item * B<Full case folding.> Perl's C<(?i)> folds U+00DF to C<ss> and RE2's

t/66-a-rule-re2-cannot-compile-is-read-the-way-sops-reads-it.t  view on Meta::CPAN

        my $err = exception {
            File::SOPS->decrypt(encrypted => $doc, identities => [ $SECRET ]);
        };

        like $err, qr/read DIFFERENTLY/,
            "$pattern: decrypt still refuses it";
    }
};

subtest 'a pattern PERL cannot compile is still refused on read' => sub {
    # (?U) is the only pattern measured to be RE2-OK and Perl-reject. \C, \g
    # and \k are named beside it in docs/adr/0048 and do not belong there:
    # measured on 3.13.3 through the path_regex oracle, RE2 rejects all three
    # (`invalid escape sequence`), so they are the lenient kind. k175.
    for my $pattern ('(?U)fo+') {
        my $doc = document_with_rule(unencrypted_regex => $pattern);

        my $err = exception {
            File::SOPS->decrypt(encrypted => $doc, identities => [ $SECRET ]);
        };