AWS-Signature-V4

 view release on metacpan or  search on metacpan

README  view on Meta::CPAN

    an unknown name, or one that does not apply (e.g. expires, which is for
    "presign"), is an error, and an undef value is the same as a missing
    argument. Named arguments:

    method, url

      the HTTP method and the full URL (mandatory). The method must be an
      HTTP token, and it is uppercased. The host, lowercased and without
      default ports, provides the host header unless it is given; user
      information (user@host) is an error, it is not stripped. The URL can
      also be just a path, starting with /, if a Host header is given. See
      "DESCRIPTION" and "SECURITY CONSIDERATIONS" for what the URL can
      hold; in particular, the query cannot hold +, which some read as a
      space and others as a plus sign: write %20 or %2B;

    headers

      the headers of the request, as a hash reference or an array reference
      of pairs. A value is a byte string, or an array reference of them,
      and repeated names are joined with a comma; undef values are skipped.
      Names must be HTTP tokens (RFC 9110: letters, digits and

TODO.md  view on Meta::CPAN


## Baseline code review

No full code review has ever been done: the first one covered only the
`main...moo` diff (Moo and Ouch), and the security review only looked for
ways to abuse the code, not for correctness. To stay within usage limits,
the review is split in chunks, one per session, in order of priority.

For each chunk:

- start a new session (or `/clear`) first;
- run `/code-review <level>`, naming the files in scope and saying that
  other files may be read for context but only the chunk is reported on,
  e.g. `/code-review high` "review lib/AWS/Signature/V4/Chunker.pm,
  Checksum.pm and their tests t/streaming.t, t/trailer.t; read other files
  only for context";
- have the findings saved to `tmp/review-<n>-<name>.md` (`tmp/` is
  git-ignored), so that nothing is lost between sessions;
- fix after each chunk, or collect all the findings first and fix them
  together.

TODO.md  view on Meta::CPAN

before a chunk when useful:

- Sonnet 5 is enough for chunks 4 and 5 (wording, error checks,
  comments), at a lower cost;
- Fable 5.1 (twice the cost of Opus 5) only if a fix in chunk 1 turns out
  to be hard and Opus's first attempt does not hold up;
- not Haiku: too easy to miss something in signing code.

For each chunk:

- start a new session (or `/clear`) first;
- ask to fix the findings in `tmp/review-<n>-<name>.md`, updating the
  POD, tests and examples affected by each fix;
- run the test suite and make sure it passes. The dependencies are
  installed in `local/` (by `carton`), so they have to be added to the
  include path: `prove -l -Ilocal/lib/perl5 t/`, or `carton exec prove -l
  t/`. Plain `prove -l t/` dies on `Can't locate Moo.pm`;
- make one commit per chunk.

- [x] **1. Signing core** (Opus 5, Fable 5.1 if needed):
      `tmp/review-1-signing-core.md`.

lib/AWS/Signature/V4.pm  view on Meta::CPAN

   return $v;
}

# (host, path, query) of a url; the host is lowercased and the port is
# removed if it is the default one, as user agents do before sending it
sub _split_url ($url) {
   # every part is optional, so this always matches
   my ($scheme, $auth, $path, $query) =
      $url =~ m{\A (?: ([A-Za-z][A-Za-z0-9+.-]*) :// ([^/?\#]*) )?
                   ([^?\#]*) (?: \? ([^\#]*) )? }x;
   fail 400, 'url path must start with "/": give an absolute url, or a path with a Host header'
      if length $path && $path !~ m{\A/};
   fail 400, 'url query must not contain "+": write %20 for a space, %2B for a plus sign'
      if defined $query && $query =~ m{\+};
   return (undef, $path, $query) unless defined $auth && length $auth;

   fail 400, 'url must not have user information (user@host)' if $auth =~ m{@};
   my ($host, $port) = $auth =~ m{\A
         (\[[0-9A-Fa-f:.]+\] | [A-Za-z0-9._~-]+)  # host
         (?: : ([0-9]*) )?   # optional port
      \z}x or fail 400, 'invalid url: ' . shown($auth);

lib/AWS/Signature/V4.pm  view on Meta::CPAN

   $total += $full * ($chunk->($chunk_size) + $chunk_size);
   my $rest = $decoded - $full * $chunk_size;
   $total += $chunk->($rest) + $rest if $rest;

   # final chunk: "0[;chunk-signature=..]\r\n", trailers, [signature line,] "\r\n"
   $total += 1 + ($signed ? 17 + 64 : 0) + 2 + $trailer + 2;
   $total += length('x-amz-trailer-signature') + 1 + 64 + 2 if $signed && $trailer;
   return $total;
}

# the path is empty or starts with "/", see _split_url
sub _canonical_path ($self, $path) {
   $path = '/' unless length $path;
   my $trailing = $path =~ m{/\z} && $path ne '/';
   my @parts = split m{/}, $path, -1;
   shift @parts;    # leading empty item, before the first /
   pop @parts if @parts && $parts[-1] eq '';    # trailing, re-added below

   if ($self->normalize_path) {
      my @out;
      for my $p (@parts) {

lib/AWS/Signature/V4.pod  view on Meta::CPAN

missing argument. Named arguments:

=over

=item C<method>, C<url>

the HTTP method and the full URL (mandatory). The method must be an HTTP
token, and it is uppercased. The host, lowercased and without default
ports, provides the C<host> header unless it is given; user information
(C<user@host>) is an error, it is not stripped.
The URL can also be just a path, starting with C</>, if a C<Host> header
is given. See L</DESCRIPTION> and L</SECURITY CONSIDERATIONS> for what
the URL can hold; in particular, the query cannot hold C<+>, which some
read as a space and others as a plus sign: write C<%20> or C<%2B>;

=item C<headers>

the headers of the request, as a hash reference or an array reference of
pairs. A value is a byte string, or an array reference of them, and
repeated names are joined with a comma; C<undef> values are skipped.
Names must be HTTP tokens (RFC 9110: letters, digits and

lib/AWS/Signature/V4/X509.pm  view on Meta::CPAN

}

# a certificate is a DER SEQUENCE that fits exactly in the data
sub _check_der ($der) {
   my ($tag, undef, $end) = _tlv($der, 0);
   fail 400, 'not a DER-encoded certificate'
      unless $tag == 0x30 && $end == length $der;
   return $der;
}

# read one DER TLV at $off => (tag, content_start, content_end)
sub _tlv ($der, $off) {
   my $max = length $der;
   $off + 2 <= $max or fail 400, 'truncated certificate';
   my $tag = ord substr $der, $off++, 1;
   my $len = ord substr $der, $off++, 1;
   if ($len & 0x80) {    # long form, DER wants it minimal and definite
      my $n = $len & 0x7F;
      fail 400, 'not a DER-encoded certificate' if $n == 0 || $n > 4;
      $off + $n <= $max or fail 400, 'truncated certificate';
      $len = 0;

lib/AWS/Signature/V4/X509.pm  view on Meta::CPAN

   ($tag, $s, $e) = _tlv($der, $e) if $tag == 0xA0;    # skip explicit version
   fail 400, 'cannot find certificate serial number' unless $tag == 0x02;
   return Math::BigInt->from_hex(unpack 'H*', substr $der, $s, $e - $s)
      ->bstr;   # DER integers here are positive; leading 0x00 is harmless
}

# The error of CryptX, without the location and the backtrace that Carp adds
# when verbose (the arguments in it may hold the password), escaped as caller
# input because it can hold the file name.
sub _clean_error ($e) {
   $e =~ s{\n\t.*}{}s;    # the backtrace lines start with a tab
   $e =~ s{\s+at \S+ line \d+\.?\s*\z}{};
   return shown($e =~ s{\A\s+|\s+\z}{}gr);
}

# $key is a file name or a scalar reference to the PEM text
sub _cryptx_signer ($algo, $key, $password = undef) {
   my $class = $algo eq 'RSA' ? 'Crypt::PK::RSA' : 'Crypt::PK::ECC';
   eval "require $class; 1" or fail 500, "cannot load $class";
   my @args = ($key, defined $password ? ($password) : ());
   my $pk = eval {

t/canonical.t  view on Meta::CPAN

subtest 'S3 under other names' => sub {
   for my $name (qw< s3-object-lambda s3-outposts s3express >) {
      my $t = AWS::Signature::V4->new(service => $name, region => 'r', credentials => {%cred});
      my ($path, undef, $r) = canon($t, 'https://h/a%20b//c/../d');
      is $path, '/a%20b//c/../d', "$name: path verbatim";
      is $r->{headers}{'x-amz-content-sha256'}, sha256_hex(''), "$name: payload header";
      like $t->presign(url => 'https://h/', time => 0)->{canonical_request},
         qr/\nUNSIGNED-PAYLOAD\z/, "$name: presign with UNSIGNED-PAYLOAD";
   }
   my $t = AWS::Signature::V4->new(service => 's3x', region => 'r', credentials => {%cred});
   is path_of($t, 'https://h/a/../b'), '/b', 'but not any name starting with s3';
};

subtest 'body as scalar reference' => sub {
   my $body = 'some payload';
   my $plain = req($std, 'https://h/', body => $body);
   my $byref = req($std, 'https://h/', body => \$body);
   is $byref->{signature}, $plain->{signature}, 'same signature by reference';
   is $body, 'some payload', 'referenced body untouched';

   my $big = 'x' x 10_000_000;

t/canonical.t  view on Meta::CPAN

   );
   for my $name (sort keys %bad) {
      is dies { $std->sign(method => 'GET', url => 'https://h/', $bad{$name}->@*) }, bad(), $name;
   }
   is req($std, 'https://h/', time => 1440938160.75)->{headers}{'x-amz-date'},
      '20150830T123600Z', 'fractional time is fine';
};

subtest 'url path and query' => sub {
   for my $url ('foo/bar', 'folder/file.txt', 'x') {
      is dies { canon($std, $url, headers => {Host => 'h'}) }, bad(qr{start with "/"}),
         "relative path $url refused";
   }
   is path_of($std, 'https://h?a=1'), '/', 'empty path is fine';
   is req($std, '/a/b', headers => {Host => 'h'})->{headers}{host}, 'h', 'absolute path with Host';
   is dies { canon($std, 'https://h/?prefix=a+b') }, bad(qr/\+/), 'plus sign in the query refused';
   is query_of($std, 'https://h/?prefix=a%2Bb%20c'), 'prefix=a%2Bb%20c', 'encoded is fine';
   is path_of($std, 'https://h/a+b'), '/a%252Bb', 'plus sign in the path is a plus sign';
   for my $url ('https://h/public/%2e%2e/admin', 'https://h/a/%2E/b', 'https://h/a/.%2e/b') {
      is dies { canon($std, $url) }, bad(qr/dot segment/), "encoded dot segment: $url";
   }

t/presign.t  view on Meta::CPAN

   'already signed';
is dies { $iam->presign(url => 'https://h/ሴ') }, bad(qr/ASCII/), 'non-ASCII url';
is dies { $iam->presign }, bad(qr/"url"/), 'no url';
is dies { $iam->presign(url => 'https://h/', expires_in => 60) },
   bad(qr/unsupported for presign: "expires_in"/), 'misspelled expires';
for my $name (qw< checksum trailers decoded_content_length streaming >) {
   is dies { $iam->presign(url => 'https://h/', $name => 1) },
      bad(qr/unsupported for presign: "$name"/), "$name refused";
}
is dies { $s3->presign(url => 'folder/file.txt', headers => {Host => 'b.s3.amazonaws.com'}) },
   bad(qr{start with "/"}), 'relative path';
is dies { $iam->presign(url => 'https://h/', payload_hash => "x\r\nX: 1") },
   bad(qr/payload_hash/), 'invalid payload_hash';
is dies { $iam->presign(url => 'https://h/', method => "GET\n/x") }, bad(qr/method/),
   'invalid method';
is $iam->presign({url => 'https://h/', time => 0})->{signature},
   $iam->presign(url => 'https://h/', time => 0)->{signature}, 'hash reference of arguments';

# X.509 variant, with a stub signer
my $cert = "-----BEGIN CERTIFICATE-----\n" . encode_base64("\x30\x03\x02\x01\x07")
   . "-----END CERTIFICATE-----\n";

t/trailer.t  view on Meta::CPAN

sub bad ($re) {
   object { prop blessed => 'Ouch'; call code => 400; call message => match $re };
}

# the reference for the signed chunks and trailers
my $key = 'AWS4' . 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';
$key = hmac_sha256($_, $key) for '20130524', 'us-east-1', 's3', 'aws4_request';
my $scope = '20130524/us-east-1/s3/aws4_request';
my $sts = sub ($kind, @parts) { join "\n", "AWS4-HMAC-SHA256-$kind", '20130524T000000Z', $scope, @parts };

sub start ($signer = $s3, %opts) {
   $signer->sign(method => 'PUT', url => 'https://h/k', time => 1369353600,
      decoded_content_length => 9, %opts);
}

# --- checksums, fed in two pieces
my %expect = (    # of "123456789"
   crc32  => encode_base64(pack('N', 0xCBF43926), ''),
   crc32c => encode_base64(pack('N', 0xE3069283), ''),
   sha1   => encode_base64(sha1('123456789'), ''),
   sha256 => encode_base64(sha256('123456789'), ''),
);
for my $algo (sort keys %expect) {
   my $ck = start($x, streaming => 'unsigned', checksum => $algo)->{chunker};
   my $body = $ck->chunk('1234') . $ck->chunk('56789') . $ck->finish;
   is $body, "4\r\n1234\r\n5\r\n56789\r\n0\r\nx-amz-checksum-$algo:$expect{$algo}\r\n\r\n",
      "unsigned, $algo, byte for byte";
}

# --- crc32c against a bitwise reference, on longer data and in many chunks
sub crc32c_ref ($data) {
   my $crc = 0xFFFFFFFF;
   for my $byte (unpack 'C*', $data) {
      $crc ^= $byte;
      $crc = ($crc & 1) ? (($crc >> 1) ^ 0x82F63B78) : ($crc >> 1) for 1 .. 8;
   }
   return encode_base64(pack('N', $crc ^ 0xFFFFFFFF), '');
}
my $long = join '', map { chr(($_ * 7) % 256) } 1 .. 5000;
my $ck = start($x, streaming => 'unsigned', checksum => 'crc32c',
   decoded_content_length => length $long)->{chunker};
$ck->chunk(substr($long, $_ * 1000, 1000)) for 0 .. 4;
like $ck->finish, qr/x-amz-checksum-crc32c:\Q${\ crc32c_ref($long)}\E\r\n/, 'crc32c on longer data';

# the test here will exercise whatever is available; we don't explicitly
# test the pure-perl version shipped with the module in case
# String::CRC32 is available. FIXME figure out how to do this withouth
# going back to using package variables.
for my $size (1 .. 9, 65535 .. 65537, 70001) {
   my $sum = AWS::Signature::V4::Checksum->new('crc32c');

t/trailer.t  view on Meta::CPAN

   my $upgraded = $raw;
   utf8::upgrade($upgraded);
   is $sum->add(\$upgraded)->base64,
      AWS::Signature::V4::Checksum->new($algo)->add(\$raw)->base64,
      "$algo: upgraded byte string sums as bytes";
   # the digest modules may clear the UTF-8 flag of the scalar they are
   # given, but the string the caller sees must be the same one
   is $upgraded, $raw, "$algo: and the input string still holds those bytes";
}
{
   my $ck = start($x, streaming => 'unsigned', checksum => 'crc32c')->{chunker};
   is dies { $ck->chunk("\x{263a}") }, bad(qr/chunk must be a byte string/), 'unsigned crc32c chunker: wide character refused';
   is dies { $ck->chunk(['123456789']) }, bad(qr/reference to one/), 'chunk of an array reference';
   is dies { $ck->chunk(bless \my $o, 'Foo') }, bad(qr/reference to one/), 'chunk of an object';
   is $ck->chunk(\'123456789'), "9\r\n123456789\r\n", 'and a scalar reference is fine';
   like $ck->finish, qr/x-amz-checksum-crc32c:\Q$expect{crc32c}\E\r\n/, 'the refused pieces were not counted';
}

# --- request side
my $r = start($x, streaming => 'unsigned', checksum => 'crc32c');
is $r->{headers}{'x-amz-content-sha256'}, 'STREAMING-UNSIGNED-PAYLOAD-TRAILER', 'unsigned marker';
is $r->{headers}{'x-amz-trailer'}, 'x-amz-checksum-crc32c', 'x-amz-trailer';
like $r->{signed_headers}, qr/x-amz-trailer/, 'and it is signed';
is $r->{headers}{'x-amz-decoded-content-length'}, 9, 'decoded length';
is $r->{headers}{'content-encoding'}, 'aws-chunked', 'content encoding';
ok !defined $r->{chunker}->_mac, 'unsigned chunker has no key';

$r = start($s3, streaming => 'signed', checksum => 'sha256');
ok !(grep { defined && !ref && ($_ eq $key || $_ eq unpack 'H*', $key) } values $r->{chunker}->%*),
   'signed chunker: the key is not in the object';
is $r->{headers}{'x-amz-content-sha256'}, 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER', 'signed trailer marker';
$r = start($s3, streaming => 1);
is $r->{headers}{'x-amz-content-sha256'}, 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'no trailer: plain marker';
ok !exists $r->{headers}{'x-amz-trailer'}, 'and no x-amz-trailer';

# --- signed chunks and trailer, against a straightforward reimplementation
{
   my $r = start($s3, streaming => 'signed', checksum => 'crc32');
   my $prev = $r->{signature};

   my $sig1 = hmac_sha256_hex($sts->('PAYLOAD', $prev, sha256_hex(''), sha256_hex('123456789')), $key);
   my $sig2 = hmac_sha256_hex($sts->('PAYLOAD', $sig1, sha256_hex(''), sha256_hex('')), $key);
   my $line = "x-amz-checksum-crc32:$expect{crc32}";
   my $tsig = hmac_sha256_hex($sts->('TRAILER', $sig2, sha256_hex("$line\n")), $key);

   my $ck = $r->{chunker};
   is $ck->chunk('123456789'), "9;chunk-signature=$sig1\r\n123456789\r\n", 'signed chunk';
   is $ck->finish,
      "0;chunk-signature=$sig2\r\n$line\r\nx-amz-trailer-signature:$tsig\r\n\r\n",
      'final chunk, trailer and trailer signature';
}

# --- caller-supplied trailers
{
   my $r = start($x, streaming => 'unsigned', trailers => ['X-Amz-Checksum-Crc64nvme']);
   is $r->{headers}{'x-amz-trailer'}, 'x-amz-checksum-crc64nvme', 'declared, lowercased';
   my $ck = $r->{chunker};
   $ck->chunk('123456789');
   is $ck->finish('x-amz-checksum-crc64nvme' => 'AAAAAAAAAAA='),
      "0\r\nx-amz-checksum-crc64nvme:AAAAAAAAAAA=\r\n\r\n", 'value given at finish';

   $r = start($s3, streaming => 1, checksum => 'sha1', trailers => ['x-foo', 'x-bar']);
   is $r->{headers}{'x-amz-trailer'}, 'x-amz-checksum-sha1,x-foo,x-bar', 'built-in first, then declared';
   $ck = $r->{chunker};
   $ck->chunk('123456789');
   my $sig1 = hmac_sha256_hex($sts->('PAYLOAD', $r->{signature}, sha256_hex(''), sha256_hex('123456789')), $key);
   my $sig2 = hmac_sha256_hex($sts->('PAYLOAD', $sig1, sha256_hex(''), sha256_hex('')), $key);
   my @lines = ("x-amz-checksum-sha1:$expect{sha1}", 'x-foo:1', 'x-bar:2');
   my $tsig = hmac_sha256_hex($sts->('TRAILER', $sig2, sha256_hex(join '', map { "$_\n" } @lines)), $key);
   is $ck->finish('x-foo' => '1', 'x-bar' => '2'),
      "0;chunk-signature=$sig2\r\n" . join('', map { "$_\r\n" } @lines)
      . "x-amz-trailer-signature:$tsig\r\n\r\n",
      'all trailers, in order, signed';

   my $mk = sub ($signer = $x) {
      my $c = start($signer, streaming => $signer == $x ? 'unsigned' : 1, checksum => 'crc32',
         trailers => ['X-Foo'])->{chunker};
      $c->chunk('123456789');
      $c;
   };
   is dies { $mk->()->finish }, bad(qr/missing value for trailer 'x-foo'/), 'missing value';
   is dies { $mk->()->finish('x-foo' => 1, 'x-nope' => 2) }, bad(qr/undeclared trailers: x-nope/),
      'undeclared value';
   is dies { $mk->()->finish('x-foo' => 1, 'x-amz-checksum-crc32' => 'x') },
      bad(qr/'x-amz-checksum-crc32' is computed/), 'computed value given';
   is dies { $mk->()->finish('x-foo' => 1, 'X-FOO' => 2) }, bad(qr/'x-foo' given twice/),

t/trailer.t  view on Meta::CPAN

      }
   }
   my $upgraded = "\xe9";
   utf8::upgrade($upgraded);
   my $body = $mk->()->finish('x-foo' => $upgraded);
   ok !utf8::is_utf8($body), 'a value stored as UTF-8 gives a byte string';
   is $body, "0\r\nx-amz-checksum-crc32:$expect{crc32}\r\nx-foo:\xe9\r\n\r\n", '... with the same bytes';
}

# --- configuration errors
ok dies { start($s3, streaming => 'unsigned') }, 'unsigned without trailers';
ok dies { start($s3, streaming => 1, checksum => 'md5') }, 'unknown checksum';
ok dies { start($s3, streaming => 1, checksum => 'crc32', trailers => ['x-amz-checksum-crc32']) }, 'same trailer twice';
ok dies { start($s3, streaming => 1, trailers => ['bad name']) }, 'invalid trailer name';
ok dies { start($s3, checksum => 'crc32') }, 'checksum without streaming';
ok dies { start($s3, trailers => ['x-foo']) }, 'trailers without streaming';
ok dies { start($s3, streaming => 'sideways') }, 'unknown streaming mode';
ok dies { start($x, streaming => 'signed', checksum => 'crc32') }, 'signed chunks need credentials';
ok lives { start($x, streaming => 'unsigned', checksum => 'crc32') }, 'unsigned works with x509';

# --- encoded_length equals the real length
my @cases = (
   [signed => 1, {}],
   [signed => 1, {checksum => 'crc32c'}, {streaming => 1, checksum => 'crc32c'}],
   [signed => 1, {trailers => {'x-amz-checksum-crc64nvme' => 12}}, {streaming => 1, trailers => ['x-amz-checksum-crc64nvme']}, {'x-amz-checksum-crc64nvme' => 'A' x 12}],
   [signed => 0, {checksum => 'sha256'}, {streaming => 'unsigned', checksum => 'sha256'}],
   [signed => 0, {checksum => 'sha1', trailers => {'x-foo' => 3}}, {streaming => 'unsigned', checksum => 'sha1', trailers => ['x-foo']}, {'x-foo' => 'abc'}],
);
for my $case (@cases) {
   my ($label, $signed, $opts, $sign, $values) = @$case;
   $sign //= {streaming => 1};
   for my $size (0, 1, 15, 16, 17, 255, 256, 300, 5000) {
      for my $chunk (16, 256, 1000) {
         my $c = start($signed ? $s3 : $x, %$sign, decoded_content_length => $size)->{chunker};
         my $data = 'z' x $size;
         my $body = '';
         for (my $off = 0; $off < $size; $off += $chunk) {
            $body .= $c->chunk(substr($data, $off, $chunk));
         }
         $body .= $c->finish(%{$values // {}});
         is length($body), AWS::Signature::V4->encoded_length($size, $chunk, signed => $signed, %$opts),
            "encoded_length: " . join(',', $signed ? 'signed' : 'unsigned', sort keys %$opts) . " size $size chunk $chunk";
      }
   }



( run in 1.488 second using v1.01-cache-2.11-cpan-234fa220139 )