AWS-Signature-V4
view release on metacpan or search on metacpan
meaning of the canonical request. The same check applies to the
values that sign adds, e.g. the session token, and to the ones that
"presign" puts in the query string instead.
An X-Amz-Content-Sha256 header is taken as the payload hash; if the
payload hash also comes from the arguments below, the two must agree;
body, body_fh
the payload, as a byte string, a reference to one (which avoids
copying large data around) or an open filehandle. The filehandle is
hashed from its current position to its end, without loading it in
memory, and put back where it was: so it must be seekable (a file,
not a pipe or a socket) and binary, without layers like :encoding or
:crlf that change the bytes read. Only one of them can be used;
payload_hash, unsigned_payload
skip the calculation and use the provided SHA-256 in hex, or
UNSIGNED-PAYLOAD. They take precedence over body and body_fh.
payload_hash can also be a STREAMING-* marker, anything else is an
only when streaming: see below.
Chunked and streaming uploads
streaming enables the aws-chunked encoding used for uploads to S3,
where the body is sent in chunks that are signed as they go. It can be
1 or signed (each chunk is signed, credentials variant only) or
unsigned (no chunk signatures, only the request headers are signed,
also OK with X.509). The decoded_content_length, i.e. the size of the
data, is mandatory. sign sets the payload hash to
STREAMING-AWS4-HMAC-SHA256-PAYLOAD (or its variants below), adds
x-amz-decoded-content-length and aws-chunked to Content-Encoding (after
any other encoding, e.g. gzip,aws-chunked, as it is the one applied
last: S3 takes that token off the end and stores what is left, here
gzip), all signed; the Content-Length to provide is the size of the
encoded body, see "encoded_length". S3 wants all chunks but the last
one to be at least 8 KiB. body, body_fh, payload_hash and
unsigned_payload do not apply.
my $length = AWS::Signature::V4->encoded_length($decoded_length, $chunk_size);
my $r = $s->sign(
method => 'PUT', url => $url,
headers => { 'Content-Length' => $length },
streaming => 1, decoded_content_length => $decoded_length,
);
# send $r->{headers}, then the body, made of encoded chunks:
my $ck = $r->{chunker};
print {$socket} $ck->chunk($_) for @chunks;
print {$socket} $ck->finish;
Trailers, i.e. headers sent after the data, e.g. for a checksum that is
only known at the end, are declared with sign, which adds the
x-amz-trailer header (signed like the others):
checksum
the name of an algorithm that the chunker computes while the chunks
go through: crc32, crc32c, sha1 or sha256. The trailer is named after
it (e.g. x-amz-checksum-crc32c);
trailers
that is not allowed in a URL path (e.g. a space) percent-encoded, the
headers that the client must send along (just host by default), and
signature, signed_headers, scope, canonical_request and string_to_sign
like in "sign".
encoded_length
my $length = AWS::Signature::V4->encoded_length($decoded, $chunk_size, %opts);
Class method that computes the size of an aws-chunked body made of
$decoded bytes of data cut in chunks of $chunk_size bytes (except for
the last one, that can be shorter), which is what to put in the
Content-Length header. The sizes must be integers, and unknown options
are an error. The options mirror the choices made for "sign":
signed
true (1, default) if the chunks are signed, false (0 or the empty
string) otherwise; signed and unsigned are also accepted, as for
streaming in "sign";
trailers
hash reference from the name of each trailer that will be given to
"finish" to the length of its value (e.g. 12 for a base64-encoded
CRC-64). As in "sign", the trailer of checksum must not be listed
here as well: declaring the same name twice is an error.
The chunker
When streaming, "sign" returns a chunker that wraps the data into
chunks, signing them if needed. It keeps track of the amount of data
and of the signatures, so use it for one body only, and feed it the
data in order.
chunk
my $encoded = $chunker->chunk($data);
Return the encoded version of a piece of data, to be sent as it is.
$data is a byte string or a reference to one, it cannot be empty. It is
an error to go past the decoded_content_length.
finish
my $encoded = $chunker->finish(%trailer_values);
Return the final, empty chunk, followed by the trailers if any. It is
an error if the amount of data is not decoded_content_length. The
values of the trailers declared in trailers are passed by name, in any
case; it is an error to omit one, to pass one twice, to pass one that
was not declared or that is computed by the chunker, or to pass a value
that is not a byte string or has a CR, LF or NUL in it. After an error,
the chunker is as it was: finish can be called again.
ERRORS
Errors are reported by throwing Ouch exceptions (see $@->code,
$@->message). The code is 400 when the problem is in what the caller
- [x] **5. Examples** (Sonnet 5 or Opus 5): `tmp/review-5-examples.md`.
Start with the medium finding in `eg/05-s3-chunked-upload.pl`.
- [x] **6. Final check** (Opus 5): run `/code-review high` on the whole
diff of the fixes (from `d7b121f`, the last commit before them) to
catch regressions the fixes introduced.
`tmp/review-6-final.md`, fixed in `ed8fd99`.
## Before release
- [x] **Confirm the `Content-Encoding` order against a real bucket.**
Done: S3 in `eu-south-1` accepted it, and the stored metadata shows
the list was parsed rather than merely tolerated. The control, which
sent `aws-chunked` alone, came back stored as no `Content-Encoding`
at all; the real case, which sent `gzip,aws-chunked`, came back
stored as `gzip`. So S3 strips the `aws-chunked` token from the end
of the list and keeps what is left, which is what RFC 9110 and
botocore describe. Both objects round-tripped byte-identical.
What follows is the question as it stood, kept because it explains
why the module emits this order and how to ask again if AWS ever
answers differently.
With `streaming` and an encoding already set by the caller, `sign`
emits `gzip,aws-chunked` (`lib/AWS/Signature/V4.pm`, in the
`$streaming` branch; asserted in `t/streaming.t` and documented in
`V4.pod` under `sign`). This was the only behaviour in the module
chosen from indirect evidence rather than from AWS itself, because
the authorities contradict each other:
- RFC 9110 §8.4 wants the encodings in the order they were applied,
and `aws-chunked` is the outermost one, applied to the already
compressed data: `gzip,aws-chunked`;
- botocore (`httpchecksum.py`) agrees, appending: `headers
["Content-Encoding"] += ",aws-chunked"`;
- the first review cited the S3 documentation for the opposite
order, and a boto3 issue reports seeing `aws-chunked,gzip` in
stored metadata.
The guess was that S3 strips the `aws-chunked` token from either
position, so that both orders would be accepted and the point would
be moot; the run above confirms the stripping, at least from the
end. It does not say what the other order would do, and there is no
reason to find out while the module sends this one.
`eg/10-s3-content-encoding-probe.pl` settled it. It uploads a
gzipped object with `streaming`, plus a control without any
`Content-Encoding` so that a rejection can be told apart from a
eg/02-s3-get-object.pl view on Meta::CPAN
if ($ENV{DRY_RUN}) {
say "GET $url";
say "$_: $r->{headers}{$_}" for sort keys $r->{headers}->%*;
exit 0;
}
my $response = HTTP::Tiny->new(verify_SSL => 1)->get($url, {headers => \%headers});
die "$response->{status} $response->{reason}\n$response->{content}\n"
unless $response->{success};
if (defined $file) {
# the whole object is in memory here, because HTTP::Tiny->get was called
# without a data_callback: for big ones, pass one and write each piece
open my $fh, '>:raw', $file or die "open('$file'): $!\n";
print {$fh} $response->{content} or die "print('$file'): $!\n";
close $fh or die "close('$file'): $!\n"; # a full disk shows up here
}
else {
binmode STDOUT;
print $response->{content};
}
eg/04-s3-presigned-urls.pl view on Meta::CPAN
for my $method (qw< GET PUT >) {
my $p = $signer->presign(
method => $method,
url => $url,
expires => $expires // 3600, # seconds, up to 7 days
);
say "$method:\n$p->{url}\n";
}
say 'Try: curl -o object.bin "<GET url>"';
# -T (--upload-file) streams the file and adds no Content-Type of its own;
# --data-binary would read it all in memory and make S3 store the object as
# application/x-www-form-urlencoded
say 'Try: curl -T file "<PUT url>"';
eg/05-s3-chunked-upload.pl view on Meta::CPAN
# The same options must reach both encoded_length and sign, otherwise the
# announced Content-Length would not count the trailer and the upload would
# fail halfway through
my @checksum = ();
my $url = "https://$bucket.s3.$region.amazonaws.com/$key";
my $r = $signer->sign(
method => 'PUT',
url => $url,
headers => {
# length of the *encoded* body: data plus the per-chunk signatures
'Content-Length' =>
AWS::Signature::V4->encoded_length($size, $chunk_size, @checksum),
},
streaming => 1,
decoded_content_length => $size,
@checksum,
);
my $chunker = $r->{chunker};
# HTTP::Tiny adds the Host header itself, from the URL, and refuses to be
eg/10-s3-content-encoding-probe.pl view on Meta::CPAN
#!/usr/bin/env perl
# Does S3 accept the Content-Encoding order that this module sends?
#
# ./10-s3-content-encoding-probe.pl BUCKET [KEY_PREFIX]
#
# With streaming, sign() has to add "aws-chunked" to whatever
# Content-Encoding the caller already set. RFC 9110 and botocore put it
# last ("gzip,aws-chunked"), because aws-chunked is applied to the
# already compressed data; the S3 documentation was read the other way
# round in an earlier review of this distribution. See TODO.md: nothing
# but a real bucket can settle it, and this is the program that asks.
#
# Two objects are uploaded: a control with no Content-Encoding of its
# own, and the real case with gzip. The control tells a wrong order apart
# from a wrong bucket, region or set of credentials. Both are deleted
# again, unless KEEP=1.
#
# The report at the end is meant to be pasted in a bug report or a chat:
# it is built from a fixed list of fields, so it carries no credentials,
eg/10-s3-content-encoding-probe.pl view on Meta::CPAN
secret_access_key => $ENV{AWS_SECRET_ACCESS_KEY},
session_token => $ENV{AWS_SESSION_TOKEN},
},
);
# something that compresses well, so that the gzipped payload is clearly
# not the plain one and a body that came back unzipped cannot be mistaken
# for a good round trip
my $plain = "the quick brown fox jumps over the lazy dog\n" x 100;
gzip \$plain, \my $gz or die "gzip failed: $GzipError\n";
my $chunk_size = 64 * 1024; # one data chunk: the last one, so S3's
my $size = length $gz; # 8 KiB minimum does not apply to it
my $chunks = int($size / $chunk_size) + ($size % $chunk_size ? 1 : 0);
my $http = HTTP::Tiny->new(verify_SSL => 1);
my $base = "https://$bucket.s3.$region.amazonaws.com/";
# HTTP::Tiny adds the Host header itself and refuses to be given one; it
# is signed all the same, and it will have the same value
sub without_host {
my ($r) = @_;
eg/10-s3-content-encoding-probe.pl view on Meta::CPAN
}
for my $c (@cases) {
my $url = $base . $c->{key};
my $r = $signer->sign(
method => 'PUT',
url => $url,
headers => {
$c->{extra}->%*,
'Content-Type' => 'text/plain',
# length of the *encoded* body: data plus the chunk framing
'Content-Length' => AWS::Signature::V4->encoded_length($size, $chunk_size),
},
streaming => 1,
decoded_content_length => $size,
);
# what the module decided to send, rather than what we assume it sends:
# this is the whole question, and it must be reported as it is
$c->{sent} = $r->{headers}{'content-encoding'};
$c->{announced} = $r->{headers}{'content-length'};
# the encoded body is built once, whole, and sent as a string. A
eg/10-s3-content-encoding-probe.pl view on Meta::CPAN
my $d = $signer->sign(method => 'DELETE', url => $url);
my $del = $http->request(DELETE => $url, {headers => without_host($d)});
$c->{leftover} = $c->{key} unless $del->{success};
}
# --- the report, safe to paste anywhere ------------------------------------
my @out = (
'--- aws-chunked ordering probe ------------------------',
"module: AWS::Signature::V4 " . (AWS::Signature::V4->VERSION // '(unknown)'),
"region: $region (bucket and key not shown)",
sprintf('payload: %d bytes plain -> %d bytes gzip, %d data chunk%s',
length $plain, $size, $chunks, $chunks == 1 ? '' : 's'),
'',
);
for my $c (@cases) {
push @out, sprintf('%-10s sent Content-Encoding: %s', $c->{name}, $c->{sent});
push @out, " dry $c->{dry}" if defined $c->{dry};
push @out, " PUT $c->{put}" if defined $c->{put};
push @out, " error $c->{put_error}"
if defined $c->{put_error} && length $c->{put_error};
push @out, " GET $c->{get}" . (defined $c->{stored}
eg/README.md view on Meta::CPAN
The output suggests how to use them with `curl`.
## 05-s3-chunked-upload.pl
Streaming upload to S3 (`aws-chunked`): the body is sent in signed
chunks, so the whole payload does not need to be hashed in advance nor
held in memory. The total size must be known, because the `Content-Length`
of the encoded body is computed with `encoded_length` and the chunker
checks that the data matches. In dry-run mode it also encodes the file and
reports how many bytes the body has, compared with the announced length.
To try a trailing checksum, set the `@checksum` variable in the script to
`(checksum => 'crc32c')`: it is passed to both `encoded_length` and
`sign`, because the announced length has to count the trailer too.
```shell
./eg/05-s3-chunked-upload.pl data.bin my-bucket uploads/data.bin
DRY_RUN=1 ./eg/05-s3-chunked-upload.pl data.bin my-bucket uploads/data.bin
```
## 06-dynamodb-json-api.pl
A JSON API, where the operation is chosen by a header
(`X-Amz-Target`), signed like all the others. It lists the DynamoDB tables
of a region.
```shell
lib/AWS/Signature/V4.pm view on Meta::CPAN
$s =~ s{([^A-Za-z0-9\-_.~])}{sprintf '%%%02X', ord $1}gmse;
return $s;
}
sub _uri_decode ($s) {
$s =~ s{%([0-9A-Fa-f]{2})}{chr hex $1}gse;
return $s;
}
# body is either a plain scalar or a reference to one; only in the latter
# case the caller's data is not copied at all
sub _body_ref ($args) {
my $body = $args->{body};
return \$args->{body} unless ref $body;
fail 400, 'body must be a scalar or a reference to a scalar'
unless ref($body) eq 'SCALAR';
return $body;
}
# AWS trims and collapses spaces, so ASCII whitespace only (/a): with
# unicode_strings, \s would also take the bytes 0xA0 and 0x85 of UTF-8 text
lib/AWS/Signature/V4.pm view on Meta::CPAN
# make sure the caller provides us *exactly* one of credentials/X509
my $x509_params = $self->x509;
my $credentials_params = $self->credentials;
fail 400, 'provide either "credentials" or "x509", not both'
if $credentials_params && $x509_params;
fail 400, 'provide (exactly) one of "credentials" or "x509"'
unless $credentials_params || $x509_params;
$self->_auth; # force building and fail fast
# after using the parameters, we get rid of sensitive data in the input
# hash for X509
delete($x509_params->{$_}) for qw< private_key private_key_password >;
return;
}
sub _build__auth ($self) {
return $self->credentials
? AWS::Signature::V4::Credentials->new($self->credentials->%*)
: AWS::Signature::V4::X509->new($self->x509->%*);
lib/AWS/Signature/V4.pm view on Meta::CPAN
unless $len =~ m{\A[0-9]+\z};
$payload_hash =
$unsigned_chunks ? 'STREAMING-UNSIGNED-PAYLOAD-TRAILER'
: @trailer_names ? 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER'
: 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD';
$h->{'x-amz-decoded-content-length'} = $len;
$h->{'x-amz-trailer'} = join ',', @trailer_names if @trailer_names;
# whatever the caller has, e.g. gzip, then aws-chunked: RFC 9110 wants
# the encodings in the order they were applied, and this one is the
# outermost, applied to the already-gzipped data (as botocore does)
my @encodings = grep { length && lc($_) ne 'aws-chunked' }
map { _trim($_) } split m{,}, $h->{'content-encoding'} // '';
$h->{'content-encoding'} = join ',', @encodings, 'aws-chunked';
}
else {
$payload_hash = _payload_hash(\%args);
}
$payload_hash = _agree($payload_hash, $given, sha256_hex(''));
$h->{'x-amz-content-sha256'} = $payload_hash
if $self->payload_header || $streaming || defined $given || !_is_sha256($payload_hash);
lib/AWS/Signature/V4.pm view on Meta::CPAN
for my $given (sort keys $declared->%*) {
my $name = lc $given;
_check_trailer_name($name);
# sign() refuses the same repetition, so do not count a line twice
fail 400, "trailer '$name' declared twice" if $seen{$name}++;
fail 400, 'the length of trailer ' . shown($given) . ' must be a non-negative integer'
unless _is_count($declared->{$given});
$trailer += length($name) + 1 + $declared->{$given} + 2;
}
my $chunk = sub ($size) { # everything around the data of a chunk
length(sprintf '%x', $size) + ($signed ? 17 + 64 : 0) + 2 + 2;
};
my $total = 0;
my $full = int($decoded / $chunk_size);
$total += $full * ($chunk->($chunk_size) + $chunk_size);
my $rest = $decoded - $full * $chunk_size;
$total += $chunk->($rest) + $rest if $rest;
# final chunk: "0[;chunk-signature=..]\r\n", trailers, [signature line,] "\r\n"
$total += 1 + ($signed ? 17 + 64 : 0) + 2 + $trailer + 2;
lib/AWS/Signature/V4.pod view on Meta::CPAN
meaning of the canonical request. The same check applies to the values
that C<sign> adds, e.g. the session token, and to the ones that
L</presign> puts in the query string instead.
An C<X-Amz-Content-Sha256> header is taken as the payload hash; if the
payload hash also comes from the arguments below, the two must agree;
=item C<body>, C<body_fh>
the payload, as a byte string, a reference to one (which avoids copying
large data around) or an open filehandle. The filehandle is hashed
from its current position to its end, without loading it in memory, and
put back where it was: so it must be seekable (a file, not a pipe or a
socket) and binary, without layers like C<:encoding> or C<:crlf> that
change the bytes read. Only one of them can be used;
=item C<payload_hash>, C<unsigned_payload>
skip the calculation and use the provided SHA-256 in hex, or
C<UNSIGNED-PAYLOAD>. They take precedence over C<body> and C<body_fh>.
C<payload_hash> can also be a C<STREAMING-*> marker, anything else is an
lib/AWS/Signature/V4.pod view on Meta::CPAN
=back
=head3 Chunked and streaming uploads
C<streaming> enables the C<aws-chunked> encoding used for uploads to S3,
where the body is sent in chunks that are signed as they go. It can be
C<1> or C<signed> (each chunk is signed, credentials variant only) or
C<unsigned> (no chunk signatures, only the request headers are signed,
also OK with X.509). The C<decoded_content_length>, i.e. the size of
the data, is mandatory. C<sign> sets the payload hash to
C<STREAMING-AWS4-HMAC-SHA256-PAYLOAD> (or its variants below), adds
C<x-amz-decoded-content-length> and C<aws-chunked> to C<Content-Encoding>
(after any other encoding, e.g. C<gzip,aws-chunked>, as it is the one
applied last: S3 takes that token off the end and stores what is left,
here C<gzip>), all signed; the C<Content-Length> to provide is the size
of the encoded body, see L</encoded_length>. S3 wants all chunks but the last
one to be at least 8 KiB. C<body>, C<body_fh>, C<payload_hash> and
C<unsigned_payload> do not apply.
my $length = AWS::Signature::V4->encoded_length($decoded_length, $chunk_size);
my $r = $s->sign(
method => 'PUT', url => $url,
headers => { 'Content-Length' => $length },
streaming => 1, decoded_content_length => $decoded_length,
);
# send $r->{headers}, then the body, made of encoded chunks:
my $ck = $r->{chunker};
print {$socket} $ck->chunk($_) for @chunks;
print {$socket} $ck->finish;
Trailers, i.e. headers sent after the data, e.g. for a checksum that is
only known at the end, are declared with C<sign>, which adds the
C<x-amz-trailer> header (signed like the others):
=over
=item C<checksum>
the name of an algorithm that the chunker computes while the chunks go
through: C<crc32>, C<crc32c>, C<sha1> or C<sha256>. The trailer is
named after it (e.g. C<x-amz-checksum-crc32c>);
lib/AWS/Signature/V4.pod view on Meta::CPAN
percent-encoded, the C<headers> that the client must send along (just
C<host> by default), and
C<signature>, C<signed_headers>, C<scope>, C<canonical_request> and
C<string_to_sign> like in L</sign>.
=head2 encoded_length
my $length = AWS::Signature::V4->encoded_length($decoded, $chunk_size, %opts);
Class method that computes the size of an C<aws-chunked> body made of
C<$decoded> bytes of data cut in chunks of C<$chunk_size> bytes (except
for the last one, that can be shorter), which is what to put in the
C<Content-Length> header. The sizes must be integers, and unknown
options are an error. The options mirror the choices made for L</sign>:
=over
=item C<signed>
true (C<1>, default) if the chunks are signed, false (C<0> or the empty
string) otherwise; C<signed> and C<unsigned> are also accepted, as for
lib/AWS/Signature/V4.pod view on Meta::CPAN
hash reference from the name of each trailer that will be given to
L</finish> to the length of its value (e.g. C<12> for a base64-encoded
CRC-64). As in L</sign>, the trailer of C<checksum> must not be listed
here as well: declaring the same name twice is an error.
=back
=head2 The chunker
When streaming, L</sign> returns a C<chunker> that wraps the data
into chunks, signing them if needed. It keeps track of the amount of
data and of the signatures, so use it for one body only, and feed it the
data in order.
=head3 chunk
my $encoded = $chunker->chunk($data);
Return the encoded version of a piece of data, to be sent as it is.
C<$data> is a byte string or a reference to one, it cannot be empty. It is
an error to go past the C<decoded_content_length>.
=head3 finish
my $encoded = $chunker->finish(%trailer_values);
Return the final, empty chunk, followed by the trailers if any. It is an
error if the amount of data is not C<decoded_content_length>. The values
of the trailers declared in C<trailers> are passed by name, in any case;
it is an error to omit one, to pass one twice, to pass one that was not
declared or that is computed by the chunker, or to pass a value that is
not a byte string or has a CR, LF or NUL in it. After an error, the
chunker is as it was: C<finish> can be called again.
=head1 ERRORS
Errors are reported by throwing L<Ouch> exceptions (see C<< $@->code >>,
lib/AWS/Signature/V4/Checksum.pm view on Meta::CPAN
return {algorithm => $args[0]} if @args == 1 && !ref $args[0];
return $class->$orig(@args);
};
sub _build__crc ($self) { $self->algorithm eq 'crc32c' ? 0xFFFFFFFF : 0 }
sub _build__sha ($self) { Digest::SHA->new($self->algorithm eq 'sha1' ? 1 : 256) }
sub encoded_size ($class, $algo) { 4 * int((SIZE_FOR->{$algo} + 2) / 3) }
sub add ($self, $ref) { # $ref: reference to a byte string
fail 400, 'checksum data must be a reference to a byte string'
unless ref $ref eq 'SCALAR' && defined $$ref;
utf8::is_utf8($$ref) && !utf8::downgrade(my $copy = $$ref, 1)
and fail 400, 'checksum data must be a byte string: encode characters first';
my $algo = $self->algorithm;
if ($algo eq 'crc32') {
$self->_crc(Compress::Raw::Zlib::crc32($$ref, $self->_crc));
}
elsif ($algo eq 'crc32c') {
$self->_crc(_crc32c($self->_crc, $ref));
}
else { $self->_sha->add($$ref) }
return $self;
}
# register after the data; String::CRC32C wants and gives it inverted
sub _crc32c ($crc, $ref) {
return CRC32C_XS->($$ref, $crc ^ 0xFFFFFFFF) ^ 0xFFFFFFFF if CRC32C_XS;
state $T0 = [
map {
my $c = $_;
$c = ($c & 1) ? (($c >> 1) ^ 0x82F63B78) : ($c >> 1) for 1 .. 8;
$c;
} 0 .. 255
];
lib/AWS/Signature/V4/Checksum.pod view on Meta::CPAN
This module is part of the L<AWS::Signature::V4> distribution and shares
its version.
=head1 SYNOPSIS
use AWS::Signature::V4::Checksum ();
my $sum = AWS::Signature::V4::Checksum->new('crc32c');
$sum->add(\$_) for @pieces; # or $sum->add(\$data)
my $value = $sum->base64; # to send as x-amz-checksum-crc32c
my $size = AWS::Signature::V4::Checksum->encoded_size('sha256'); # 44
=head1 DESCRIPTION
This module computes, a piece at a time, the checksums that S3 accepts
in the C<x-amz-checksum-*> headers and trailers: C<crc32>, C<crc32c>,
C<sha1> and C<sha256>. C<crc32> comes from L<Compress::Raw::Zlib>, the SHA
ones from L<Digest::SHA>, C<crc32c> from L<String::CRC32C> if it is
installed. Otherwise C<crc32c> is computed in pure Perl, at roughly 12 MB/s:
install L<String::CRC32C> to use it with big data.
C<crc64nvme> is not provided: for that and any other algorithm, first
figure out if you I<really> need something that is not provided out of
the box with one of the alternatives above, then if still convinced
calculate the value yourself and give it to
L<AWS::Signature::V4::Chunker/finish>.
You normally do not need to use this module: L<AWS::Signature::V4>
uses it on your behalf when you pass C<checksum> to
L<AWS::Signature::V4/sign> or L<AWS::Signature::V4/encoded_length>, and
the chunker feeds it with the data. B<It is however safe to use it
directly>, e.g. to compute a checksum in advance and send it as a normal
header, because it is self-contained and does not depend on any signing
material. Note that it is not tied to the AWS signature in any way.
=head1 INTERFACE
=head2 new
my $sum = AWS::Signature::V4::Checksum->new($algorithm);
Start a new checksum. C<$algorithm> is one of C<crc32>, C<crc32c>,
C<sha1> and C<sha256>, in lowercase. Other names are not validated here;
L<AWS::Signature::V4/sign> checks them before getting to this point, so
check against L</is_supported_algorithm> if you call this directly.
=head2 add
$sum->add(\$bytes);
Update the checksum with a piece of data. The argument is a I<reference>
to a byte string, to avoid copying data around. Return the object, so
calls can be chained.
It is an error (an L<Ouch> exception with code C<400>) to pass anything
else than a reference to a defined string, or a string with characters
that do not fit in a byte: encode them first, e.g. with
C<Encode::encode('UTF-8', $text)>.
=head2 base64
my $value = $sum->base64;
lib/AWS/Signature/V4/Checksum.pod view on Meta::CPAN
consume the object for the CRCs, but for the SHA algorithms it does
(L<Digest::SHA> resets after C<digest>): call it once, at the end.
=head2 encoded_size
my $length = AWS::Signature::V4::Checksum->encoded_size($algorithm);
Class method returning the length of the base64 value for an algorithm
(8 for the CRCs, 28 for C<sha1>, 44 for C<sha256>). It is what
L<AWS::Signature::V4/encoded_length> needs to work out the size of the
trailer without having the data.
=head2 is_supported_algorithm
my $bool = AWS::Signature::V4::Checksum->is_supported_algorithm($name);
Class method returning a (Perl) true/false value indicating whether the
provided C<$name> is the name of a supported algorithm or not.
=head2 supported_algorithms
lib/AWS/Signature/V4/Chunker.pm view on Meta::CPAN
has scope => (is => 'ro');
# The signing key, for signed chunks only, can sign any request for the same
# service, region and day: it is kept in a closure, so that no accessor gives
# it back and dumping the object does not show it. The closure can still sign
# with it, so the chunker is no safer to share than the signer.
has _mac => (
is => 'ro',
init_arg => 'key',
coerce => sub ($key) {
ref $key eq 'CODE' ? $key : sub ($data) { hmac_sha256_hex($data, $key) };
},
);
# signature of the previous chunk: the chain must not be changed from outside
has _previous => (is => 'rw', init_arg => 'previous');
has expected => (is => 'ro', required => 1); # declared size of the data
has checksum => (is => 'ro'); # name of the built-in checksum, if any
has trailer_names => (is => 'ro', default => sub { [] });
has _seen => (is => 'rw', init_arg => undef, default => 0);
has _done => (is => 'rw', init_arg => undef, default => 0);
has _sum => (is => 'lazy', init_arg => undef);
sub BUILD ($self, $args) {
fail 500, 'signed chunks need a "key"' if $self->signed && !$self->_mac;
return;
}
sub _build__sum ($self) {
my $algo = $self->checksum;
return defined $algo ? AWS::Signature::V4::Checksum->new($algo) : undef;
}
sub _sign ($self, $data_ref) {
my $string_to_sign = join "\n", 'AWS4-HMAC-SHA256-PAYLOAD',
$self->amzdate, $self->scope, $self->_previous,
sha256_hex(''), sha256_hex($$data_ref);
return $self->_previous($self->_mac->($string_to_sign));
}
# encoded chunk for the data (a byte string or a reference to one)
sub chunk ($self, $data) {
fail 400, 'chunker already finished' if $self->_done;
fail 400, 'chunk must be a byte string or a reference to one'
if ref $data && ref $data ne 'SCALAR';
my $ref = ref $data ? $data : \$data;
my $len = length($$ref) // 0;
fail 400, 'empty chunk: call finish() to terminate the stream' unless $len;
_is_bytes($ref) or fail 400, 'chunk must be a byte string: encode characters first';
my $seen = $self->_seen + $len;
fail 400, "more data than the declared @{[ $self->expected ]} bytes"
if $seen > $self->expected;
$self->_seen($seen);
$self->_sum->add($ref) if $self->_sum;
my $head = $self->signed
? sprintf('%x;chunk-signature=%s', $len, $self->_sign($ref))
: sprintf('%x', $len);
return "$head\r\n" . $$ref . "\r\n";
}
# the final, empty chunk, followed by the trailers if any; the values of the
lib/AWS/Signature/V4/Chunker.pod view on Meta::CPAN
headers => { 'Content-Length' => AWS::Signature::V4->encoded_length($size, $chunk_size) },
);
my $chunker = $r->{chunker};
print {$socket} $chunker->chunk($_) for @pieces;
print {$socket} $chunker->finish;
=head1 DESCRIPTION
This module wraps the data of an S3 upload into the C<aws-chunked>
encoding: each piece is prefixed by its size and, for the signed
variants, by a signature that depends on the previous one.
B<You are not supposed to use this module directly.> Objects are created
by L<AWS::Signature::V4/sign> when called with C<streaming>, and returned
in the C<chunker> key of its result. Only then do they hold what they need
to sign: the derived signing key, the scope, the date and the signature of
the request itself (the "seed" of the chain). The constructor is
therefore an internal detail and its arguments are not a stable
interface; the options that matter (C<streaming>, C<checksum>,
lib/AWS/Signature/V4/Chunker.pod view on Meta::CPAN
What you do use are the two methods below, which are part of the public
interface. A chunker is good for one body: create a new one, i.e. call
C<sign> again, for each upload.
=head1 INTERFACE
=head2 chunk
my $encoded = $chunker->chunk($data);
Return the encoded version of a piece of data, to be sent as it is.
C<$data> is a byte string or a reference to one (to avoid copying large
pieces), it cannot be empty. Other kinds of references, objects included,
are refused. It is an error to pass characters that do not
fit in a byte, or to go past the C<decoded_content_length> declared to
C<sign>. Pieces must be passed in the order they are sent.
If a C<checksum> was requested, the piece also updates the running
checksum.
=head2 finish
my $encoded = $chunker->finish(%trailer_values);
Return the final, empty chunk, followed by the trailers if any, and
their signature for the signed variants. It is an error if the amount of
data passed so far is not the declared C<decoded_content_length>, or if
the chunker was already finished.
The values of the trailers declared with C<trailers> are passed by name,
in any case (C<X-Foo> and C<x-foo> are the same trailer). It is an error
to omit one, to pass one twice, to pass one that was not declared or that
the chunker computes by itself (see C<checksum>), or to pass a value that
is not a byte string or has a CR, LF or NUL in it. The trailer of a
C<checksum> is added automatically.
All the errors are found before anything changes: after one, the
lib/AWS/Signature/V4/Error.pod view on Meta::CPAN
This module is part of the L<AWS::Signature::V4> distribution and shares
its version.
=head1 SYNOPSIS
use AWS::Signature::V4::Error qw< fail shown >;
fail 400, 'missing parameter "url"';
fail 400, 'bad option', {option => 'foo', got => $value}; # data for logs
=head1 DESCRIPTION
B<This module is internal: you are not supposed to use it.> It is what the
modules of the distribution use to throw L<Ouch> exceptions, see
L<AWS::Signature::V4/ERRORS> for what the callers can expect.
It exists to do two things that a bare C<ouch> would not do: report the
error at the line of the caller, like C<croak>, and keep the arguments of
the calls out of the backtrace kept by the exception, because they can
include secrets like passwords and tokens.
=head1 INTERFACE
=head2 fail
fail $code, $message;
fail $code, $message, $data;
Throw an L<Ouch> exception. All arguments go to C<ouch> as they are.
The code is C<400> when the problem is in what the caller provided, C<500>
when it is in the module itself.
The optional C<$data> is kept by the exception and is available with
C<< $@->data >> (and C<< $@->hashref >>), but it is not part of the
message, so it is not shown by just printing the error. Use it for the
details that are useful in a log, e.g. what was received, and keep it
out of what is shown to end users. It is also kept out of the backtrace:
the exception stores C<$data> itself, but the arguments of the calls are
not recorded in C<< $@->trace >>, so they can safely carry secrets.
Nothing in the distribution passes C<$data> at the moment, but it is part
of the interface and you can rely on it being passed through.
=head2 shown
fail 400, 'unknown checksum ' . shown($name);
Return a value provided by the caller in a form that is safe to put in a
message: every character that is not printable ASCII is escaped as
C<\x{...}>, so that the message cannot forge lines in a log; C<undef>
lib/AWS/Signature/V4/X509.pm view on Meta::CPAN
return $value;
}
sub _b64 ($der) { encode_base64($der, '') }
# ---- X.509 helpers (no external dependencies) ------------------------------
sub _slurp ($file) {
open my $fh, '<:raw', $file or fail 400, 'open(' . shown($file) . "): $!";
local $/;
my $data = <$fh>;
close $fh;
return $data;
}
sub _to_ders ($cert) { # PEM bundles hold many certificates
return _check_der($cert) unless $cert =~ m{-----BEGIN};
my @ders = map { decode_base64($_) }
$cert =~ m{-----BEGIN CERTIFICATE-----(.*?)-----END CERTIFICATE-----}sg;
@ders or fail 400, 'no CERTIFICATE block in PEM';
return map { _check_der($_) } @ders;
}
sub _to_der ($cert) {
return _check_der($cert) unless $cert =~ m{-----BEGIN};
$cert =~ m{-----BEGIN CERTIFICATE-----(.*?)-----END CERTIFICATE-----}s
or fail 400, 'no CERTIFICATE block in PEM';
return _check_der(decode_base64($1));
}
# a certificate is a DER SEQUENCE that fits exactly in the data
sub _check_der ($der) {
my ($tag, undef, $end) = _tlv($der, 0);
fail 400, 'not a DER-encoded certificate'
unless $tag == 0x30 && $end == length $der;
return $der;
}
# read one DER TLV at $off => (tag, content_start, content_end)
sub _tlv ($der, $off) {
my $max = length $der;
lib/AWS/Signature/V4/X509.pod view on Meta::CPAN
name AWS knows. The class is documented for those who work on the
distribution, not as a public interface.
The constructor loads and checks everything: it reads the files, decodes
the certificates (PEM or DER) and finds the serial number, and loads the
key with L<CryptX> unless a C<signer> is provided. Any problem is reported
by an L<Ouch> exception with code C<400>, so it is seen when the signer is
created and not when the first request is signed. It does not check that
the certificate is valid otherwise, or that the key matches it: only
that each certificate is a DER SEQUENCE (definite, minimal lengths) that
fills its data exactly. The error of L<CryptX> about the key goes in the
message without its location or backtrace, which may hold the password,
and escaped, as it may hold the file name.
C<private_key> and C<private_key_password> have no public accessor and are
dropped as soon as the key is loaded, so that the object does not keep
them around.
=head1 THE INTERFACE OF A VARIANT
my $secret = 'S3CR3T-value';
sub thrown (@args) { my $e = dies { fail(@args) }; $e }
subtest 'code and message' => sub {
my $e = thrown(400, 'bad input');
isa_ok $e, ['Ouch'], 'an Ouch exception';
is $e->code, 400, 'code';
is $e->message, 'bad input', 'message';
is $e->data, undef, 'no data by default';
is thrown(500, 'internal')->code, 500, 'other code';
};
subtest 'the third argument is passed down' => sub {
my $e = thrown(400, 'bad option', {option => 'foo', got => $secret});
is $e->code, 400, 'code';
is $e->message, 'bad option', 'message';
is $e->data, {option => 'foo', got => $secret}, 'data is available';
is $e->hashref->{data}, {option => 'foo', got => $secret}, 'and it is in hashref too';
unlike "$e", qr{\Q$secret\E|foo}, 'data is not in the stringified error';
unlike $e->message, qr{\Q$secret\E}, 'nor in the message';
my $d = [1, 2, 3];
is thrown(400, 'array', $d)->data, [1, 2, 3], 'not only hashes';
is thrown(400, 'scalar', 'plain')->data, 'plain', 'nor references';
};
subtest 'data and arguments stay out of the trace' => sub {
my $e = do {
package Some::Caller;
sub run ($password) { main::thrown(400, 'oops', {got => $secret}) }
run('Hunter2');
};
is $e->data, {got => $secret}, 'data kept';
unlike $e->trace, qr{\Q$secret\E|Hunter2|HASH\(}, 'no arguments in the trace';
};
subtest 'reported where the caller is' => sub {
my $line = __LINE__ + 1;
my $e = dies { fail 400, 'here', {some => 'data'} };
like "$e", qr{ at \S*error\.t line $line\b}, 'file and line of the caller';
};
done_testing;
my $ck = streaming($s, decoded_content_length => 9, checksum => 'sha256',
trailers => ['x-foo'], signed_headers => ['host'])->{chunker};
$ck->chunk('123456789');
ouch_like sub { $ck->finish }, 400, 'missing trailer value';
like $ck->finish('x-foo' => 1),
qr{x-amz-checksum-sha256:\Q@{[ encode_base64(sha256('123456789'), '') ]}\E\r\n},
'the checksum survives a failed finish';
$ck = streaming($s, decoded_content_length => 4)->{chunker};
$ck->chunk('abc');
ouch_like sub { $ck->chunk('xy') }, 400, 'too much data';
ok lives { $ck->chunk('x') }, 'a chunk that fits is still accepted';
ok lives { $ck->finish }, 'and the stream can be completed';
done_testing;
t/streaming.t view on Meta::CPAN
is(AWS::Signature::V4->encoded_length(0, 10), 86, 'empty body: only the final chunk');
is(AWS::Signature::V4->encoded_length(20, 10), 2 * (1 + 17 + 64 + 2 + 10 + 2) + 86 - 0,
'two full chunks');
is(AWS::Signature::V4->encoded_length(25, 10), 2 * (1 + 17 + 64 + 2 + 10 + 2) + (1 + 17 + 64 + 2 + 5 + 2) + 86,
'partial last chunk');
# body_fh: the hash of the file, without loading it
my ($fh, $file) = tempfile(UNLINK => 1);
binmode $fh;
my $data = join '', map { chr($_ % 256) } 1 .. 300_000;
print {$fh} $data;
close $fh;
open my $in, '<:raw', $file or die;
seek $in, 5, 0;
my $std = AWS::Signature::V4->new(service => 'service', region => 'r',
credentials => {access_key_id => 'A', secret_access_key => 'S'});
my $viafh = $std->sign(method => 'PUT', url => 'https://h/', body_fh => $in, time => 0);
my $viabody = $std->sign(method => 'PUT', url => 'https://h/', body => substr($data, 5), time => 0);
is $viafh->{signature}, $viabody->{signature}, 'body_fh hashes from the current position to the end';
is tell($in), 5, 'and the file position is restored';
ok dies { $std->sign(method => 'PUT', url => 'https://h/', body_fh => $in, body => 'x') },
'body and body_fh together are refused';
my $pfh = $std->presign(url => 'https://h/', body_fh => $in, time => 0);
my $pbody = $std->presign(url => 'https://h/', body => substr($data, 5), time => 0);
is $pfh->{signature}, $pbody->{signature}, 'body_fh works with presign too';
sub bad ($re) {
object { prop blessed => 'Ouch'; call code => 400; call message => match $re };
}
# body_fh must be put back where it was, so it must be seekable and binary
open my $pipe, '-|', $^X, '-e', 'print "hello"' or die "pipe: $!";
is dies { $std->sign(method => 'PUT', url => 'https://h/', body_fh => $pipe) },
bad(qr/seekable/), 'a pipe is refused';
t/trailer.t view on Meta::CPAN
sha1 => encode_base64(sha1('123456789'), ''),
sha256 => encode_base64(sha256('123456789'), ''),
);
for my $algo (sort keys %expect) {
my $ck = start($x, streaming => 'unsigned', checksum => $algo)->{chunker};
my $body = $ck->chunk('1234') . $ck->chunk('56789') . $ck->finish;
is $body, "4\r\n1234\r\n5\r\n56789\r\n0\r\nx-amz-checksum-$algo:$expect{$algo}\r\n\r\n",
"unsigned, $algo, byte for byte";
}
# --- crc32c against a bitwise reference, on longer data and in many chunks
sub crc32c_ref ($data) {
my $crc = 0xFFFFFFFF;
for my $byte (unpack 'C*', $data) {
$crc ^= $byte;
$crc = ($crc & 1) ? (($crc >> 1) ^ 0x82F63B78) : ($crc >> 1) for 1 .. 8;
}
return encode_base64(pack('N', $crc ^ 0xFFFFFFFF), '');
}
my $long = join '', map { chr(($_ * 7) % 256) } 1 .. 5000;
my $ck = start($x, streaming => 'unsigned', checksum => 'crc32c',
decoded_content_length => length $long)->{chunker};
$ck->chunk(substr($long, $_ * 1000, 1000)) for 0 .. 4;
like $ck->finish, qr/x-amz-checksum-crc32c:\Q${\ crc32c_ref($long)}\E\r\n/, 'crc32c on longer data';
# the test here will exercise whatever is available; we don't explicitly
# test the pure-perl version shipped with the module in case
# String::CRC32 is available. FIXME figure out how to do this withouth
# going back to using package variables.
for my $size (1 .. 9, 65535 .. 65537, 70001) {
my $sum = AWS::Signature::V4::Checksum->new('crc32c');
my $data = substr $long x 15, 0, $size;
my ($head, $tail) = (substr($data, 0, $size / 3), substr($data, $size / 3));
$sum->add(\$head)->add(\$tail);
is $sum->base64, crc32c_ref($data), "crc32c, $size bytes";
}
# wide characters are refused by every algorithm, references to anything else
# than a defined string too; upgraded byte strings are fine
for my $algo (sort keys %expect) {
my $sum = AWS::Signature::V4::Checksum->new($algo);
is dies { $sum->add(\"\x{263a}") }, bad(qr/byte string/), "$algo: wide character refused";
is dies { $sum->add('123') }, bad(qr/reference/), "$algo: not a reference";
is dies { $sum->add(\undef) }, bad(qr/reference/), "$algo: undef";
# stored as UTF-8, with bytes that differ from the characters: what is
t/trailer.t view on Meta::CPAN
[signed => 1, {trailers => {'x-amz-checksum-crc64nvme' => 12}}, {streaming => 1, trailers => ['x-amz-checksum-crc64nvme']}, {'x-amz-checksum-crc64nvme' => 'A' x 12}],
[signed => 0, {checksum => 'sha256'}, {streaming => 'unsigned', checksum => 'sha256'}],
[signed => 0, {checksum => 'sha1', trailers => {'x-foo' => 3}}, {streaming => 'unsigned', checksum => 'sha1', trailers => ['x-foo']}, {'x-foo' => 'abc'}],
);
for my $case (@cases) {
my ($label, $signed, $opts, $sign, $values) = @$case;
$sign //= {streaming => 1};
for my $size (0, 1, 15, 16, 17, 255, 256, 300, 5000) {
for my $chunk (16, 256, 1000) {
my $c = start($signed ? $s3 : $x, %$sign, decoded_content_length => $size)->{chunker};
my $data = 'z' x $size;
my $body = '';
for (my $off = 0; $off < $size; $off += $chunk) {
$body .= $c->chunk(substr($data, $off, $chunk));
}
$body .= $c->finish(%{$values // {}});
is length($body), AWS::Signature::V4->encoded_length($size, $chunk, signed => $signed, %$opts),
"encoded_length: " . join(',', $signed ? 'signed' : 'unsigned', sort keys %$opts) . " size $size chunk $chunk";
}
}
}
is(AWS::Signature::V4->encoded_length(66560, 65536), 66824, 'still the AWS docs value without trailers');
ok dies { AWS::Signature::V4->encoded_length(10, 5, checksum => 'nope') }, 'encoded_length: unknown checksum';
( run in 2.581 seconds using v1.01-cache-2.11-cpan-16aabcbfbcb )