Amazon-S3-Lite
view release on metacpan or search on metacpan
share/README.md view on Meta::CPAN
temporary credential sets.
- credentials
An object providing credential getters. The object must respond to:
$creds->aws_access_key_id
$creds->aws_secret_access_key
$creds->token # may return undef
Any object that satisfies this interface is accepted -
[Amazon::Credentials](https://metacpan.org/pod/Amazon%3A%3ACredentials), [Paws::Credential::\*](https://metacpan.org/pod/Paws%3A%3ACredential%3A%3A%2A), or your own. The
getters are called at request time, so objects that refresh expiring
credentials transparently are supported.
- logger
An object providing the standard log methods:
$logger->trace(...)
$logger->debug(...)
$logger->info(...)
$logger->warn(...)
$logger->error(...)
If not supplied, the module looks for [Log::Log4perl](https://metacpan.org/pod/Log%3A%3ALog4perl). If available,
it calls `Log::Log4perl::easy_init` with the configure log level (or
WARN) and logs to STDERR. If Log::Log4perl is not installed, a
minimal internal logger.
- host
Override the S3 endpoint host. Defaults to `s3.amazonaws.com`.
Useful for S3-compatible services (MinIO, Ceph, LocalStack).
- secure
Use HTTPS. Default is 1 (true). Set to 0 only for testing against
local S3-compatible endpoints.
- timeout
HTTP request timeout in seconds. Default is 30.
## Credential resolution order
When no `credentials` object is passed, credentials are resolved in
this order:
1. Constructor arguments `aws_access_key_id` and `aws_secret_access_key`.
2. Environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
and optionally `AWS_SESSION_TOKEN`.
3. [Amazon::Credentials](https://metacpan.org/pod/Amazon%3A%3ACredentials), if installed. This covers IAM instance roles,
Lambda execution roles, ECS task roles, and `~/.aws/credentials`
profiles.
4. If none of the above yield credentials, the constructor croaks.
# METHODS
All methods croak on unrecoverable errors (network failure, HTTP 5xx).
HTTP 404 is not an exception - methods that can meaningfully return
`undef` for a missing resource do so.
## list\_objects\_v2
my $result = $s3->list_objects_v2($bucket, %options);
Lists objects in `$bucket` using the S3 ListObjectsV2 API.
Options:
- prefix
Limit results to keys beginning with this string.
- delimiter
Group keys sharing a common prefix up to this delimiter. Grouped
prefixes are returned in `common_prefixes`.
- max\_keys
Maximum number of objects to return per call (1-1000, default 1000).
- continuation\_token
Resume a truncated listing from a prior call's
`next_continuation_token`.
- start\_after
Return only keys lexicographically after this value.
Returns a hashref:
{
bucket => 'my-bucket',
prefix => 'logs/',
is_truncated => 0,
next_continuation_token => undef, # set when is_truncated is true
key_count => 42,
objects => [
{
key => 'logs/2024-01-01.gz',
size => 102400,
last_modified => '2024-01-01T00:00:00.000Z',
etag => 'abc123',
storage_class => 'STANDARD',
},
...
],
common_prefixes => [], # populated when delimiter is set
}
## list\_all\_objects\_v2
my @objects = $s3->list_all_objects_v2($bucket, %options);
Convenience wrapper around ["list\_objects\_v2"](#list_objects_v2) that automatically
follows continuation tokens and returns a flat list of all matching
object hashrefs in a single call.
Accepts the same options as `list_objects_v2` except
`continuation_token` (which is managed internally) and `delimiter`
(which is silently ignored - see below).
my @logs = $s3->list_all_objects_v2('my-bucket', prefix => 'logs/');
foreach my $obj (@logs) {
printf "%s %d bytes\n", $obj->{key}, $obj->{size};
}
Be mindful of memory when listing buckets with large numbers of
objects. For very large listings, use ["list\_objects\_v2"](#list_objects_v2) directly
and process each page as it arrives.
`delimiter` and `common_prefixes` are not supported by this method.
The purpose of `list_all_objects_v2` is a complete flat listing of
all matching keys. Hierarchical directory-style traversal using
`delimiter` is inherently page-by-page and should use
["list\_objects\_v2"](#list_objects_v2) directly.
Returns a (possibly empty) list of object hashrefs, each with the same
fields as the elements of `objects` in the `list_objects_v2`
response.
- log\_level
Log level for the internal logger. Accepted values: `trace`, `debug`,
`info`, `warn`, `error`, `fatal`. Default is `warn`. Only consulted
when no `logger` object is supplied and Log::Log4perl is not available
or not yet initialized.
## get\_object
my $obj = $s3->get_object($bucket, $key);
my $obj = $s3->get_object($bucket, $key, %options);
Fetches the object at `$key` in `$bucket`.
Returns `undef` if the key does not exist (HTTP 404).
Returns a hashref on success:
{
content => '...', # raw bytes; absent when filename is used
content_type => 'application/json',
content_length => 1024,
etag => 'abc123',
last_modified => 'Tue, 01 Jan 2024 00:00:00 GMT',
metadata => { # x-amz-meta-* headers, lowercased
source => 'lambda',
},
}
Options:
- range
An HTTP Range header value, e.g. `bytes=0-1023`, for partial fetches.
- filename
Path to a local file where the object body should be written. When
supplied, the response body is streamed directly to disk via
HTTP::Tiny's `:content_file` mechanism and `content` is omitted from
the returned hashref. The file is created or overwritten.
my $meta = $s3->get_object('my-bucket', 'data/dump.csv',
filename => '/tmp/dump.csv',
);
# $meta->{content} is absent; file is on disk
This is the recommended approach for large objects in Lambda where
holding the full body in memory is undesirable.
## head\_object
my $meta = $s3->head_object($bucket, $key);
Fetches metadata for `$key` without retrieving the object body.
Useful for existence checks and reading `x-amz-meta-*` headers
cheaply.
Returns `undef` if the key does not exist (HTTP 404).
Returns a hashref on success with the same fields as `get_object`
except `content`, which is always absent.
## put\_object
$s3->put_object($bucket, $key, $data, %options);
Stores `$data` at `$key` in `$bucket`. `$data` may be:
- A scalar string (the object body verbatim)
- A reference to a scalar (avoids copying large strings)
- An open filehandle or [IO::File](https://metacpan.org/pod/IO%3A%3AFile) object (body is read to EOF)
When passing a filehandle, `content_length` becomes required unless
HTTP::Tiny can determine the size from the handle (i.e. the handle is
backed by a real file). For in-memory handles (`IO::Scalar`, etc.)
you must supply `content_length` explicitly, or the method will
croak.
# Scalar
$s3->put_object('my-bucket', 'hello.txt', 'Hello, world!',
content_type => 'text/plain',
);
# Filehandle
open my $fh, '<', '/tmp/data.csv' or die $!;
$s3->put_object('my-bucket', 'data.csv', $fh,
content_type => 'text/csv',
);
Options:
- content\_type
MIME type for the object. Defaults to `application/octet-stream`.
- content\_length
Required when `$data` is an in-memory filehandle. Optional (and
ignored) for scalar data, where length is computed automatically.
- metadata
Hashref of user-defined metadata. Keys should be bare names - the
`x-amz-meta-` prefix is added automatically.
metadata => { source => 'lambda', job_id => '42' }
- acl
Canned ACL string, e.g. `private` (default), `public-read`.
Returns the ETag of the stored object on success. Croaks on failure.
## copy\_object
$s3->copy_object(
src_bucket => 'src-bucket',
src_key => 'original/key.json',
share/README.md view on Meta::CPAN
}
elsif ( $cfg->{queue_arn} ) {
printf "SQS: id=%s arn=%s\n", $cfg->{id}, $cfg->{queue_arn};
}
print " events: ", join(', ', @{ $cfg->{events} }), "\n";
}
Retrieves the current notification configuration for `$bucket`.
Handles both Lambda (`CloudFunctionConfiguration`) and SQS
(`QueueConfiguration`) entries, which are the XML element names
the S3 API returns regardless of how the configuration was created.
Returns an arrayref of configuration hashrefs, each containing:
- id
The configuration entry identifier.
- lambda\_arn
The Lambda function ARN. Present for Lambda notification entries;
`undef` for SQS entries.
- queue\_arn
The SQS queue ARN. Present for SQS notification entries;
`undef` for Lambda entries.
- events
Arrayref of event type strings.
- filters
Arrayref of hashrefs, each with `name` (`prefix` or `suffix`)
and `value`.
Returns an empty arrayref if no notification configuration is set.
Croaks on failure.
## remove\_bucket\_notification\_configuration
$s3->remove_bucket_notification_configuration($bucket);
Removes all notification configurations from `$bucket` by sending an
empty `NotificationConfiguration` document to S3. After this call S3
will no longer deliver any events for the bucket.
Returns true on success. Croaks on failure.
# ERROR HANDLING
Methods croak on:
- Network-level failures (connection refused, timeout, DNS failure)
- HTTP 5xx responses from S3
- Unexpected HTTP 3xx responses that could not be resolved
Methods return `undef` on:
- HTTP 404 (key or bucket not found), where the return type allows it
All other HTTP error codes (400, 403, 409, etc.) cause a croak with a
message containing the HTTP status line and the S3 error body where
available.
# DEPENDENCIES
- [HTTP::Tiny](https://metacpan.org/pod/HTTP%3A%3ATiny) (core since Perl 5.14)
- [Amazon::Signature4::Lite](https://metacpan.org/pod/Amazon%3A%3ASignature4%3A%3ALite)
- [XML::Twig](https://metacpan.org/pod/XML%3A%3ATwig) (for parsing list and copy responses)
- [Digest::MD5](https://metacpan.org/pod/Digest%3A%3AMD5) (core, for Content-MD5 headers)
- [MIME::Base64](https://metacpan.org/pod/MIME%3A%3ABase64) (core)
- [URI::Escape](https://metacpan.org/pod/URI%3A%3AEscape)
- [Carp](https://metacpan.org/pod/Carp) (core)
Optional:
- [Amazon::Credentials](https://metacpan.org/pod/Amazon%3A%3ACredentials) - automatic credential discovery from IAM
roles, ECS task roles, ~/.aws/credentials, and environment.
- [Log::Log4perl](https://metacpan.org/pod/Log%3A%3ALog4perl) - structured logging; if present, used in
preference to the built-in minimal logger.
# LAMBDA USAGE NOTES
In a Lambda container, credentials come from the execution role via
the ECS credential provider endpoint (indicated by
`AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` in the environment).
[Amazon::Credentials](https://metacpan.org/pod/Amazon%3A%3ACredentials) handles this automatically when installed and
is the recommended approach. If you prefer not to take that
dependency, the Lambda runtime also populates `AWS_ACCESS_KEY_ID`,
`AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` directly, which
this module picks up automatically from the environment.
**Region note:** The `list_buckets` method is a global S3 operation
and is always signed against `us-east-1`, regardless of the region
supplied to the constructor. This is an S3 requirement, not a
limitation of this module, and is handled transparently - your
object's region is not changed.
**Cold start:** Because this module depends only on [HTTP::Tiny](https://metacpan.org/pod/HTTP%3A%3ATiny) (Perl
core), [XML::Twig](https://metacpan.org/pod/XML%3A%3ATwig), [AWS::Signature4](https://metacpan.org/pod/AWS%3A%3ASignature4), and [URI::Escape](https://metacpan.org/pod/URI%3A%3AEscape), it adds
minimal overhead to Lambda container image builds compared to
LWP-based S3 clients.
# TESTING
When testing against LocalStack, be aware that LocalStack is more
lenient than real S3 regarding SigV4 requirements. In particular,
LocalStack may accept requests where the `x-amz-content-sha256`
header is missing or where session token handling is incorrect. Tests
that pass against LocalStack should always be verified against real S3
before release.
# SEE ALSO
[Amazon::S3](https://metacpan.org/pod/Amazon%3A%3AS3) - the full-featured S3 client this module draws from
[Amazon::S3::Thin](https://metacpan.org/pod/Amazon%3A%3AS3%3A%3AThin) - another excellent lightweight S3 client with a
similar philosophy, broader feature coverage, and a longer track
record. Uses LWP by default and returns raw [HTTP::Response](https://metacpan.org/pod/HTTP%3A%3AResponse)
( run in 1.169 second using v1.01-cache-2.11-cpan-788537b7465 )