view release on metacpan or  search on metacpan

lib/App/ElasticSearch/Utilities.pm  view on Meta::CPAN

=head1 NAME

App::ElasticSearch::Utilities - Utilities for Monitoring ElasticSearch

=head1 VERSION

version 8.9

=head1 SYNOPSIS

This distribution contains modules for interacting with ElasticSearch and
OpenSearch and utility scripts.

=head1 SCRIPTS

This a set of utilities to make monitoring ElasticSearch clusters much simpler.

=head2 SEARCHING

    scripts/es-aggregate.pl - Utility to search and aggregate index contents
    scripts/es-search.pl - Utility to search and explore index contents

=head2 MONITORING

    scripts/es-graphite-dynamic.pl - Perform index maintenance on daily indexes
    scripts/es-index-fields.pl - Collect and report on field statistics for indices
    scripts/es-index-scan.pl - Scan for potential index issues
    scripts/es-nodes.pl - View node information
    scripts/es-status.pl - Command line utility for ES Metrics
    scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster

=head2 MAINTENANCE

    scripts/es-alias-manager.pl - Manage index aliases automatically
    scripts/es-daily-index-maintenance.pl - Perform index maintenance on daily indexes
    scripts/es-index-blocks.pl - Report and fix any blocks on indices
    scripts/es-open.pl - Open any closed indices matching a index parameters

=head2 MANAGEMENT

    scripts/es-apply-settings.pl - Apply settings to all indexes matching a pattern
    scripts/es-cluster-settings.pl - Manage cluster settings
    scripts/es-copy-index.pl - Copy an index from one cluster to another
    scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster

The App::ElasticSearch::Utilities module simply serves as a wrapper around the scripts for packaging and
distribution.

=head2 USAGE

The tools are all wrapped in their own documentation, please see:

    $UTILITY --help
    $UTILITY --manual

For individual options and capabilities

=head2 PATTERNS

Patterns are used to match an index to the aliases it should have.  A few symbols are expanded into
regular expressions.  Those patterns are:

    *       expands to match any number of any characters.
    DATE    expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD
    ANY     expands to match any number of any characters.

=head1 CONFIG FILES

Some options may be specified in the B</etc/es-utils.yaml>, B<$HOME/.es-utils.yaml>
or B<$HOME/.config/es-utils/config.yaml> file:

    ---
    base: logstash
    days: 7
    host: esproxy.example.com
    port: 80
    timeout: 10
    proto: https
    http-username: bob
    password-exec: /home/bob/bin/get-es-passwd.sh

=head1 ARGS

From App::ElasticSearch::Utilities:

    --local         Use localhost as the elasticsearch host
    --host          ElasticSearch host to connect to
    --port          HTTP port for your cluster
    --proto         Defaults to 'http', can also be 'https'
    --http-username HTTP Basic Auth username
    --password-exec Script to run to get the users password
    --insecure      Don't verify TLS certificates
    --cacert        Specify the TLS CA file
    --capath        Specify the directory with TLS CAs
    --cert          Specify the path to the client certificate
    --key           Specify the path to the client private key file
    --noop          Any operations other than GET are disabled, can be negated with --no-noop
    --timeout       Timeout to ElasticSearch, default 10
    --keep-proxy    Do not remove any proxy settings from %ENV
    --index         Index to run commands against
    --base          For daily indexes, reference only those starting with "logstash"
                     (same as --pattern logstash-* or logstash-DATE)
    --pattern       Use a pattern to operate on the indexes
    --days          If using a pattern or base, how many days back to go, default: 1

See also the "CONNECTION ARGUMENTS" and "INDEX SELECTION ARGUMENTS" sections from App::ElasticSearch::Utilities.

=head1 CONNECTION ARGUMENTS

Arguments for establishing a connection with the cluster.  Unless specified otherwise, these options
can all be set in the globals file.

=over

=item B<local>

Assume ElasticSearch is running locally, connect to localhost.

=item B<host>

Use a different hostname or IP address to connect.

=item B<port>

Defaults to 9200.

lib/App/ElasticSearch/Utilities.pm  view on Meta::CPAN

If we save this to "$HOME/bin/get-passwd.sh" we can execute a script
like this:

    $ es-search.pl --http-username bob --password-exec $HOME/bin/get-passwd.sh \
                    --base secure-data --fields

Though it's probably best to set this in your ~/.es-utils.yml file:

    ---
    host: secured-cluster.example.org
    port: 443
    proto: https
    http-username: bob
    password-exec: /home/bob/bin/get-passwd.sh

=head3 CLI::Helpers and Password Prompting

If all the fails to yield a password, the last resort is to use CLI::Helpers::prompt() to ask the user for their
password.  If the user is using version 1.1 or higher of CLI::Helpers, this call will turn off echo and readline magic
for the password prompt.

=head1 INDEX SELECTION ARGUMENTS

=over

=item B<base>

In an environment using monthly, weekly, daily, or hourly indexes.  The base index name is everything without the date.
Parsing for bases, also provides splitting and matching on segments of the index name delineated by the '-' character.
If we have the following indexes:

    web-dc1-YYYY.MM.DD
    web-dc2-YYYY.MM.DD
    logstash-dc1-YYYY.MM.DD
    logstash-dc2-YYYY.MM.DD

Valid bases would be:

    web
    web-dc1
    web-dc2
    logstash
    logstash-dc1
    logstash-dc2
    dc1
    dc2

Combining that with the days option can provide a way to select many indexes at once.

=item B<days>

How many days backwards you want your operation to be relevant.

=item B<datesep>

Default is '.' Can be set to an empty string for no separator.

=item B<pattern>

A pattern to match the indexes.  Can expand the following key words and characters:

    '*'    expanded to '.*'
    'ANY'  expanded to '.*'
    'DATE' expanded to a pattern to match a date,

The indexes are compared against this pattern.

=back

=head1 OVERVIEW

In addition to the scripts, the libraries provide a simplistic interface to
write your own scripts. It builds C<CLI::Helpers> to provide consistent options
for scripts.

    use App::ElasticSearch::Utilities qw(:all);
    use Data::Printer;

    my $res = es_result('_cluster/health');
    p($res)

See the contents of the scripts for examples.

=head1 EXPORT

This module use L<Sub::Exporter> so you can customize exports.

=head2 Export Groups

The following export groups are provided.

=over 2

=item B<:default> - Default exports

    es_connect()
    es_indices()
    es_request()
    es_utils_initialize()

=item B<:config>

    es_globals()
    es_utils_initialize()

=item B<:human>

    es_format_numeric()
    es_human_count()
    es_human_size()

=item B<:index>

    :default
    es_index_bases()
    es_index_days_old()
    es_index_fields()
    es_index_shards()
    es_index_segments()
    es_index_stats()
    es_index_strip_date()
    es_index_valid()

=item B<:indices>