view release on metacpan or  search on metacpan

README.mkdn  view on Meta::CPAN

    scripts/es-aggregate.pl - Utility to search and aggregate index contents
    scripts/es-search.pl - Utility to search and explore index contents

## MONITORING

    scripts/es-graphite-dynamic.pl - Perform index maintenance on daily indexes
    scripts/es-index-fields.pl - Collect and report on field statistics for indices
    scripts/es-index-scan.pl - Scan for potential index issues
    scripts/es-nodes.pl - View node information
    scripts/es-status.pl - Command line utility for ES Metrics
    scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster

## MAINTENANCE

    scripts/es-alias-manager.pl - Manage index aliases automatically
    scripts/es-daily-index-maintenance.pl - Perform index maintenance on daily indexes
    scripts/es-index-blocks.pl - Report and fix any blocks on indices
    scripts/es-open.pl - Open any closed indices matching a index parameters

## MANAGEMENT

    scripts/es-apply-settings.pl - Apply settings to all indexes matching a pattern
    scripts/es-cluster-settings.pl - Manage cluster settings
    scripts/es-copy-index.pl - Copy an index from one cluster to another
    scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster

The App::ElasticSearch::Utilities module simply serves as a wrapper around the scripts for packaging and
distribution.

## USAGE

The tools are all wrapped in their own documentation, please see:

    $UTILITY --help
    $UTILITY --manual

For individual options and capabilities

## PATTERNS

Patterns are used to match an index to the aliases it should have.  A few symbols are expanded into
regular expressions.  Those patterns are:

    *       expands to match any number of any characters.
    DATE    expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD
    ANY     expands to match any number of any characters.

# CONFIG FILES

Some options may be specified in the **/etc/es-utils.yaml**, **$HOME/.es-utils.yaml**
or **$HOME/.config/es-utils/config.yaml** file:

    ---
    base: logstash
    days: 7
    host: esproxy.example.com
    port: 80
    timeout: 10
    proto: https
    http-username: bob
    password-exec: /home/bob/bin/get-es-passwd.sh

# ARGS

From App::ElasticSearch::Utilities:

    --local         Use localhost as the elasticsearch host
    --host          ElasticSearch host to connect to
    --port          HTTP port for your cluster
    --proto         Defaults to 'http', can also be 'https'
    --http-username HTTP Basic Auth username
    --password-exec Script to run to get the users password
    --insecure      Don't verify TLS certificates
    --cacert        Specify the TLS CA file
    --capath        Specify the directory with TLS CAs
    --cert          Specify the path to the client certificate
    --key           Specify the path to the client private key file
    --noop          Any operations other than GET are disabled, can be negated with --no-noop
    --timeout       Timeout to ElasticSearch, default 10
    --keep-proxy    Do not remove any proxy settings from %ENV
    --index         Index to run commands against
    --base          For daily indexes, reference only those starting with "logstash"
                     (same as --pattern logstash-* or logstash-DATE)
    --pattern       Use a pattern to operate on the indexes
    --days          If using a pattern or base, how many days back to go, default: 1

See also the "CONNECTION ARGUMENTS" and "INDEX SELECTION ARGUMENTS" sections from App::ElasticSearch::Utilities.

# CONNECTION ARGUMENTS

Arguments for establishing a connection with the cluster.  Unless specified otherwise, these options
can all be set in the globals file.

- **local**

    Assume ElasticSearch is running locally, connect to localhost.

- **host**

    Use a different hostname or IP address to connect.

- **port**

    Defaults to 9200.

- **proto**

    Defaults to 'http', can also be 'https'.

- **http-username**

    If HTTP Basic Authentication is required, use this username.

    See also the ["HTTP Basic Authentication"](#http-basic-authentication) section for more details

- **password-exec**

    If HTTP Basic Authentication is required, run this command, passing the arguments:

        <command_to_run> <es_host> <es_username>

    The script expects the last line to contain the password in plaintext.

- **noop**

    Prevents any communication to the cluster from making changes to the settings or data contained therein.
    In short, it prevents anything but HEAD and GET requests, **except** POST requests to the \_search endpoint.

- **timeout**

    Timeout for connections and requests, defaults to 10.

- **keep-proxy**

    By default, HTTP proxy environment variables are stripped. Use this option to keep your proxy environment variables
    in tact.

- **insecure**

    Don't verify TLS certificates

- **cacert**

    Specify a file with the TLS CA certificates.

- **capath**

    Specify a directory containing the TLS CA certificates.

- **cert**

    Specify the path to the TLS client certificate file..

- **key**

    Specify the path to the TLS client private key file.

# AUTHENTICATION

HTTP Basic Authorization is only supported when the `proto` is set to **https**
as not to leak credentials all over.

The username is selected by going through these mechanisms until one is found:

    --http-username
    'http-username' in /etc/es-utils.yml or ~/.es-utils.yml
    Netrc element matching the hostname of the request
    CLI::Helpers prompt()

Once the username has been resolved, the following mechanisms are tried in order:

    Netrc element matching the hostname of the request
    Password executable defined by --password-exec
    'password-exec' in /etc/es-utils.yml, ~/.es-utils.yml
    CLI::Helpers prompt()

## Password Exec

It is **BAD** practice to specify passwords as a command line argument, or store it in a plaintext
file.  There are cases where this may be necessary, but it is not recommended.  The best method for securing  your
password is to use the **password-exec** option.

This option must point to an executable script.  That script will be passed two arguments, the hostname and the username
for the request.  It expects the password printed to STDOUT as the last line of output.  Here's an example password-exec setup
using Apple Keychain:

    #!/bin/sh

    HOSTNAME=$1;
    USERNAME=$2;

    /usr/bin/security find-generic-password -w -a "$USERNAME" -s "$HOSTNAME"

If we save this to "$HOME/bin/get-passwd.sh" we can execute a script
like this:

    $ es-search.pl --http-username bob --password-exec $HOME/bin/get-passwd.sh \
                    --base secure-data --fields

Though it's probably best to set this in your ~/.es-utils.yml file:

    ---
    host: secured-cluster.example.org
    port: 443
    proto: https
    http-username: bob
    password-exec: /home/bob/bin/get-passwd.sh

### CLI::Helpers and Password Prompting

If all the fails to yield a password, the last resort is to use CLI::Helpers::prompt() to ask the user for their
password.  If the user is using version 1.1 or higher of CLI::Helpers, this call will turn off echo and readline magic
for the password prompt.

# INDEX SELECTION ARGUMENTS

- **base**

    In an environment using monthly, weekly, daily, or hourly indexes.  The base index name is everything without the date.
    Parsing for bases, also provides splitting and matching on segments of the index name delineated by the '-' character.
    If we have the following indexes:

        web-dc1-YYYY.MM.DD
        web-dc2-YYYY.MM.DD
        logstash-dc1-YYYY.MM.DD
        logstash-dc2-YYYY.MM.DD

    Valid bases would be:

        web
        web-dc1
        web-dc2
        logstash
        logstash-dc1
        logstash-dc2
        dc1
        dc2

    Combining that with the days option can provide a way to select many indexes at once.

- **days**

    How many days backwards you want your operation to be relevant.

- **datesep**

    Default is '.' Can be set to an empty string for no separator.

- **pattern**

    A pattern to match the indexes.  Can expand the following key words and characters:

        '*'    expanded to '.*'
        'ANY'  expanded to '.*'
        'DATE' expanded to a pattern to match a date,

    The indexes are compared against this pattern.

# OVERVIEW

In addition to the scripts, the libraries provide a simplistic interface to
write your own scripts. It builds `CLI::Helpers` to provide consistent options
for scripts.

    use App::ElasticSearch::Utilities qw(:all);
    use Data::Printer;

    my $res = es_result('_cluster/health');
    p($res)

See the contents of the scripts for examples.

# EXPORT

README.mkdn  view on Meta::CPAN

        :default
        es_index_bases()
        es_index_days_old()
        es_index_fields()
        es_index_shards()
        es_index_segments()
        es_index_stats()
        es_index_strip_date()
        es_index_valid()

- **:indices**

        :default
        es_indices_meta()

- **:maintenance**

        :default
        es_close_index()
        es_open_index()
        es_delete_index()
        es_optimize_index()
        es_apply_index_settings()

- **:all** - All exportable functions

        :config
        :default
        :index
        :indices
        :human
        :maintenance
        es_basic_auth()
        es_flatten_hash()
        es_local_index_meta()
        es_master()
        es_nodes()
        es_node_stats()
        es_pattern()
        es_settings()
        es_segment_stats()

## Configuration

It is possible to control how and when `@ARGV` is processed to prevent conflicts.

# CONFIG FUNCTIONS

## es\_utils\_initialize()

Takes an optional reference to an `@ARGV` like array. Performs environment and
argument parsing.

## es\_globals($key)

Grab the value of the global value from the es-utils.yaml files.

## es\_basic\_auth($host)

Get the user/password combination for this host.  This is called from LWP::UserAgent if
it recieves a 401, so the auth condition must be satisfied.

Returns the username and password as a list.

# CONNECTION FUNCTIONS

## es\_connect

Without options, this connects to the server defined in the args.  If passed
an array ref, it will use that as the connection definition.

## es\_request(\[$handle\],$command,{ method => 'GET', uri\_param => { a => 1 } }, {})

Retrieve URL from ElasticSearch, returns a hash reference

First hash ref contains options, including:

    uri_param           Query String Parameters
    index               Index name
    type                Index type
    method              Default is GET

If the request is not successful, this function will throw a fatal exception.
If you'd like to proceed you need to catch that error.

# INDEX FUNCTIONS

## es\_index\_strip\_date( 'index-name' )

Returns the index name with the date removed.

## es\_index\_bases( 'index-name' )

Returns an array of the possible index base names for this index

## es\_index\_days\_old( 'index-name' )

Return the number of days old this index is.

## es\_index\_shards( 'index-name' )

Returns the number of replicas for a given index.

## es\_index\_valid( 'index-name' )

Checks if the specified index is valid

## es\_index\_fields('index-name')

Returns a hash reference with the following data:

    key_name:
      type: field_data_type
      # If the field is nested
      nested_path: nested_path
      nested_key: nested_key

## es\_index\_segments( 'index-name' )

Exposes GET /$index/\_segments

Returns the segment data from the index in hashref:

## es\_index\_stats( 'index-name' )

Exposes GET /$index/\_stats

Returns a hashref

# INDICES FUNCTIONS

## es\_indices\_meta

Returns the hash of index meta data.

# MAINTENANCE FUNCTIONS

## es\_close\_index('index-name')

Closes an index

## es\_open\_index('index-name')

Open an index

## es\_delete\_index('index-name')

Deletes an index

## es\_optimize\_index('index-name')

Optimize an index to a single segment per shard

## es\_apply\_index\_settings('index-name', { settings })

Apply a HASH of settings to an index.

# FORMAT FUNCTIONS

## es\_human\_count

Takes a number and returns the number as a string in docs, thousands, millions, or billions.

    1_000     -> "1.00 thousand",
    1_000_000 -> "1.00 million",

## es\_human\_size

Takes a number and returns the number as a string in bytes, Kb, Mb, Gb, or Tb using base 1024.

    1024        -> '1.00 Kb',
    1048576     -> '1.00 Mb',
    1073741824  -> '1.00 Gb',

## es\_format\_numeric

Takes a value and the minimum digits of significance.

# FUNCTIONS

## es\_pass\_exec(host, username)

Called from es\_basic\_auth to exec a program, capture the password
and return it to the caller.  This allows the use of password vaults
and keychains.

## es\_pattern

Returns a hashref of the pattern filter used to get the indexes
    {
        string => '\*',
        re     => '.\*',
    }

## es\_master(\[$handle\])

Returns true (1) if the handle is to the the cluster master, or false (0) otherwise.

## es\_nodes

Returns the hash of index meta data.

## es\_indices

Returns a list of active indexes matching the filter criteria specified on the command
line.  Can handle indices named:

    logstash-YYYY.MM.DD
    dcid-logstash-YYYY.MM.DD
    logstash-dcid-YYYY.MM.DD
    logstash-YYYY.MM.DD-dcid

Makes use of --datesep to determine where the date is.

Options include:

- **state**

    Default is 'open', can be used to find 'closed' indexes as well.

- **check\_state**

    Default is 1, set to 0 to disable state checks.  The combination of the default
    with this option and the default for **state** means only open indices are returned.

- **check\_dates**

    Default is 1, set to 0 to disable checking index age.

## es\_segment\_stats($index)

Return the number of shards and segments in an index as a hashref

## es\_settings()

Exposes GET /\_settings

Returns a hashref

## es\_node\_stats()

Exposes GET /\_nodes/stats

Returns a hashref