view release on metacpan or  search on metacpan

scripts/es-copy-index.pl  view on Meta::CPAN

B<REQUIRED>: name of the source index for the copy

=item B<destination>

Optional: change the name of the index on the destination cluster

=item B<block>

Batch size of docs to process in one retrieval, default is 1,000

=item B<mapping>

Path to a file containing JSON mapping to use on the destination index
instead of the mapping directly from the source index.

=item B<settings>

Path to a file containing JSON settings to use on the destination index
instead of the settings directly from the source index.

=item B<append>

This mode skips the index mapping and settings configuration and just being indexing
documents from the source into the destination.

=item B<help>

Print this message and exit

=item B<manual>

Print detailed help with examples

=back

=head1 EXAMPLES

=head2 Copy to different cluster

   es-copy-index.pl --from localhost --to remote.cluster.com --source logstash-2013.01.11

=head2 Rename an existing index

   es-copy-index.pl --from localhost --source logstash-2013.01.11 --destination logs-2013.01.11

=head2 Subset an existing index

   es-copy-index.pl --from localhost \
        --source logstash-2013.01.11 \
        --destination secure-2013.01.11 \
        category:'(authentication authorization)'

=head2 Changing settings and mappings

   es-copy-index.pl --from localhost \
        --source logstash-2013.01.11 \
        --destination testing-new-settings-old-data-2013.01.11 \
        --settings new_settings.json \
        --mappings new_mappings.json

=head2 Building an Incident Index using append

Let's say we were investigating an incident and wanted to have
an index that contained the data we were interested in.  We could use different
retention rules for incident indexes and we could arbitrarily add data to them based
on searches being performed on the source index.

Here's our initial query, a bad actor on our admin login page.

   es-copy-index.pl --from localhost \
        --source logstash-2013.01.11 \
        --destination incident-rt1234-2013.01.11 \
        src_ip:1.2.3.4 dst:admin.exmaple.com and file:'\/login.php'

Later on, we discover there was another actor:

   es-copy-index.pl --from localhost \
        --source logstash-2013.01.11 \
        --destination incident-rt1234-2013.01.11 \
        --append \
        src_ip:4.3.2.1 dst:admin.exmaple.com and file:'\/login.php'

The B<incident-rt1234-2013.01.11> index will now hold all the data from both of those queries.

=head1 Query Syntax Extensions

The search string is pre-analyzed before being sent to ElasticSearch.  The following plugins
work to manipulate the query string and provide richer, more complete syntax for CLI applications.

=head2 App::ElasticSearch::Utilities::QueryString::Barewords

The following barewords are transformed:

    or => OR
    and => AND
    not => NOT

=head2 App::ElasticSearch::Utilities::QueryString::Text

Provides field prefixes to manipulate the text search capabilities.

=head3 Terms Query via '='

Provide an '=' prefix to a query string parameter to promote that parameter to a C<term> filter.

This allows for exact matches of a field without worrying about escaping Lucene special character filters.

E.g.:

    user_agent:"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1"

Is evaluated into a weird query that doesn't do what you want.   However:

    =user_agent:"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1"

Is translated into:

    { term => { user_agent => "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1" } }

=head3 Wildcard Query via '*'

Provide an '*' prefix to a query string parameter to promote that parameter to a C<wildcard> filter.

This uses the wild card match for text fields to making matching more intuitive.

E.g.:

    *user_agent:"Mozilla*"

Is translated into:

    { wildcard => { user_agent => "Mozilla* } }

=head3 Regexp Query via '/'

Provide an '/' prefix to a query string parameter to promote that parameter to a C<regexp> filter.

If you want to use regexp matching for finding data, you can use:

    /message:'\\bden(ial|ied|y)'

Is translated into: