App-ElasticSearch-Utilities

 view release on metacpan or  search on metacpan

README.mkdn  view on Meta::CPAN

## es\_indices\_meta

Returns the hash of index meta data.

## es\_indices

Returns a list of active indexes matching the filter criteria specified on the command
line.  Can handle indices named:

    logstash-YYYY.MM.DD
    dcid-logstash-YYYY.MM.DD
    logstash-dcid-YYYY.MM.DD
    logstash-YYYY.MM.DD-dcid

Makes use of --datesep to determine where the date is.

Options include:

- **state**

    Default is 'open', can be used to find 'closed' indexes as well.

- **check\_state**

lib/App/ElasticSearch/Utilities.pm  view on Meta::CPAN

=head2 es_indices_meta

Returns the hash of index meta data.

=head2 es_indices

Returns a list of active indexes matching the filter criteria specified on the command
line.  Can handle indices named:

    logstash-YYYY.MM.DD
    dcid-logstash-YYYY.MM.DD
    logstash-dcid-YYYY.MM.DD
    logstash-YYYY.MM.DD-dcid

Makes use of --datesep to determine where the date is.

Options include:

=over 4

=item B<state>

Default is 'open', can be used to find 'closed' indexes as well.

lib/App/ElasticSearch/Utilities/QueryString/IP.pm  view on Meta::CPAN

sub _build_priority { 25 }


sub handle_token {
    my ($self,$token) = @_;
    if( my ($term,$match) = split /\:/, $token, 2 ) {
        # These are not 100% accurate IP matchers, but they are fast
        if(     $match =~ m|^\d{1,3}(?:\.\d{1,3}){1,3}(?:/\d+)$|
            or  $match =~ m|^[0-9a-fA-F:]+(?:/\d+)$|
        ) {
            my $cidr = Net::CIDR::Lite->new();
            $cidr->add($match);
            my @range = split /-/, ($cidr->list_range)[0];
            return { condition => { range => { $term => { gte => $range[0], lte => $range[1] } } } };
        }
    }
    return;
}

1;

__END__

scripts/es-copy-index.pl  view on Meta::CPAN

        category:'(authentication authorization)'

=head2 Changing settings and mappings

   es-copy-index.pl --from localhost \
        --source logstash-2013.01.11 \
        --destination testing-new-settings-old-data-2013.01.11 \
        --settings new_settings.json \
        --mappings new_mappings.json

=head2 Building an Incident Index using append

Let's say we were investigating an incident and wanted to have
an index that contained the data we were interested in.  We could use different
retention rules for incident indexes and we could arbitrarily add data to them based
on searches being performed on the source index.

Here's our initial query, a bad actor on our admin login page.

   es-copy-index.pl --from localhost \
        --source logstash-2013.01.11 \
        --destination incident-rt1234-2013.01.11 \
        src_ip:1.2.3.4 dst:admin.exmaple.com and file:'\/login.php'

Later on, we discover there was another actor:

   es-copy-index.pl --from localhost \
        --source logstash-2013.01.11 \
        --destination incident-rt1234-2013.01.11 \
        --append \
        src_ip:4.3.2.1 dst:admin.exmaple.com and file:'\/login.php'

The B<incident-rt1234-2013.01.11> index will now hold all the data from both of those queries.

=head1 Query Syntax Extensions

The search string is pre-analyzed before being sent to ElasticSearch.  The following plugins
work to manipulate the query string and provide richer, more complete syntax for CLI applications.

=head2 App::ElasticSearch::Utilities::QueryString::Barewords

The following barewords are transformed:

t/01-querystring.t  view on Meta::CPAN

                'must' => [
                    {
                        'query_string' => {
                            'query' => 'src_ip:1.2.3.4 AND NOT dst:www.example.com'
                        }
                    }
                ]
            }
        },
    ],
    '01-ip-cidr-expansion' => [
        [qw(src_ip:10.0.0.0/8)],
        {
          'bool' => {
            'must' => [
              {
                'range' => {
                  'src_ip' => { gte => '10.0.0.0', lte => '10.255.255.255' }
                }
              }
            ]

t/02-index-data.t  view on Meta::CPAN

foreach my $days_old ( @days_old ) {
    # Query String Parser Testing
    my $lt = $now->clone->subtract( days => $days_old );
    my $date = $lt->strftime('%Y.%m.%d');
    my %tests = (
        "type-$date" => {
            es_index_bases      => 'type',
            es_index_days_old   => $days_old,
            es_index_strip_date => 'type',
        },
        "type-dcid-$date" => {
            es_index_bases      => 'type,type-dcid',
            es_index_days_old   => $days_old,
            es_index_strip_date => 'type-dcid',
        },
        "type_dcid_$date" => {
            es_index_bases      => 'type,type_dcid',
            es_index_days_old   => $days_old,
            es_index_strip_date => 'type_dcid',
        },
        "type_dcid_$date-0001" => {
            es_index_bases      => 'type,type_dcid',
            es_index_days_old   => $days_old,
            es_index_strip_date => 'type_dcid',
        },
    );
    # Install the test globally
    foreach my $t (keys %tests) {
        $TESTS{$t} = $tests{$t};
    }
}


foreach my $t (sort keys %TESTS) {