view release on metacpan or  search on metacpan

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

package Algorithm::ToNumberMunger;

use 5.006;
use strict;
use warnings;

use Carp         qw(carp croak);
use Scalar::Util qw(looks_like_number);

=head1 NAME

Algorithm::ToNumberMunger - Compile declarative specs into closures that munge raw values into numbers.

=head1 VERSION

Version 0.0.1

=cut

our $VERSION = '0.0.1';

# Feature hashing (the 'hash' munger) is a tight per-byte FNV-1a loop with a
# 32-bit modular multiply. That is exactly the kind of work XS is good at and
# pure Perl is bad at (both for speed and, on a 32-bit perl, for correctness of
# the wrap-around), so we compile it in C when we can. Everything else here is a
# hash lookup or a couple of flops -- crossing the XS boundary per row would
# only make those slower, so they stay pure Perl. If the XS did not build (no
# compiler at install time) we fall back to a pure-Perl FNV-1a, which is exact
# on a 64-bit perl.
our $HAVE_XS = 0;
eval {
	require XSLoader;
	XSLoader::load( __PACKAGE__, $VERSION );
	$HAVE_XS = 1;
	1;
};

=head1 SYNOPSIS

    use Algorithm::ToNumberMunger;

    # one munger from a spec hash
    my $code = Algorithm::ToNumberMunger->build(
        { munger => 'enum', map => { GET => 0, POST => 1, PUT => 2 } },
    );
    my $n = $code->('POST');          # 1

    # a whole table of them at once, from a 'field => spec' hash
    my $by_tag = Algorithm::ToNumberMunger->build_all(
        \%mungers,
    );
    my $row_value = $by_tag->{method}->($raw{method});

=head1 DESCRIPTION

Many numeric pipelines -- anomaly detectors, feature stores, CSV loaders --
want every column to be a number, but the values they are handed are not always
numbers to begin with: an HTTP method is a string, a timestamp is a formatted
date, a high-cardinality label wants bucketing. An B<input munger> turns such a
raw value into a single number. Munging happens on the input side, before a row
is stored.

Mungers are declared as a plain data spec -- a hash naming a built-in munger and
carrying that munger's parameters -- so a table of them can be read straight out
of JSON or a config file:

    {
        "method": { "munger": "enum",  "map": { "GET": 0, "POST": 1 } },
        "bytes":  { "munger": "log",   "offset": 1 },
        "label":  { "munger": "hash",  "buckets": 1024 }
    }

B<Any field without an entry is raw> and is passed through unchanged; this module
is only concerned with fields that name a munger.

This class does not read or write files. It B<compiles> a spec into a closure
that maps one raw value to one number, so a caller can build its mungers once
from configuration and then apply them per row with no re-parsing. All
configuration errors are caught at build time; the returned closure only croaks
on genuinely un-mungeable I<input>.

=head1 CLASS METHODS

=head2 build

    my $code = ...->build( \%spec );
    my $code = ...->build( \%spec, $tag_name );   # $tag_name only sharpens errors

Compile a single munger spec into a coderef. C<%spec> must contain a C<munger>
key naming one of the L</BUILT-IN MUNGERS>; the remaining keys are that munger's
parameters. Croaks on an unknown munger name or an invalid parameter set. The
optional second argument is only used to make error messages point at a tag.

=cut

# name => builder. Each builder validates its slice of the spec up front and
# returns the per-value closure. Keeping them in a table (rather than a big
# if/elsif) is what makes known_mungers() and has_munger() cheap and honest.
my %BUILDERS = (
	enum            => \&_build_enum,
	frozen_freq_map => \&_build_frozen_freq_map,
	bool            => \&_build_bool,
	length          => \&_build_length,
	entropy         => \&_build_entropy,

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

Named-map enum (lookup rules as L</dns_rcode_enum>; numeric inputs pass
through) mapping DNS RR type names to their IANA numbers: C<A> 1, C<NS> 2,
C<CNAME> 5, C<SOA> 6, C<NULL> 10, C<PTR> 12, C<MX> 15, C<TXT> 16, C<AAAA> 28,
C<SRV> 33, C<NAPTR> 35, C<DS> 43, C<RRSIG> 46, C<DNSKEY> 48, C<TLSA> 52,
C<SVCB> 64, C<HTTPS> 65, C<AXFR> 252, C<ANY> (or C<*>) 255, C<URI> 256,
C<CAA> 257, and the rest of the commonly-observed registry. The query-type mix
is a classic DNS-tunneling feature -- C<TXT>/C<NULL>-heavy traffic where
C<A>/C<AAAA> is normal.

=head2 syslog_severity_enum

    { munger => 'syslog_severity_enum' }

Named-map enum (lookup rules as L</dns_rcode_enum>; numeric inputs pass
through) mapping syslog severity names to their RFC 5424 codes: C<emerg> 0
(alias C<panic>), C<alert> 1, C<crit> 2, C<err> 3 (alias C<error>),
C<warning> 4 (alias C<warn>), C<notice> 5, C<info> 6 (alias
C<informational>), C<debug> 7. Genuinely ordinal -- lower is more severe --
so a threshold split on it is meaningful.

=head2 syslog_facility_enum

    { munger => 'syslog_facility_enum' }

Named-map enum (lookup rules as L</dns_rcode_enum>; numeric inputs pass
through) mapping syslog facility names to their RFC 5424 codes: C<kern> 0,
C<user> 1, C<mail> 2, C<daemon> 3, C<auth> 4 (alias C<security>), C<syslog> 5,
C<lpr> 6, C<news> 7, C<uucp> 8, C<cron> 9, C<authpriv> 10, C<ftp> 11, C<ntp>
12, C<audit> 13, C<alert> 14, C<clock> 15, and C<local0>-C<local7> 16-23.

=head2 ip_proto_enum

    { munger => 'ip_proto_enum', default => -1 }

Named-map enum (lookup rules as L</dns_rcode_enum>; numeric inputs pass
through) mapping IP protocol names to their IANA protocol numbers: C<icmp> 1,
C<igmp> 2, C<ipip> 4 (alias C<ipencap>), C<tcp> 6, C<egp> 8, C<udp> 17,
C<dccp> 33, C<ipv6> 41, C<rsvp> 46, C<gre> 47, C<esp> 50, C<ah> 51,
C<icmpv6> 58 (alias C<ipv6-icmp>), C<ospf> 89, C<pim> 103, C<sctp> 132,
C<udplite> 136. The map is frozen here rather than delegated to
C<getprotobyname> so a value munges to the same number on every host.

=head2 tls_version_enum

    { munger => 'tls_version_enum', default => -1 }

Named-map enum (lookup rules as L</dns_rcode_enum>) mapping a TLS/SSL protocol
version name to an B<ordinal>: C<SSLv2> 0, C<SSLv3> 1, C<TLSv1> 2, C<TLSv1.1>
3, C<TLSv1.2> 4, C<TLSv1.3> 5, with the common spelling variants (C<ssl3>,
C<tls1.2>, ...) accepted. Ordinal so "older than expected" is a monotone
feature a threshold split can express. Because these ordinals are this
module's invention rather than a wire encoding, numeric inputs are B<not>
passed through -- a C<1.2> would land on the wrong scale -- and croak like
any other unmapped value (or take the C<default>).

=head2 http_method_enum

    { munger => 'http_method_enum', default => -1 }

Named-map enum (lookup rules as L</dns_rcode_enum>) for the registered HTTP
request methods: C<GET> 0, C<HEAD> 1, C<POST> 2, C<PUT> 3, C<DELETE> 4,
C<CONNECT> 5, C<OPTIONS> 6, C<TRACE> 7, C<PATCH> 8. HTTP has no numeric
method encoding, so these are unordered ordinals of this module's invention
(a canonical map beats every set inventing its own numbering) and numeric
inputs are not passed through. An unlisted -- possibly probing -- method
croaks unless a C<default> is given, and that unlisted-method signal is often
the interesting one.

=head2 sip_method_enum

    { munger => 'sip_method_enum', default => -1 }

Named-map enum (lookup rules as L</dns_rcode_enum>) for the SIP request
methods: C<INVITE> 0, C<ACK> 1, C<BYE> 2, C<CANCEL> 3, C<REGISTER> 4,
C<OPTIONS> 5, C<PRACK> 6, C<SUBSCRIBE> 7, C<NOTIFY> 8, C<PUBLISH> 9,
C<INFO> 10, C<REFER> 11, C<MESSAGE> 12, C<UPDATE> 13. Like
L</http_method_enum> these are ordinals of this module's invention, so
numeric inputs are not passed through.

=head2 dhcp_msgtype_enum

    { munger => 'dhcp_msgtype_enum' }

Named-map enum (lookup rules as L</dns_rcode_enum>; numeric inputs pass
through) mapping DHCP message-type names to their option-53 values:
C<DISCOVER> 1, C<OFFER> 2, C<REQUEST> 3, C<DECLINE> 4, C<ACK> 5, C<NAK> 6,
C<RELEASE> 7, C<INFORM> 8 -- each also accepted with the C<DHCP> prefix
(C<DHCPDISCOVER>, ...) that most tooling logs.

=head2 app_proto_enum

    { munger => 'app_proto_enum', default => -1 }

Named-map enum (lookup rules as L</dns_rcode_enum>) for Suricata's
C<app_proto> field -- the detected application-layer protocol on a flow or
alert (C<http>, C<dns>, C<tls>, C<ssh>, C<smtp>, C<dcerpc>, C<quic>, ...),
including C<failed> and C<unknown>, which are usually the very rows worth
keeping. These are unordered ordinals of this module's invention (Suricata
logs a string, not a number), so numeric inputs are B<not> passed through.
C<ssl> is accepted as an alias for C<tls> and C<ikev2> for C<ike>. This is
distinct from L</ip_proto_enum>, which numbers the L4 protocol
(C<tcp>/C<udp>/...) rather than the app layer riding on it.

=head2 tcp_state_enum

    { munger => 'tcp_state_enum', default => -1 }

Named-map enum (lookup rules as L</dns_rcode_enum>) mapping the TCP state
machine, as Suricata logs it under C<flow.tcp.state>, to an B<ordinal> along
the connection lifecycle: C<none> 0, C<syn_sent> 1, C<syn_recv> 2,
C<established> 3, C<fin_wait1> 4, C<fin_wait2> 5, C<closing> 6, C<time_wait>
7, C<close_wait> 8, C<last_ack> 9, C<closed> 10. Ordinal so "further along
teardown than expected" is a monotone feature a threshold split can express.
Being ordinals of our own invention, numeric inputs are not passed through.

=head2 flow_state_enum

    { munger => 'flow_state_enum', default => -1 }

Named-map enum (lookup rules as L</dns_rcode_enum>) for Suricata's
C<flow.state>: C<new> 0, C<established> 1, C<closed> 2, C<bypassed> 3,