Algorithm-ToNumberMunger

 view release on metacpan or  search on metacpan

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

    my $code = ...->build( \%spec );
    my $code = ...->build( \%spec, $tag_name );   # $tag_name only sharpens errors

Compile a single munger spec into a coderef. C<%spec> must contain a C<munger>
key naming one of the L</BUILT-IN MUNGERS>; the remaining keys are that munger's
parameters. Croaks on an unknown munger name or an invalid parameter set. The
optional second argument is only used to make error messages point at a tag.

=cut

# name => builder. Each builder validates its slice of the spec up front and
# returns the per-value closure. Keeping them in a table (rather than a big
# if/elsif) is what makes known_mungers() and has_munger() cheap and honest.
my %BUILDERS = (
	enum            => \&_build_enum,
	frozen_freq_map => \&_build_frozen_freq_map,
	bool            => \&_build_bool,
	length          => \&_build_length,
	entropy         => \&_build_entropy,
	ngram           => \&_build_ngram,
	char            => \&_build_char,

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

=item * a B<combining> munger, keyed by its output tag and carrying a C<from>
B<list> (C<< from => ['bytes_out', 'bytes_in'] >>), reads several source
fields and fills that one column -- this is how a ratio becomes a single
feature without precomputing it upstream (see L</ratio> and L</combine>). The
sources are raw input fields, not other (possibly munged) columns;

=item * every remaining tag is B<raw> and passed through unchanged.

=back

Coverage is validated up front: C<compile> croaks if two mungers write the same
column, if an C<into> names a column not in C<tags>, if a munger key is neither a
tag nor an expander, if an expander's output count does not match its C<into>,
or if a C<from> list is given to a munger that cannot combine inputs. The
returned plan has two methods, both returning an arrayref of numbers in C<tags>
order: C<apply_named(\%hash)> (keyed by field name, the only form that supports
expanders and combiners) and C<apply_positional(\@row)> (positional; croaks if
the set has any expanding or combining munger, since a shared or combined
source cannot be expressed by position).

=cut

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

=cut

sub known_mungers { my @names = sort keys %BUILDERS; return @names }
sub has_munger    { return exists $BUILDERS{ $_[1] } }

=head1 BUILT-IN MUNGERS

Every munger returns a plain number and, where the input cannot be interpreted,
croaks -- the Writer would reject a non-numeric field anyway, so failing at the
munger gives a better message. Parameters are validated when the munger is
built, not per row.

=head2 enum

    { munger => 'enum', map => { GET => 0, POST => 1 }, default => -1 }

Categorical string to number via an explicit C<map>. All map values must be
numeric. Without a C<default>, an unmapped input croaks; with one, unmapped
inputs (including C<undef>) yield the default.

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

		return $v;
	};
} ## end sub _build_clamp

=head2 num

    { munger => 'num', base => 16 }        # '0x1a' or '1a' -> 26
    { munger => 'num' }                    # plain numeric coercion

Parse a string as a number in C<base> (2-36, default 10). Base 10 simply
validates and numifies. Other bases accept the digits C<0-9a-z> below the
base, case-insensitively, an optional leading C<->, and the conventional
prefix for that base (C<0x> for 16, C<0b> for 2, C<0o> for 8). Plenty of
tooling logs flag words and IDs in hex (C<0x2f>), which the Writer would
reject as non-numeric; this munger is the bridge. Croaks on anything that is
not a clean number in the chosen base.

=cut

sub _build_num {
	my ( $spec, $where ) = @_;

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

the empty string. A true C<ignore_case> matches case-insensitively; the
L</match> trust note applies here too.

=item * C<replace> - replace every match of the regex C<pattern> with the
literal string C<with> (default: delete the matches). C<ignore_case> as above.

=back

C<then> is a full munger spec and may be any built-in that takes one value --
including another C<chain>. All step parameters and the terminal spec are
validated at build time. An undef input enters the chain as the empty string;
whether an empty result is acceptable is the terminal's call (C<entropy> and
C<length> score it C<0>, C<num> croaks).

The multi-output form works too: put C<into> on the B<chain> and the C<parts>
on the terminal, e.g. C<trim> a sloppy timestamp before a L</datetime>
C<sin>/C<cos> expansion.

=cut

# op => step builder; each validates its slice of the step spec at build time
# and returns a string-to-string closure. Steps only ever see a defined string
# (the chain entry point turns undef into '').
my %CHAIN_OPS = (
	lc => sub {
		return sub { return lc $_[0] }
	},
	uc => sub {
		return sub { return uc $_[0] }
	},
	trim => sub {

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

			croak "eps munger$where: $err" if $on_error eq 'die';
			return ( $on_error + 0 ) x $n;
		}
		return @replies[@take];
	}; ## end $code = sub
	return ( $code, $n );
} ## end sub _build_eps_multi

# A compiled munging plan for one set, produced by Mungers->compile. It turns an
# input record into a fully-numeric row in tags order; the Writer then only has
# to validate and append. Kept in its own package so the assembly logic is
# testable without a Writer or the filesystem.
package Algorithm::ToNumberMunger::Plan;

use strict;
use warnings;
use Carp qw(croak);

sub tags { return $_[0]->{tags} }

# Assemble a row from a name-keyed record. Scalar/raw columns read their own tag

t/mungers.t  view on Meta::CPAN

	is( $sipm->('REGISTER'), 4,  'sip_method_enum REGISTER' );
	is( $sipm->('update'),   13, 'sip_method_enum update, case-insensitive' );

	my $dhcp = $M->build( { munger => 'dhcp_msgtype_enum' } );
	is( $dhcp->('DISCOVER'),     1, 'dhcp_msgtype_enum DISCOVER' );
	is( $dhcp->('DHCPDISCOVER'), 1, 'dhcp_msgtype_enum DHCP-prefixed form' );
	is( $dhcp->('nak'),          6, 'dhcp_msgtype_enum nak, case-insensitive' );
	is( $dhcp->(8),              8, 'dhcp_msgtype_enum passes numeric input through' );

	eval { $M->build( { munger => 'dns_rcode_enum', default => 'nope' } ) };
	like( $@, qr/'default' must be numeric/, 'named-map enum validates default at build time' );
}

# ---- scale ----------------------------------------------------------------
{
	my $c = $M->build( { munger => 'scale', min => 0, max => 10 } );
	is( $c->(5),  0.5, 'scale midpoint' );
	is( $c->(15), 1.5, 'scale unclamped overshoot' );

	my $cl = $M->build( { munger => 'scale', min => 0, max => 10, clamp => 1 } );
	is( $cl->(15), 1, 'scale clamps high' );

t/mungers.t  view on Meta::CPAN

	eval { $c->('not-an-ip') };
	like( $@, qr/not a parseable IP address/, 'ip_class croaks on garbage without default' );
	eval { $c->('10.0.0.256') };
	like( $@, qr/not a parseable IP address/, 'ip_class rejects out-of-range octets' );

	my $d = $M->build( { munger => 'ip_class', default => -1 } );
	is( $d->('not-an-ip'), -1, 'ip_class default for garbage' );
	is( $d->(undef),       -1, 'ip_class default for undef' );
	eval { $M->build( { munger => 'ip_class', default => 'x' } ) };
	like( $@, qr/'default' must be numeric/, 'ip_class validates default at build time' );
}

# ---- cidr -------------------------------------------------------------------
{
	my $c = $M->build(
		{
			munger  => 'cidr',
			nets    => [ '10.10.0.0/16', '10.0.0.0/8', '2001:db8:5::/48' ],
			default => -1,
		}