view release on metacpan or  search on metacpan

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

		scalar  => \@scalar,
		expand  => \@expand,
		combine => \@combine,
		},
		"${class}::Plan";
} ## end sub compile

=head2 known_mungers

    my @names = ...->known_mungers;

The sorted list of built-in munger names this version understands.

=head2 has_munger

    if ( ...->has_munger('enum') ) { ... }

True if the named munger is built in.

=cut

sub known_mungers { my @names = sort keys %BUILDERS; return @names }
sub has_munger    { return exists $BUILDERS{ $_[1] } }

=head1 BUILT-IN MUNGERS

Every munger returns a plain number and, where the input cannot be interpreted,
croaks -- the Writer would reject a non-numeric field anyway, so failing at the
munger gives a better message. Parameters are validated when the munger is
built, not per row.

=head2 enum

    { munger => 'enum', map => { GET => 0, POST => 1 }, default => -1 }

Categorical string to number via an explicit C<map>. All map values must be
numeric. Without a C<default>, an unmapped input croaks; with one, unmapped
inputs (including C<undef>) yield the default.

=cut

sub _build_enum {
	my ( $spec, $where ) = @_;

	my $map = $spec->{map};
	croak "enum munger$where requires a 'map' hashref"
		unless ref $map eq 'HASH';

	for my $k ( keys %$map ) {
		croak "enum munger$where: map value for '$k' ('"
			. ( defined $map->{$k} ? $map->{$k} : 'undef' )
			. "') is not numeric"
			unless looks_like_number( $map->{$k} );
	}

	my $has_default = exists $spec->{default};
	my $default     = $spec->{default};
	croak "enum munger$where: 'default' must be numeric"
		if $has_default && !looks_like_number($default);

	# Copy so a later edit of the caller's spec cannot mutate a live munger.
	my %m = %$map;
	return sub {
		my ($v) = @_;
		return $m{$v}   if defined $v && exists $m{$v};
		return $default if $has_default;
		croak "enum munger$where: no mapping for '" . ( defined $v ? $v : 'undef' ) . "'";
	};
} ## end sub _build_enum

=head2 frozen_freq_map

    { munger => 'frozen_freq_map', counts => { jpg => 40213, exe => 12, scr => 3 },
      total => 67560 }
    # defaults: mode => 'neg_log_prob', smoothing => 1, unseen => 'rare'

Frequency-encoding from a B<precomputed, frozen> count table: the rarer a value
was when the table was built, the more anomalous it scores. This is C<enum>'s
cousin -- a value-to-number map -- except the numbers are derived from observed
C<counts> rather than hand-authored, with the smoothing and unseen-value policy
that "rare = interesting" needs. It stays a stateless munger: the table is
computed offline and shipped in C<info.json>; this class only I<applies> it.

C<counts> maps each value to how many times it was seen. C<total> is the overall
observation count; it defaults to the sum of C<counts>, but may be given
explicitly and larger so you can B<prune the long tail> out of C<counts> while
still computing correct probabilities. The emitted number depends on C<mode>:

=over 4

=item * C<neg_log_prob> (default) - self-information C<-ln(prob)>: rare values
score high, common ones low. This is the axis "rare = interesting" describes and
what an Isolation Forest splits on most naturally.

=item * C<freq> - the probability itself, C<(count + smoothing) / denom>.

=item * C<log_count> - C<ln(1 + count)>, the count with its heavy tail tamed.

=item * C<count> - the raw count.

=back

Probabilities use add-one style C<smoothing> (default C<1>), treating "unseen" as
one aggregate bucket: C<prob(v) = (count + smoothing) / (total + smoothing*(V+1))>
where C<V> is the number of listed values. C<unseen> controls what a value absent
from the table maps to -- C<'rare'> (default) emits that value under the current
mode as if it had been seen zero times (for C<neg_log_prob>/C<freq> this is the
smoothed unseen bucket, for C<count>/C<log_count> it is C<0>), or a number to
force a fixed default. Because an unseen value is usually the very thing you are
hunting, mapping it to "maximally rare" rather than erroring is the point.

C<frozen_freq_map> only suits B<bounded, moderate-cardinality> columns (extensions,
vendor classes, named pipes, keyboard layouts, link addresses): the table lives
in C<info.json>, so a huge one bloats every read -- building one past
C<$Algorithm::ToNumberMunger::FROZEN_FREQ_MAP_WARN_KEYS>
values (default 10000) warns. For unbounded cardinality (JA3, full user-agents)
use L</hash> instead, which needs no table but keeps only decorrelation, not
commonness.

=cut