view release on metacpan or  search on metacpan

lib/Algorithm/ToNumberMunger.pm  view on Meta::CPAN

	my $V = keys %$counts;
	carp "frozen_freq_map munger$where: 'counts' has $V keys; a table this large bloats "
		. "info.json -- consider the 'hash' munger for unbounded cardinality"
		if $V > $FROZEN_FREQ_MAP_WARN_KEYS;

	my $total = defined $spec->{total} ? $spec->{total} : $sum;
	croak "frozen_freq_map munger$where: 'total' must be numeric"
		unless looks_like_number($total);
	croak "frozen_freq_map munger$where: 'total' ($total) must be >= sum of counts ($sum)"
		if $total < $sum;

	my $mode = defined $spec->{mode} ? $spec->{mode} : 'neg_log_prob';
	croak "frozen_freq_map munger$where: unknown mode '$mode' (known: " . join( ', ', sort keys %FREQ_MODE ) . ')'
		unless $FREQ_MODE{$mode};

	my $s = defined $spec->{smoothing} ? $spec->{smoothing} : 1;
	croak "frozen_freq_map munger$where: 'smoothing' must be a non-negative number"
		unless looks_like_number($s) && $s >= 0;

	my $unseen = defined $spec->{unseen} ? $spec->{unseen} : 'rare';
	croak "frozen_freq_map munger$where: 'unseen' must be 'rare' or a number"
		unless $unseen eq 'rare' || looks_like_number($unseen);

	# An unseen value under neg_log_prob has probability s/denom; with no
	# smoothing that is 0 and -ln(0) is infinite, which would poison the column.
	# Refuse to build rather than emit inf.
	croak "frozen_freq_map munger$where: mode 'neg_log_prob' with unseen => 'rare' needs "
		. "smoothing > 0 (an unseen value would otherwise be infinitely surprising)"
		if $mode eq 'neg_log_prob' && $unseen eq 'rare' && $s == 0;

	# Smoothed-probability denominator, treating "unseen" as one extra bucket.
	my $denom = $total + $s * ( $V + 1 );

	# raw count -> emitted number under the chosen mode.
	my $emit_for = sub {
		my ($c) = @_;
		return $c            if $mode eq 'count';
		return log( 1 + $c ) if $mode eq 'log_count';
		my $p = ( $c + $s ) / $denom;
		return $p if $mode eq 'freq';
		return -log($p);    # neg_log_prob
	};

	my %emit         = map { $_ => $emit_for->( $counts->{$_} ) } keys %$counts;
	my $unseen_value = $unseen eq 'rare' ? $emit_for->(0) : $unseen;

	return sub {
		my ($v) = @_;
		return defined $v && exists $emit{$v} ? $emit{$v} : $unseen_value;
	};
} ## end sub _build_frozen_freq_map

=head2 http_enum

    { munger => 'http_enum' }
    { munger => 'http_enum', strict => 1 }

Collapse an HTTP status code to its class: C<1xx> to C<1>, C<2xx> to C<2>, C<3xx>
to C<3>, and so on (i.e. C<int(code / 100)>). This is the usual bucketing for an
HTTP status column -- the forest cares far more about "was this a 4xx vs a 2xx"
than about C<403> vs C<404>, and it keeps the feature low-cardinality without
having to spell out every code in an C<enum> C<map>. The input must be numeric.

By default any numeric input is bucketed, so a bogus C<700> would quietly become
C<7>. With a true C<strict>, inputs outside the valid HTTP status range
(C<100>-C<599>) croak instead, so a malformed code is caught at write time rather
than smuggled into the model as a spurious class.

=head2 smtp_enum

    { munger => 'smtp_enum' }
    { munger => 'smtp_enum', strict => 1 }

The SMTP counterpart of L</http_enum>: collapse an SMTP reply code to its leading
digit (C<int(code / 100)>), since that digit I<is> the reply's meaning -- C<2yz>
completion, C<3yz> intermediate, C<4yz> transient failure, C<5yz> permanent
failure. As with C<http_enum> this keeps the column low-cardinality and lets the
forest weigh "a 5xx where a 2xx was expected" without enumerating every code.

With a true C<strict>, inputs outside the valid SMTP reply range (C<200>-C<599>)
croak. SMTP never issues C<1yz> replies in practice (no command permits a
positive-preliminary reply), so the strict floor is C<200> rather than
C<http_enum>'s C<100>.

=head2 sip_enum

    { munger => 'sip_enum' }
    { munger => 'sip_enum', strict => 1 }

The SIP counterpart of L</http_enum>: collapse a SIP status code to its leading
digit (C<int(code / 100)>). SIP reuses HTTP's class scheme but adds a sixth
class -- C<1xx> provisional, C<2xx> success, C<3xx> redirection, C<4xx> client
error, C<5xx> server error, C<6xx> global failure.

With a true C<strict>, inputs outside the valid SIP status range (C<100>-C<699>)
croak. The ceiling is C<699> rather than C<http_enum>'s C<599> precisely because
of that C<6xx> global-failure class.

=head2 ftp_enum

    { munger => 'ftp_enum' }
    { munger => 'ftp_enum', strict => 1 }

The FTP counterpart of L</http_enum>, for FTP reply codes: C<int(code / 100)>,
bucketing into C<1yz>-C<5yz>. With a true C<strict>, inputs outside C<100>-C<599>
croak.

=head2 rtsp_enum

    { munger => 'rtsp_enum' }
    { munger => 'rtsp_enum', strict => 1 }

The RTSP counterpart of L</http_enum>. RTSP (RFC 2326) deliberately reuses
HTTP's status scheme, so codes collapse to their leading digit the same way.
With a true C<strict>, inputs outside C<100>-C<599> croak.

=head2 nntp_enum

    { munger => 'nntp_enum' }
    { munger => 'nntp_enum', strict => 1 }