Algorithm-Classifier-IsolationForest
view release on metacpan or search on metacpan
the voting method can be switched between majority and mean.
- adjust how iforest info displays tag info
0.4.0 2026-07-03/22:45
- Implement named features and methods for testing single rows using
tagged data.
- The C backend can be built and installed at install time, meaning
nothing needs built unless changing the opts. See the docs for
more details.
- new IF_NO_OPENMP=1 selects/builds the serial C backend: no
libgomp linkage and no OpenMP runtime in the process at all
(unlike OMP_NUM_THREADS=1, which just caps the thread count);
IF_NO_OPENMP=0 re-enables OpenMP over a serial install default
- new IF_RUNTIME_BUILD=1 ignores the prebuilt object and forces
the classic runtime build even when the flags match
- iforest accel updated to reflect various changes to the C backend
- scoring: the per-leaf path-length adjustment c(size) is now
precomputed at tree-pack time and stored in the (previously
unused) third slot of packed leaf records, removing a log()
call per point per tree from the C scoring hot loop -- about
25% faster axis-mode scoring; results are bit-identical
Makefile.PL view on Meta::CPAN
# When Inline::C is usable at configure time (and IF_NO_C isn't set), the
# generated Makefile gains a rule that compiles the Inline::C backend once
# during `make` and installs the shared object with the distribution, so
# users never pay the first-load compile and need neither a C compiler nor
# the Inline modules at run time.
#
# The IF_* environment variables (IF_OPT, IF_ARCH, IF_NATIVE, IF_NO_OPENMP)
# are captured *now* -- set them when running `perl Makefile.PL`, not
# `make` -- and recorded in the generated
# Algorithm::Classifier::IsolationForest::BuildFlags module. At run time
# those recorded values are the defaults: a process requesting the same
# flags loads the prebuilt object via XSLoader; one requesting different
# flags (or setting IF_RUNTIME_BUILD=1) falls back to the classic runtime
# Inline::C build under _Inline/. Validation mirrors the module's own,
# since these values reach a compiler command line.
# ---------------------------------------------------------------------------
my $if_opt = '-O3';
if ( defined $ENV{IF_OPT} ) {
if ( $ENV{IF_OPT} =~ /\A-O[0123sgz]\z/ ) {
$if_opt = $ENV{IF_OPT};
} else {
benchmarking/bench-fit-parallel.pl view on Meta::CPAN
#!/usr/bin/perl
# benchmarking/bench-fit-parallel.pl
#
# Measures fit() throughput across a sweep of parallel_fit worker
# counts. The training data is the same for every run; only the
# `parallel_fit` constructor option changes.
#
# parallel_fit forks workers and divvies up n_trees across them, with
# each worker fitting its share in a child process (via the C tree
# builder when Inline::C is available, same as a non-parallel fit) and
# returning the trees to the parent via Storable on a pipe. Speedup is
# bounded by:
# * core count
# * Storable freeze/thaw + pipe IPC overhead per worker
# * fork() cost (small but non-zero)
#
# With the C builder, a single-process fit is often fast enough that
# this fixed per-worker overhead costs more than splitting the work
# saves -- watch for parallel_fit making things *slower* on small/
# medium datasets in the sweep below.
#
# Run with:
# perl -Ilib benchmarking/bench-fit-parallel.pl
use strict;
use warnings;
use lib '../lib';
benchmarking/bench-sklearn-scoring.pl view on Meta::CPAN
results[label] = {
'score_samples': bench(lambda X=X_q: clf.score_samples(X), bench_secs),
'predict': bench(lambda X=X_q: clf.predict(X), bench_secs),
'decision_function': bench(lambda X=X_q: clf.decision_function(X), bench_secs),
}
print(json.dumps(results))
END_PY
# -----------------------------------------------------------------------
# Run Python (one subprocess for all experiments)
# -----------------------------------------------------------------------
my $sk;
if ( defined $python_bin ) {
my ( $py_fh, $py_path ) = tempfile( SUFFIX => '.py', UNLINK => 1 );
print $py_fh $py_script;
close $py_fh;
my $specs = join( ' ', map { qq("$_->{train_csv}|$_->{query_csv}|@{[exp_key($_)]}") } @experiments );
my $raw = `$python_bin "$py_path" $BENCH_SECS $specs 2>/dev/null`;
$sk = eval { JSON::PP->new->decode($raw) };
benchmarking/bench-streamd.pl view on Meta::CPAN
#!/usr/bin/perl
# benchmarking/bench-streamd.pl
#
# Benchmarks `iforest streamd` end to end: the script spawns its own
# daemon on a temp Unix socket and pumps rows through the JSON-lines
# protocol, measuring points/second wall-clock at the client.
#
# Sections:
# 1. in-process baseline -- score_learn on an identical model in this
# process; the ceiling everything else is measured against. The
# gap between it and the socket numbers is protocol + JSON + IPC
# overhead, not model work.
# 2. batch-size sweep -- prequential rows per {"rows": [...]}
# message; directly informs streamc's --batch choice.
# 3. modes -- prequential vs score vs learn at a fixed
# batch size.
# 4. row forms -- positional arrays vs tagged objects (the
# tagged form pays hashref building + tagged_row_to_array).
# 5. concurrent clients -- total throughput with 1/2/4 connections
# pumping at once. The daemon is a single select loop sharing one
# model, so this should stay ~flat: it measures fairness overhead,
# not parallel speedup.
# 6. command latency -- ping round trips (pure protocol floor)
# and the wall cost of an on-demand save.
#
# Reference numbers (2026-07-08, 8-core dev box, C backend,
# Cpanel::JSON::XS, 100 trees, window 2048, eta 32, 5 features):
# in-process score_learn ~2,770 pts/s; over the socket ~2,700 pts/s at
# any batch >= 16 (~2% overhead) and ~2,500 pts/s even at batch 1;
# score mode ~29,000 pts/s (no learning -- the tree walk is cheap, the
# learn is what costs); tagged objects ~21,000 vs positional ~30,000 in
# score mode; throughput is flat across 1/2/4 concurrent clients (one
# shared model, one loop -- by design); ping ~29,000 round trips/s;
# save ~140 ms. The wire is nowhere near the bottleneck -- the model
# is.
#
# Run with:
# perl -Ilib benchmarking/bench-streamd.pl
benchmarking/bench-streamd.pl view on Meta::CPAN
# every section measures steady-state, full-window work.
srand(42);
my $warm = make_data( 3000, $NF );
srand(43);
my $feed = make_data( 60000, $NF );
my $c = connect_daemon();
pump( $c, $warm, 500, 'learn' );
# -----------------------------------------------------------------------
# 1. In-process baseline: the same stream loop without the wire
# -----------------------------------------------------------------------
print "\n--- 1. in-process score_learn baseline (no socket, no JSON) ---\n";
my $base = Algorithm::Classifier::IsolationForest::Online->new(
n_trees => $N_TREES,
window_size => $WINDOW,
max_leaf_samples => $ETA,
seed => 42,
);
$base->learn($warm);
{
my @rows = @{$feed}[ 0 .. 9999 ];
my $t0 = time;
examples/online-streaming.pl view on Meta::CPAN
# online-streaming.pl
#
# Online (streaming) Isolation Forest on a drifting stream. The stream starts
# as a Gaussian blob at the origin, then drifts to a blob at (6, 6). An
# offline model would keep flagging the new regime forever; the online model
# forgets points as they age out of its sliding window, so within one window
# of the drift it treats the new regime as normal and the OLD regime as the
# anomaly.
#
# Points are processed prequentially (score-then-learn), the standard way to
# evaluate a streaming detector: every score reflects the model as it stood
# before that point influenced it.
#
# Run from the distribution root:
# perl -Ilib examples/online-streaming.pl
# or, if the module is installed:
# perl examples/online-streaming.pl
use strict;
use warnings;
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
pivot = a[hi];
i = lo;
for (j = lo; j < hi; j++) {
if (a[j] < pivot) { _dswap(&a[i], &a[j]); i++; }
}
_dswap(&a[i], &a[hi]);
return i;
}
/* Quickselect: returns the k-th smallest (0-indexed) of a[0..n-1],
* reordering a[] in the process (fine -- it's a private scratch copy).
* O(n) average case vs. a full O(n log n) sort. */
static double _kth_smallest(double *a, int n, int k) {
int lo = 0, hi = n - 1;
while (lo < hi) {
int p = _partition_lomuto(a, lo, hi);
if (p == k) return a[p];
if (p < k) lo = p + 1; else hi = p - 1;
}
return a[lo];
}
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
# use_c => 1 and use_c => 0 build bit-identical trees (one ulp in a
# split value cascades into a structurally different tree). The
# -march speedup comes from AVX2 vectorization, not contraction,
# so this costs little (verified against the fit-determinism and
# scoring-parity tests).
my $opt_level = $opt;
$opt_level .= " -march=$arch -ffp-contract=off" if length $arch;
# IF_NO_OPENMP=1 forces the serial C build: the OpenMP compile attempt
# is skipped, so the object has no libgomp linkage and never starts an
# OpenMP runtime in the process. Distinct from OMP_NUM_THREADS=1,
# which runs the parallel code on a single thread but still loads
# libgomp. An explicit IF_NO_OPENMP=0 re-enables OpenMP over a
# no-openmp configure-time default.
my $no_omp
= defined $ENV{IF_NO_OPENMP}
? ( $ENV{IF_NO_OPENMP} ? 1 : 0 )
: $def_no_omp;
# The prebuilt object is only trusted when the effective flags match
# what it was compiled with; any difference -- or an explicit
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
my $reloaded = Algorithm::Classifier::IsolationForest->load('model.json');
# Extended Isolation Forest (oblique hyperplane splits)
my $eif = Algorithm::Classifier::IsolationForest->new(
mode => 'extended',
seed => 42,
);
$eif->fit(\@data);
# Parallel training (fork-based, Unix-like platforms): build the
# n_trees across several worker processes.
my $iforest = Algorithm::Classifier::IsolationForest->new(
n_trees => 200,
sample_size => 256,
seed => 42,
parallel_fit => 4, # 4 forked workers
);
$iforest->fit(\@data);
# Pre-pack a dataset to skip the per-call input-walk cost when the
# same data gets scored many times (interactive tuning, dashboards).
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
each stored value to double precision to preserve this parity; axis
mode matches exactly, while extended mode can still differ on rare
libm rounding ties (double vs long-double transcendentals).
By default this C builder is single-threaded per call, because Perl's
RNG state isn't safe to share across OpenMP threads. Two ways to scale
fit() across cores are available (see below for why they don't compose):
=over 4
=item * C<parallel_fit> forks N worker processes, each building its
share of the trees with the (still single-threaded) C builder. Fixed
IPC/serialisation overhead per worker means this can cost more than it
saves once a fit already completes in milliseconds; it's most useful
once a single-process fit is large enough that the fork/Storable
overhead is small relative to the work being split.
=item * C<use_openmp_fit> builds trees across OpenMP threads within a
single process (one tree per thread), using a separate, thread-safe
PRNG seeded per tree index instead of Perl's C<rand()>. This means
trees built with C<use_openmp_fit> are I<not> bit-identical to the
default C<use_c> path for the same seed -- but a fixed seed and
C<n_trees> still reproduce the same trees regardless of
C<OMP_NUM_THREADS> or how OpenMP schedules the work. It's off by
default (unlike C<use_c>/C<use_openmp>, which only ever change speed,
this changes which trees get built) and only takes effect when C<use_c>
is also on and OpenMP is linked in.
=back
These two do NOT compose, despite both existing to parallelise fit().
A process that has run any OpenMP region -- including plain
C<score_samples()>/C<predict()> with the default C<use_openmp> -- and
then C<fork()>s (as C<parallel_fit> does) hands each child a copy of
libgomp's thread pool whose worker threads did not survive the fork. A
child that then starts its own C<#pragma omp parallel> region (as
C<use_openmp_fit> would) tries to reuse that now-invalid pool and
hangs. This is a general limitation of combining C<fork()> with OpenMP,
not something fixable from Perl, so C<parallel_fit>'s forked workers
always use the single-threaded C builder regardless of
C<use_openmp_fit> -- setting both just means C<parallel_fit> wins and
C<use_openmp_fit> has no effect for that call.
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
benchmark with C<iforest bench> before assuming) while avoiding the
C<-march=native> caveats described under L</Tuning the C build>.
Bit-for-bit result parity with the pure-Perl backend is preserved
either way (see C<IF_ARCH> below).
The C<IF_*> build flags described below are captured when
C<perl Makefile.PL> runs -- set them in the environment of I<that>
command, not of C<make> -- and recorded in the generated
C<Algorithm::Classifier::IsolationForest::BuildFlags> module, which
thereby also fixes what the prebuilt object was compiled with. At run
time the recorded values serve as the defaults, so a process started
with no C<IF_*> variables set uses the prebuilt object as-is.
Setting C<IF_*> variables at run time keeps working exactly as in
releases without prebuilt support: if the requested flags differ from
the recorded ones, the prebuilt object (compiled with the wrong flags
for the request) is skipped and the module compiles at first load into
C<_Inline/> -- which does need C<Inline::C> and a compiler on that
machine. Two related knobs exist:
=over 4
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
container base image): blanket C<-march=native> pulls in whatever
instruction sets the build host happens to have, including AVX-512 on
some Intel CPUs -- which is known to trigger clock throttling under
sustained heavy use and can make throughput I<worse> than a
conservative target like C<x86-64-v3> (AVX2, no AVX-512). If in doubt,
benchmark both before committing to one.
=item * C<IF_NO_OPENMP=1> -- build (or select) the serial C backend: the
OpenMP compile attempt is skipped entirely, so the resulting object has
no libgomp linkage and never starts an OpenMP runtime inside the
process. This differs from C<OMP_NUM_THREADS=1>, which merely runs the
parallel code on one thread but still loads libgomp. Set at
C<perl Makefile.PL> time it yields a serial prebuilt object; set at run
time against an OpenMP prebuilt install it triggers a runtime serial
build (needing a compiler). An explicit C<IF_NO_OPENMP=0> re-enables
OpenMP over a serial configure-time default.
=back
Whichever of these are used, the cached artefact under C<_Inline/> is
pinned to that build's instruction set -- delete C<_Inline/> (or use a
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
- n_trees :: number of isolation trees in the ensemble
default :: 100
- sample_size :: sub-sample size used to build each tree... max samples
default :: 256
- max_depth :: per-tree height limit... if not defined is set to ceil(log2(psi))
default :: undef
- seed :: optional integer to seed srand with for reproducible trees...
see perldoc -f srand for more info. This number is processed via abs(int()).
default :: undef
- mode :: if it should be IF or EIF
axis :: classic axis-parallel splits (IF)
extended :: oblique hyperplane splits (EIF)
default :: axis
- extension_level :: extended mode only... how many features take partin each
split. 0 behaves like a single-feature (axis) cut; the
maximum (n_features - 1) uses every varying feature. undef
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
predict methods is therefore the PER-TREE cutoff
here, not a forest-level score cutoff.
score_samples() returns the fraction of trees
voting anomalous -- still in [0, 1], but discrete
in steps of 1/n_trees. contamination composes: fit()
learns the per-tree cutoff that flags the requested
fraction of the training set.
default :: mean
- parallel_fit :: positive integer N => build the trees across N forked
worker processes during fit(). Each worker gets a derived seed
(parent seed + worker_id * 1009) so the parallel fit is
reproducible across runs at fixed worker count -- but the trees
produced are NOT bit-identical to a serial fit with the same
seed, because the RNG draws happen in a different order.
Inference is unaffected. Falls back silently to serial on
platforms without a real fork() (e.g. Windows without Cygwin).
default :: undef (serial)
- use_c :: boolean, override whether the Inline::C backend is used for
both scoring and fit()'s tree builder. When false the instance
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
(one tree per thread) instead of the single-threaded C builder.
Opt-in and off by default: unlike use_c/use_openmp, this changes
which trees get built. Perl's RNG isn't safe to call from
multiple OS threads sharing one interpreter, so this path seeds
an independent PRNG per tree from the tree index rather than
Drand01() -- trees differ from the use_c (single-threaded)
and pure-Perl paths even with the same seed, though a fixed
seed and n_trees still reproduce the same trees regardless of
OMP_NUM_THREADS or scheduling. Does NOT compose with
parallel_fit: a forked child starting its own OpenMP region
after the parent process has used OpenMP for anything can
hang (a general fork()+libgomp limitation), so parallel_fit's
workers always use the single-threaded C builder regardless
of this setting -- setting both just means parallel_fit wins.
Ignored (clamped to 0) when use_c is false or OpenMP isn't
linked in.
default :: 0
- feature_names :: optional arrayref of per-feature labels enabling the
*_tagged methods (and required by mungers below).
default :: undef
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
host_entropy => { munger => 'entropy', from => 'host' },
},
);
$forest->fit_tagged(\@raw_rows);
my $score = $forest->score_sample_tagged(
{ method => 'POST', bytes => 51234, host => 'kq3xv9z2.example' } );
The spec is pure data and is B<saved with the model>, so a loaded model
munges scoring input exactly as it did training input -- the
consistency that makes munging part of the model rather than an
upstream preprocessing step. Points worth knowing:
=over 4
=item * Only tagged input is munged. Positional rows passed to C<fit>
or the scoring methods are taken as already numeric; L</munge_rows>
applies the scalar mungers to positional rows for callers (like the
CLI) that want the same transformation there. Packed datasets
(L</pack_data(\@data)>) are never munged.
=item * Under a munger plan, tagged-row validation is the plan's: a
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
croak "fork failed: $!" unless defined $pid;
if ( $pid == 0 ) {
# child
close $rh;
binmode $wh;
if ( defined $self->{seed} ) {
srand( $self->{seed} + $w * 1009 );
}
# Deliberately never _build_forest_openmp here, even when
# use_openmp_fit is on: if this process (or the parent that
# fork()ed us) already ran any OpenMP region before this
# fork -- including plain score_samples()/predict() with
# the default use_openmp -- libgomp's thread pool exists
# but its worker threads didn't survive the fork. A child
# starting its own #pragma omp parallel region then tries
# to reuse that now-invalid pool and hangs. This is a
# general fork()+libgomp limitation, not fixable from here,
# so forked workers always use the single-threaded C
# builder (or pure Perl) instead. See t/03-fit-determinism.t
# and the NATIVE ACCELERATION docs for the observed hang and
lib/Algorithm/Classifier/IsolationForest.pm view on Meta::CPAN
# random draws come from a thread-private PRNG seeded per tree index
# rather than Drand01() -- Perl's RNG state can't be shared safely
# across OpenMP threads -- so the resulting trees are NOT bit-identical
# to the use_c (serial) or pure-Perl paths for the same seed, though a
# fixed seed + n_trees still reproduce the same trees regardless of
# OMP_NUM_THREADS. This is why it's gated by the separate, opt-in
# use_openmp_fit knob rather than reusing use_c/use_openmp.
#
# Only called from fit()'s non-forked branch. _fit_trees_parallel's
# workers never call this, even when use_openmp_fit is on: a forked
# child starting its own OpenMP region after the parent process has
# used OpenMP for anything (this includes plain score_samples()) can
# hang -- see the comment above that branch for the fork()+libgomp
# hazard this avoids.
#
# build_forest_openmp_xs hands back three arrayrefs of per-tree packed
# buffers (the same SoA layout _pack_tree produces) instead of Perl tree
# structures -- that's how it avoids any Perl API call inside its
# parallel region. _unpack_forest converts them back into the ordinary
# nested-arrayref tree shape so to_json/from_json/_rebuild_c_trees don't
# need to know this path exists.
lib/Algorithm/Classifier/IsolationForest/App/Command/accel.pm view on Meta::CPAN
$ext->fit( \@data );
$ext->score_samples( [ [ 0.5, 0.5, 0.5 ] ] );
my $has_c = $Algorithm::Classifier::IsolationForest::HAS_C ? 1 : 0;
my $has_openmp = $Algorithm::Classifier::IsolationForest::HAS_OPENMP ? 1 : 0;
my $has_simd = $Algorithm::Classifier::IsolationForest::HAS_SIMD ? 1 : 0;
my $c_source = $Algorithm::Classifier::IsolationForest::C_SOURCE;
# Whether a prebuilt object was installed with the dist at all --
# independent of whether this process ended up using it.
my $prebuilt_installed = 0;
{
local $@;
my $bf = eval {
require Algorithm::Classifier::IsolationForest::BuildFlags;
Algorithm::Classifier::IsolationForest::BuildFlags::flags();
};
$prebuilt_installed = 1 if ref $bf eq 'HASH' && $bf->{prebuilt};
}
lib/Algorithm/Classifier/IsolationForest/App/Command/csv2plot.pm view on Meta::CPAN
use strict;
use warnings;
use Algorithm::Classifier::IsolationForest ();
use Algorithm::Classifier::IsolationForest::App -command;
use File::Slurp qw(read_file write_file);
use File::Spec ();
use File::Temp qw(tempfile);
sub opt_spec {
return (
[ 'i=s', 'Input CSV for processing.', { 'completion' => 'files' } ],
[ 'o=s', 'PNG file to output to. Default: plot.png', { 'default' => 'plot.png', 'completion' => 'files' } ],
[ 'w', 'If the file specified via -o exists, over write it.' ],
[
'p=s',
'Type of plot. Default: auto',
{ 'default' => 'auto', completion => [ 'auto', '2heat', '3range', '3binary' ] }
],
[ 'print', 'Print what would be used with gnuplot instead of calling gnuplot' ],
[ 'open', 'Call xdg-open to open the generated graph.' ],
[ 'small-points', 'Use smaller points (ps 0.8) for dense 3range datasets.' ],
lib/Algorithm/Classifier/IsolationForest/App/Command/csv2plot.pm view on Meta::CPAN
3range and 3binary require data outputted from predict with the -d flag.
For N-dimensional data, columns 1 and 2 are always used for the x/y axes.
';
} ## end sub description
sub validate {
my ( $self, $opt, $args ) = @_;
if ( !defined( $opt->{'i'} ) ) {
$self->usage_error('-i has not been specified for a file to process');
} elsif ( !-f $opt->{'i'} ) {
$self->usage_error( '-i, "' . $opt->{'i'} . '", is not a file or does not exist' );
} elsif ( !-r $opt->{'i'} ) {
$self->usage_error( '-i, "' . $opt->{'i'} . '", is not readable' );
}
if ( -e $opt->{'o'} && !$opt->{'w'} ) {
$self->usage_error( '-o, "' . $opt->{'o'} . '", already exists and -w is not given' );
} elsif ( -e $opt->{'o'} && !-f $opt->{'o'} ) {
$self->usage_error( '-o, "' . $opt->{'o'} . '", already exists and -w given but file is not a file' );
lib/Algorithm/Classifier/IsolationForest/App/Command/predict.pm view on Meta::CPAN
use File::Slurp qw(read_file write_file);
use Scalar::Util qw(looks_like_number);
sub opt_spec {
return (
[
'm=s',
'Input model JSON file path/name.',
{ 'default' => 'iforest_model.json', 'completion' => 'files' }
],
[ 'i=s', 'Input CSV for processing.', { 'completion' => 'files' } ],
[ 'o=s', 'Output to this file instead of printing.', { 'completion' => 'files' } ],
[ 'w', 'If the file specified via -o exists, over write it.', { 'completion' => 'files' } ],
[ 't=f', 'Alternative threshold value to use. 0 < $val < 1' ],
[ 'd', 'Include the input data in the output.' ],
);
} ## end sub opt_spec
sub abstract { 'Processes the data using the score_predict_samples using the specified model' }
sub description {
lib/Algorithm/Classifier/IsolationForest/App/Command/predict.pm view on Meta::CPAN
stored doubles.
$feat1,...,$featN,$score,$predict
';
} ## end sub description
sub validate {
my ( $self, $opt, $args ) = @_;
if ( !defined( $opt->{'i'} ) ) {
$self->usage_error('-i has not been specified for a file to process');
} elsif ( !-f $opt->{'i'} ) {
$self->usage_error( '-i, "' . $opt->{'i'} . '", is not a file or does not exist' );
} elsif ( !-r $opt->{'i'} ) {
$self->usage_error( '-i, "' . $opt->{'i'} . '", is not readable' );
}
if ( !-f $opt->{'m'} ) {
$self->usage_error( '-m, "' . $opt->{'m'} . '", is not a file or does not exist' );
} elsif ( !-r $opt->{'m'} ) {
$self->usage_error( '-m, "' . $opt->{'m'} . '", is not readable' );
lib/Algorithm/Classifier/IsolationForest/App/Command/stream.pm view on Meta::CPAN
-s -> seed
-c -> contamination
-t -> feature_names
';
} ## end sub description
sub validate {
my ( $self, $opt, $args ) = @_;
if ( !defined( $opt->{'i'} ) ) {
$self->usage_error('-i has not been specified for a file to process');
} elsif ( !-f $opt->{'i'} ) {
$self->usage_error( '-i, "' . $opt->{'i'} . '", is not a file or does not exist' );
} elsif ( !-r $opt->{'i'} ) {
$self->usage_error( '-i, "' . $opt->{'i'} . '", is not readable' );
}
if ( -e $opt->{'m'} && !-r $opt->{'m'} ) {
$self->usage_error( '-m, "' . $opt->{'m'} . '", exists but is not readable' );
}
lib/Algorithm/Classifier/IsolationForest/App/Command/streamd.pm view on Meta::CPAN
use File::Slurp qw(read_file write_file);
use File::Path qw(make_path);
use File::Basename qw(dirname);
use File::Spec ();
use Scalar::Util qw(looks_like_number);
use IO::Socket::UNIX ();
use IO::Select ();
use POSIX qw(setsid strftime);
use Errno qw();
# The daemon is a singleton per process, so its runtime state lives in
# file-scoped lexicals rather than being threaded through every helper.
my $JSON; # JSON::MaybeXS codec (required at runtime, see execute)
my $OIF; # the online model
my %OPT; # resolved options
my $LOG_FH; # log handle (STDERR in foreground without --log)
my %CONN; # fileno => { sock, inbuf, outbuf, mode }
my $DIRTY = 0; # learned anything since the last save?
my $RUN; # cleared by SIGTERM/SIGINT
my $SAVE_NOW; # set by SIGUSR1
my $REOPEN_LOG; # set by SIGHUP
lib/Algorithm/Classifier/IsolationForest/App/Command/streamd.pm view on Meta::CPAN
<- {"score": 0.74, "label": 1, "tag": "r-2"}
The same rows work from the shell via
`iforest streamc --set web --jsonl -i rows.jsonl`.
Modes are prequential (score each row against the model as it stood, then
learn it -- the default), learn (learn only), and score (score only);
"mode" on a row/rows message overrides the connection default set by
the mode command for that message. A bad row gets an {"error": ...}
reply on that message only; the connection and the daemon live on (for
a "rows" batch, rows before the failing one were already processed).
Multiple concurrent connections are supported; rows are applied to the
one shared model in the order their lines arrive, which defines the
stream order.
--set NAME runs a named instance: the set name is appended to
--model-dir (so its saves, latest.json, and default log live under
their own subdirectory) and the socket/pid become <set>.sock /
<set>.pid under the run dir -- with --set, --socket and --pid name the
base run dir instead of the files. Several sets run side by side, each
lib/Algorithm/Classifier/IsolationForest/Online.pm view on Meta::CPAN
and whose distribution may drift. There is no C<fit()>: the model
C<learn>s points as they arrive and, once more than C<window_size> points
have been seen, forgets the oldest point for every new one so the model
always reflects the most recent C<window_size> points of the stream.
Trees never store data points. Each node keeps only a running count of
the points that passed through it and the bounding box of their feature
values. A leaf splits once enough points have accumulated (see
C<max_leaf_samples> and C<growth>); because the actual points are gone,
the split simulates them by sampling uniformly inside the leaf's bounding
box. Forgetting reverses the process: counts are decremented along the
forgotten point's path and a subtree whose count falls below its split
requirement is collapsed back into a leaf.
Scoring follows the classic Isolation Forest intuition -- anomalies
isolate at shallow depth -- but normalises by the depth budget
C<log(n/max_leaf_samples) / log(4)> of the current window rather than the
batch model's C<c(psi)>. Scores are in (0, 1] with high values
anomalous, directly comparable in spirit (though not numerically) to the
parent class's scores.
t/02-accel-selection.t view on Meta::CPAN
seed => 1,
use_c => 0,
use_openmp => 1,
);
is( $f->{_use_c}, 0, '_use_c is 0' );
is( $f->{_use_openmp}, 0, '_use_openmp clamped to 0 since C backend is off' );
}; ## end 'use_openmp clamped to 0 when use_c is off' => sub
subtest 'use_c => 1 clamped against $HAS_C in the constructor' => sub {
# Fake "no Inline::C compiled" by localising the package flags. The
# XS subs themselves remain defined in this process (they were loaded
# at module init), but the constructor's contract is to clamp against
# $HAS_C so a fresh build without Inline::C wouldn't end up calling
# an undefined sub at score time.
local $Algorithm::Classifier::IsolationForest::HAS_C = 0;
local $Algorithm::Classifier::IsolationForest::HAS_OPENMP = 0;
my $f = $CLASS->new(
n_trees => 10,
sample_size => 16,
seed => 1,
t/02-accel-selection.t view on Meta::CPAN
seed => 9,
use_c => 0,
);
$f->fit( \@data );
eval { $f->pack_data( \@data ) };
like( $@, qr/requires the Inline::C backend/, 'pack_data refuses to run without the C backend' );
}; ## end 'pack_data croaks when use_c is off' => sub
# ------------------------------------------------------------------------
# CLI: `iforest accel` should run cleanly and report a status consistent
# with the package flags this process sees. We bypass it entirely if
# bin/iforest isn't there (e.g. unusual install layouts).
# ------------------------------------------------------------------------
SKIP: {
my $bin = File::Spec->rel2abs('bin/iforest');
skip "bin/iforest not found", 1 unless -x $bin;
subtest 'iforest accel CLI reports a consistent status' => sub {
my $out = `$^X -Ilib $bin accel 2>&1`;
is( $?, 0, 'iforest accel exits 0' );
like( $out, qr/Algorithm::Classifier::IsolationForest acceleration status/, 'prints the status header' );
like( $out, qr/Inline::C\s*:/, 'mentions Inline::C' );
like( $out, qr/OpenMP\s*:/, 'mentions OpenMP' );
like( $out, qr/SIMD\s*:/, 'mentions SIMD' );
like( $out, qr/C object\s*:/, 'mentions the C object source' );
like( $out, qr/Active backend:/, 'prints an Active backend line' );
# The C object line must always resolve to one of the three
# states, consistent with the availability row in the same
# output (the CLI runs in its own process, so we compare the
# CLI's output against itself, not against this process's
# flags -- prebuilt vs runtime may legitimately differ).
if ( $out =~ /Inline::C\s*:\s*available/ ) {
like(
$out,
qr/C object\s*:\s*(prebuilt at install time|compiled at run time)/,
'C object line says prebuilt or runtime when C is active'
);
like(
$out,
qr/Active backend:.*-- (prebuilt at install time|compiled at run time)/,
'Active backend summary includes the C object source'
);
} else {
like( $out, qr/C object\s*:\s*none/, 'C object line says none without a C backend' );
}
# Cross-check the per-feature status lines against the package
# flags the test process observed. This is what makes this test
# actually verify selection rather than just "doesn't crash".
if ($HAS_C) {
like( $out, qr/Inline::C\s*:\s*available/, 'CLI reports Inline::C available, matching $HAS_C' );
} else {
like( $out, qr/Inline::C\s*:\s*not available/, 'CLI reports Inline::C not available, matching $HAS_C' );
}
if ($HAS_OPENMP) {
like( $out, qr/OpenMP\s*:\s*available/, 'CLI reports OpenMP available, matching $HAS_OPENMP' );
} else {
like( $out, qr/OpenMP\s*:\s*not available/, 'CLI reports OpenMP not available, matching $HAS_OPENMP' );
t/03-fit-determinism.t view on Meta::CPAN
#
# Covers:
# 1. Each backend, run twice with the same seed, builds bit-identical
# trees (not just bit-identical scores -- the actual tree structure).
# 2. Pure-Perl and serial-C build BIT-IDENTICAL trees for the same seed,
# in both axis and extended mode, and across every `missing`
# strategy -- because the serial C builder draws randomness through
# Drand01(), the same generator Perl's own rand()/srand() use.
# 3. use_openmp_fit is reproducible for a fixed seed + n_trees
# regardless of OMP_NUM_THREADS (checked across separate
# subprocesses, since that's the only way to force a different
# thread count) -- it deliberately does NOT match the Drand01-based
# backends (documented behaviour: it uses its own thread-safe PRNG),
# so that's checked as a "differs, but each is internally consistent"
# property rather than cross-backend equality.
# 4. use_openmp_fit + parallel_fit together is still safe and
# reproducible -- but does NOT actually run forked workers in
# OpenMP. A forked child starting its own OpenMP region after
# the parent process has used OpenMP for anything (including
# plain score_samples()) can hang -- a general fork()+libgomp
# limitation -- so parallel_fit's workers always use the
# single-threaded C builder regardless of use_openmp_fit. This
# was caught by an earlier version of this test hanging.
use strict;
use warnings;
use Test::More;
use File::Temp qw(tempfile);
use Config;
t/03-fit-determinism.t view on Meta::CPAN
);
is(
fit_trees( %opts, use_c => 1 ),
fit_trees( %opts, use_c => 0 ),
"[missing=$missing] use_c => 1 and use_c => 0 build the same trees"
);
} ## end for my $missing (qw(zero impute nan))
}; ## end 'pure-Perl and serial-C build identical trees for the same seed' => sub
# ------------------------------------------------------------------------
# 3. use_openmp_fit: reproducible in-process for a fixed seed.
# ------------------------------------------------------------------------
subtest 'use_openmp_fit is reproducible in-process for a fixed seed' => sub {
plan skip_all => 'OpenMP not linked in' unless $HAS_C && $HAS_OPENMP;
for my $mode (qw(axis extended)) {
my %opts = (
n_trees => 40,
sample_size => 48,
seed => 41,
mode => $mode,
use_c => 1,
use_openmp_fit => 1,
);
is( fit_trees(%opts), fit_trees(%opts), "[$mode] two use_openmp_fit fits with the same seed match" );
} ## end for my $mode (qw(axis extended))
}; ## end 'use_openmp_fit is reproducible in-process for a fixed seed' => sub
subtest 'use_openmp_fit trees are valid, but not required to match Drand01 backends' => sub {
plan skip_all => 'OpenMP not linked in' unless $HAS_C && $HAS_OPENMP;
# use_openmp_fit uses its own thread-safe PRNG (documented: Perl's
# RNG state can't be shared across OpenMP threads), so it is NOT
# expected to match the serial-C/pure-Perl trees for the same seed.
# What matters is that it still produces a valid, usable model.
my $f = $CLASS->new(
n_trees => 40,
t/03-fit-determinism.t view on Meta::CPAN
is( scalar @{ $f->{trees} }, 40, 'builds exactly n_trees trees' );
my $s = $f->score_samples( \@data );
my $bad = grep { !defined $_ || $_ <= 0 || $_ > 1 } @$s;
is( $bad, 0, 'every score is in (0, 1]' );
}; ## end 'use_openmp_fit trees are valid, but not required to match Drand01 backends' => sub
# ------------------------------------------------------------------------
# 4. use_openmp_fit: reproducible regardless of OMP_NUM_THREADS.
#
# Thread count can't be changed mid-process reliably (OpenMP's runtime
# is normally set up once), so this spawns two subprocesses that build
# an identical dataset and fit with an identical seed, differing only in
# OMP_NUM_THREADS, and compares their tree JSON byte-for-byte.
# ------------------------------------------------------------------------
subtest 'use_openmp_fit reproducibility does not depend on OMP_NUM_THREADS' => sub {
plan skip_all => 'OpenMP not linked in' unless $HAS_C && $HAS_OPENMP;
my ( $fh, $path ) = tempfile( SUFFIX => '.pl', UNLINK => 1 );
print $fh <<'END_CHILD';
use strict;
use warnings;
t/03-fit-determinism.t view on Meta::CPAN
or diag("child output:\n$out4");
is( $out1, $out4, 'same seed + n_trees builds identical trees under OMP_NUM_THREADS=1 vs 4' );
}; ## end 'use_openmp_fit reproducibility does not depend on OMP_NUM_THREADS' => sub
# ------------------------------------------------------------------------
# 5. use_openmp_fit + parallel_fit together: must not hang, and must
# still be reproducible -- but see the file-top comment: parallel_fit's
# forked workers ignore use_openmp_fit and always use the
# single-threaded C builder, specifically BECAUSE a forked child
# starting its own OpenMP region after this same process has already
# run OpenMP (subtests 3/4 above did, via use_openmp_fit) can hang.
# This subtest runs deliberately late in the file, after use_openmp_fit
# has already executed in-process above, to exercise exactly that
# ordering.
# ------------------------------------------------------------------------
subtest 'use_openmp_fit + parallel_fit does not hang and is reproducible' => sub {
plan skip_all => 'no fork() on this platform'
unless $can_fork && $HAS_C && $HAS_OPENMP;
my %opts = (
n_trees => 60,
sample_size => 48,
seed => 51,
t/04-accel-tuning.t view on Meta::CPAN
#
# Verifies the C-build tuning environment variables (read once, at
# module load, in the block that compiles the Inline::C backend):
#
# * IF_NO_C=1 -- skips building the C backend entirely
# * IF_OPT=<level> -- overrides the default -O3
# * IF_ARCH=<value> -- adds -march=<value>
# * IF_NATIVE=1 -- shorthand for IF_ARCH=native, superseded by IF_ARCH
#
# Since these are read once at load time, each combination needs its own
# fresh perl process -- this spawns one per case and inspects
# $Algorithm::Classifier::IsolationForest::{HAS_C,OPT_LEVEL} from its
# output, the same technique t/03-fit-determinism.t uses for
# OMP_NUM_THREADS. Also checks that IF_OPT/IF_ARCH validate their input
# rather than passing it through to a compiler command line unchecked.
use strict;
use warnings;
use Test::More;
use File::Temp qw(tempfile);
t/05-prebuilt-env.t view on Meta::CPAN
# Exercises the load-time environment knobs that arrived with install-time
# prebuilt object support:
#
# * IF_NO_C=1 -- no C backend at all
# * IF_NO_OPENMP=1 -- backend (C or not) must report no OpenMP
# * IF_RUNTIME_BUILD=1 -- any C backend in use must be a runtime build,
# never the prebuilt object
# * scoring parity -- every variant scores the same data identically
#
# The IF_* environment is read once at first module load, so each scenario
# needs a fresh perl; we probe via subprocesses. Nothing here asserts that
# a C backend is *available*: on a pure-Perl host every probe degrades to
# HAS_C=0 and the parity checks still apply.
use strict;
use warnings;
use Test::More;
# Under the harness blib/ exists and carries any prebuilt object; from a
# bare checkout fall back to lib/.
my @inc = -d 'blib/lib' ? ('-Mblib') : ('-Ilib');
t/05-prebuilt-env.t view on Meta::CPAN
open my $fh, '-|', $^X, @inc, '-e', $probe_code
or die "can't spawn $^X: $!";
my $line = <$fh>;
close $fh;
chomp( $line //= '' );
return split /\|/, $line, -1;
} ## end sub load_probe
my @baseline = load_probe();
is( scalar @baseline, 5, 'baseline probe produced all fields' )
or BAIL_OUT("subprocess probe failed; output: @baseline");
my ( $has_c, $has_openmp, $c_source ) = @baseline;
ok( $has_c == 0 || $has_c == 1, "baseline HAS_C is 0/1 (got '$has_c')" );
if ($has_c) {
ok( $c_source eq 'prebuilt' || $c_source eq 'runtime',
"C_SOURCE is prebuilt/runtime when HAS_C (got '$c_source')" );
} else {
is( $c_source, '', 'C_SOURCE empty without C backend' );
}
diag("baseline: HAS_C=$has_c HAS_OPENMP=$has_openmp C_SOURCE=$c_source");
t/80-sklearn-comparison-undef.t view on Meta::CPAN
my $python_bin;
for my $cmd (qw(python3 python)) {
my $probe = `$cmd -c "import sklearn; print('ok')" 2>/dev/null`;
if ( defined $probe && $probe =~ /\bok\b/ ) {
$python_bin = $cmd;
last;
}
}
# -----------------------------------------------------------------------
# Python helper: scores all datasets in one subprocess. Each argv spec
# is "csv_path|label|n_train"; the CSV concatenates training rows then
# test rows, and n_train is the split point. Test rows may use the
# token "nan" / "undef" / empty for missing values.
#
# sklearn score_samples convention: lower = more anomalous (opposite of
# Perl), so we negate sklearn scores before computing rank correlation.
# -----------------------------------------------------------------------
my $py_script = <<'END_PY';
import sys, json
import numpy as np
t/80-sklearn-comparison.t view on Meta::CPAN
unless ( defined $python_bin ) {
plan skip_all => 'Python with scikit-learn is not installed; skipping cross-language comparison';
}
# -----------------------------------------------------------------------
# Python helper: train sklearn IsolationForest per dataset and emit JSON.
#
# Receives "csv_path|label" pairs on argv and returns a JSON object keyed
# by label with the corresponding score_samples() output. Batching all
# datasets into a single subprocess avoids paying Python + sklearn import
# cost more than once.
#
# sklearn score_samples convention: lower score = more anomalous. That
# is the opposite direction from this module (higher = more anomalous),
# so we negate sklearn scores before computing rank correlation.
# -----------------------------------------------------------------------
my $py_script = <<'END_PY';
import sys, json
import numpy as np
from sklearn.ensemble import IsolationForest
t/90-cli-commands.t view on Meta::CPAN
#!perl
# 90-cli-commands.t
#
# Smoke-tests the iforest CLI subcommands by running bin/iforest in a
# subprocess against a temp model + CSV. We chain them through a
# realistic workflow:
#
# fit -> info
# -> bench
# -> pack -> predict (packed input)
# -> predict (raw CSV input, for regression baseline)
#
# Each subtest checks the relevant artefacts and output snippets but
# stays loose on the exact wording -- we want to catch breakage, not
# pin the formatting.
t/92-streamc.t view on Meta::CPAN
#!perl
# 92-streamc.t
#
# Integration test for `iforest streamc`: spawns a streamd daemon (as
# t/91 does) and drives it through the real streamc CLI in a
# subprocess. Covers command mode (--ping/--stats/--save, exit codes,
# --json), CSV stream mode in all three --mode settings with -d,
# --jsonl tagged rows, client- and daemon-side error attribution by
# input line, --set socket resolution, and connect-failure behaviour.
#
# Skipped on Windows (Unix sockets + fork) and without JSON::MaybeXS.
use strict;
use warnings;
use Test::More;
use File::Temp qw(tempdir);
( run in 1.335 second using v1.01-cache-2.11-cpan-600a1bdf6e4 )