App-Test-Generator
view release on metacpan or search on metacpan
workflow's own bookkeeping commits (same logic as mutate.yml), so the
diff always covers everything since mutation.json was last meaningfully
regenerated.
- Fix bin/test-generator-mutate setting $ENV{LCSAJ_TARGETS} to the
mutation-scoped file list before running the baseline system('prove')
check. This caused t/LCSAJ-Runtime.t (which calls DB::DB() directly
against its own file) to fail spuriously whenever --file or
--changed_only selected a subset of lib files, aborting the entire
mutation run before any mutants were tested. LCSAJ_TARGETS is now
assigned after baseline verification, scoped only to the mutation runs.
- Fix t/LCSAJ-Runtime.t's hit-recording subtest assuming %TARGET is
empty. During mutation testing LCSAJ_TARGETS is intentionally scoped
to the file under mutation (never a test file), so %TARGET is
non-empty and the subtest's direct DB::DB() call was filtered out,
making the 'hit recorded' assertion fail for every per-mutant prove
run. This produced a fake 100% kill rate that invalidated all mutation
results. Fixed by using local %TARGET = () within that subtest.
- Fix t/function.t's BEGIN-time LCSAJ_TARGETS subtest spawning its
child perl process via qx{}/backticks with a shell-quoted -e
argument (single quotes). qx{} always goes through a shell, and
cmd.exe on Windows does not treat single quotes as a quoting
character, so the child process failed to parse its own -e
script ("Can't find string terminator"). Switched to list-form
system() under Capture::Tiny::capture_merged(), which passes the
-e argument directly with no shell involved on any platform.
- Fix LCSAJ::_save_lcsaj() building an output directory path by
joining $dir with $file's full path when $file has no 'lib/'
segment to strip (e.g. a File::Temp tempfile). On Windows this
embedded a second drive letter mid-path and made
File::Path::make_path fail with "Invalid argument"; on other
platforms it silently created a deeply nested mirror of $file's
absolute path under $dir. Now falls back to the basename when
the stripped path is still absolute.
- Removed Test::Needs gates that never reflect a genuine dependency:
t/params_validate.t, t/moosex_params_validate.t (Params::Validate
and MooseX::Params::Validate/MooseX::Types::Moose appear only in
heredoc fixture text parsed by PPI/Safe::reval, never truly loaded),
t/params_validate_strict.t (already a hard PREREQ_PM), and the
HTML::Genealogy::Map gate in t/fuzz.t (leftover from deleted
t/conf/html_genealogy_map.{conf,yml} fixtures). The
MooseX::Params::Validate fix restores real extraction coverage
that was previously silently skipped on any machine without Moose
installed.
- Added a Test::Needs gate for Type::Params/Types::Common to the
signature_for subtest in t/integration.t: that fixture is compiled
by a genuinely spawned perl -T subprocess, which
Test::Without::Module cannot fake "missing" for, so the dependency
must be a hard skip rather than left to croak.
- Fix Planner::Fixture::plan() comparing $isolation->{$method}
directly against the 'shared_fixture' string. Planner::Isolation::
plan() actually returns a per-method hashref with a 'fixture' key
(plus optional env/filesystem/time/network keys), not a bare mode
string, so the comparison always stringified a hash reference and
was always false. In practice this meant every method planned by
Planner::build_plan() silently got 'new_per_test' fixture mode,
even purity-'pure' methods that should have gotten a shared
fixture. Each module's own unit tests passed because they fed
Fixture::plan() a fabricated flat-string isolation map that never
matched Isolation::plan()'s real output shape; only composing the
two through build_plan() in a new end-to-end test caught the
mismatch. Fixed to read $isolation->{$method}{fixture}; updated
Fixture.pm's POD and t/Planner-Fixture.t, t/Planner-submodules_unit.t,
and t/function.t's fixtures to the real per-method hashref shape.
- Fix mutation.json incorrectly added to .gitignore while the file is
tracked by git; removed the .gitignore entry and untracked the file
with git rm --cached so it is no longer committed.
- Fix mutate.yml (deployed to dependent repos) still using floating
shogo82148/actions-setup-perl@v1 after dashboard.yml was pinned;
both deployed workflow files now pin to the same version.
- Fix POD-embedded CI workflow template referencing stale action
versions (actions/checkout@v5, actions-setup-perl@v1) after
dashboard.yml was updated; template now matches dashboard.yml.
- Fix README.md SEE ALSO link label still saying "Test Coverage Report"
after the POD was updated to "Test Dashboard".
- Fix dashboard.yml "Publish test dashboard" step running unguarded on
pull_request events, publishing PR-derived coverage output to the
public gh-pages site; now skipped on pull_request like the other
commit steps.
- Fix dashboard.yml "Save updated mutation results" guard
(if: github.event_name == 'workflow_run') discarding the freshly
regenerated mutation.json on plain push runs; generate-test-dashboard
regenerates mutation.json on every run regardless of trigger, so the
guard now matches its siblings (if: github.event_name !=
'pull_request').
- Fix dashboard.yml silently skipping the coverage snapshot when
cover_html/cover.json is missing, masking real dashboard-generation
failures; the step now emits an ::error:: annotation, dumps
cover_db/cover_html/coverage for diagnosis, and fails the job.
- Fix mutate.yml "Find last code commit" silently falling back to
HEAD~1 when no base commit could be parsed from git log, narrowing
the mutation diff without any visible indication; now emits an
::warning:: annotation when BASE is empty.
- Fix dashboard.yml and mutate.yml commit steps calling a shared
.github/scripts/commit-if-changed.sh helper; dependent repos only
receive the two workflow YAML files when copying them in (see
"now portable to any CPAN module" below), so the script was missing
on every consumer and CI failed with "No such file or directory".
Commit/push logic is now inlined directly in each step.
- Fix extract-schemas _load_target_module() loading the target module
via eval "require $package" (string eval); the package name is now
validated against Perl's package-name grammar and the module is
loaded by converting it to a file path and calling require on that,
so it is never compiled as arbitrary Perl source.
- Fix test-generator-mutate --changed_only interpolating --base_sha
into a backtick-executed git diff command; --base_sha is now
validated against a restrictive character class and git diff is
invoked via list-form open() instead of the shell.
- Fix test-generator-index _mutant_file_report() computing a
slash-sanitised $filename that was never used, leaving the report
path built from the raw, unsanitised $file; it now rejects any '..'
path segment in $file and verifies the resolved output path stays
under the report directory before writing.
- Fix extract-schemas _load_target_module() losing the drive letter
when reconstructing the lib-dir candidate with File::Spec->catdir(),
so the @INC-walking loop never found a 'lib' directory on Windows
when the input file and the cwd were on different drives; now uses
catpath() to keep the volume attached.
- Fix SchemaExtractor _infer_type_from_expression() and
_parse_modern_signature() using hand-rolled brace/bracket depth
counters to split comma-separated lists, which mishandled nested
brackets (e.g. a signature default value containing a hashref
literal could be split on the comma inside the hashref); both now
use Text::Balanced::extract_bracketed() to skip over nested
structures correctly.
- Fix Mutator.pm's _dedup_mutants(), _is_redundant_mutation(), and
apply_mutant() reaching into Mutant objects via direct hash access
(e.g. $m->{line}), bypassing the class's own accessor methods; all
six call sites now use the Mutant accessors ($m->line, $m->original,
$m->transform, etc).
- Fix Devel::App::Test::Generator::LCSAJ::Runtime's DB::DB debugger
hook calling abs_path() (a filesystem stat) on every single
statement executed under the debugger; resolved paths are now
memoised per raw $file in %NORM_CACHE.
- Fix lib/App/Test/Generator/Sample/Module.pm declaring
package Test::App::Generator::Sample::Module (reversed word order)
instead of App::Test::Generator::Sample::Module, so the package
name did not match its location under
lib/App/Test/Generator/Sample/; renamed throughout the file.
- Fix Generator.pm SYNOPSIS extract-schemas example passing two
_extract_defaults_from_code so that inner closure parameters
(my ($x, $y) = @_ inside a sub ref) are not mistaken for the outer
method's parameters.
[Bug fixes]
- Fix Mutation::NumericBoundary's conditional-context detection
(used for Mutator's fast-mode dedup) always evaluating false: PPI
wraps a condition's content in PPI::Statement::Expression, so an
operator's immediate parent is never literally
PPI::Structure::Condition. Operators inside if/unless/while/until
conditions were always tagged 'expression'. Now uses the same
ancestor-walking _in_conditional() helper already shared by
BooleanNegation and ReturnUndef.
- Fix Analyzer::SideEffect.pm's mutates_self and mutates_globals
detection ($self->{field} = ... assignment and %ENV/%SIG/@ARGV
mutation) matching against the raw method body instead of the
comment/string-stripped $code_only, so a field-assignment-like
fragment inside a string literal or comment could be mistaken for
an actual mutation. Both checks now match against $code_only, same
as the keyword/operator counts already fixed in a prior release.
- Fix Emitter::Perl.pm's _emit_method_tests() having no dispatch
branch for the boundary_tests plan flag: TestStrategy.pm sets
$plan{boundary_tests} when a method's schema carries non-empty
_yamltest_hints, but the corresponding $TEST_BOUNDARY constant was
defined and never read, so methods planned for boundary testing
silently got zero generated test code for it. Added the dispatch
line plus a new _emit_boundary_test() that emits one smoke-test
block per boundary_values/invalid_inputs hint value.
- Fix TestStrategy.pm's generate_plan() extracting side_effects and
dependencies from each method's schema _analysis but never using
them, while the module's own DESCRIPTION claimed the plan was
based on them; verified empirically that schema side-effect data
has zero effect on the generated plan. Removed the dead
extraction and corrected DESCRIPTION to attribute side-effect- and
dependency-driven planning to Planner::Mock and Planner::Isolation,
where it actually happens.
- Fix SchemaExtractor.pm's include_private POD describing the
always-included _new/_init/_build method-name override as an
exact-name match; _find_methods actually does a prefix match
(/^_(new|init|build)/), so e.g. _build_attribute and _init_logger
are also force-included, matching common Moose builder/
initializer naming conventions. Confirmed via git log -L that the
prefix behaviour is original and deliberate; POD corrected to
match.
- Fix Generator.pm's render_hash() POD describing $href's values as
always hashrefs; a scalar value that is a recognised type string
is silently expanded to { type => $value } before being skipped-
with-a-warning, which the POD did not mention.
- Fix Generator.pm's _perl_quote() POD omitting that the strings
'true'/'false' are special-cased to the Perl boolean constants
!!1/!!0 rather than being single-quoted like other strings.
- Fix Analyzer::Complexity.pm and Analyzer::Return.pm POD describing
their $method argument as an App::Test::Generator::Model::Method
object; SchemaExtractor's actual callers pass a plain hashref with
a body/source key, not a Model::Method instance. POD corrected to
describe the real calling convention.
- Fix README.md's extract-schemas usage example passing two
positional arguments and referencing a nonexistent
lib/Sample/Module.pm with a .yaml extension; corrected to the real
invocation (extract-schemas lib/App/Test/Generator/Sample/Module.pm
&& fuzz-harness-generator -r schemas/greet.yml) and updated its
GitHub Actions example to the same pinned action SHAs already used
in dashboard.yml/mutate.yml.
[Enhancements]
- Add comprehensive black-box subtests to t/unit.t validating the
public, POD-documented API of every .pm file under lib/ (all 27
modules), written strictly against each module's documented
Arguments/Returns contract rather than incidental implementation
behaviour. Where a module's existing dedicated test file
(e.g. t/Planner-Isolation.t, t/TestStrategy.t,
t/Template_unit.t) already exercised its public API exhaustively
in equivalent black-box style, no duplicate coverage was added to
t/unit.t.
- Add missing public-API POD (Purpose, Arguments, Returns, API
specification) to every previously undocumented public method in
Model::Method.pm, which had none beyond a VERSION section; to
Template.pm's get_data_section(); and to Planner.pm's build_plan(),
which previously carried only an internal TODO-style comment
despite being a public method.
- Add Notes sections to Mutation::BooleanNegation, ::ConditionalInversion,
::NumericBoundary, and ::ReturnUndef documenting the context and
line_content fields each mutant carries for Mutator's fast-mode
dedup, and to CoverageGuidedFuzzer.pm documenting that a
target_sub die is only recorded as a bug when the triggering input
is schema-valid.
0.38 Mon May 18 21:19:40 EDT 2026
[Bug fixes]
- Fix missing TER1/TER2/TER3 column in mutation dashboard
- remove duplicate _lcsaj_coverage_for_file calls
- move skipped-annotation note inside summary div
- Fix MUTANT_SKIP_BEGIN/END regex to match only lines where the annotation is the entire content,
preventing false positives in comments and POD
0.37 Mon May 18 14:58:48 EDT 2026
[Enhancements]
- Add MUTANT_SKIP_BEGIN / MUTANT_SKIP_END source annotations to exclude
lines from mutation testing; mismatched markers are fatal. Skipped
line counts appear in the per-file mutation report and summary table.
[Bug fixes]
- Fix fuzz schema generation looking in xt/conf instead of t/conf.
- Fix mutate.yml for external repos: add commit step so mutation.json
is persisted after each run, enabling the dashboard workflow_run
trigger to pick up fresh results.
- Fix BSD::Resource dependency failing on Windows; guard with OS check.
0.36 Sat May 9 18:54:04 EDT 2026
[Bug fixes]
- Fix uninitialized $version warning in CPAN Testers "no failures"
message when the CPAN API returns a 404 (no release data available).
- Fix "Can't open lib/App/Test/Generator.pm" error when test-generator-index
is run from a repository other than ATG itself; ATG version is now
found by searching @INC with fallback to the configured module_file path.
0.35 Sat May 9 09:40:08 EDT 2026
- Added detect_scattered_failures() root cause detector. Surfaces a
weak-confidence advisory when failures and passes coexist across
2 or more common Perl series with no detectable version cliff or
OS pattern, suggesting flaky tests, optional dependency
differences, or CGI environment assumptions rather than a
compatibility issue. Complements detect_universal_failure() which
handles the opposite case of near-total failure. Confidence is
intentionally set to 0.40 (Weak) since this is a catch-all
signal rather than a precise diagnosis, ensuring it appears below
stronger signals in the root causes table when multiple detectors
fire simultaneously.
- Added detect_universal_failure() root cause detector. Surfaces a
high-confidence warning when failures occur across 3 or more
distinct Perl versions and 2 or more OS types with fewer than
10% passing reports, indicating a likely broken release rather
than a version- or platform-specific compatibility issue. Likely
causes listed in evidence: missing file in tarball, broken
Makefile.PL, or undeclared dependency. Integrated into
detect_root_causes() where it is evaluated first and sorted by
confidence alongside the existing OS, locale, and Perl version
cliff detectors.
- Fixed blib/ paths appearing in coverage table instead of lib/
paths. Devel::Cover instruments blib/ during testing; paths are
now normalised to lib/ for display, with deduplication against
any native lib/ entry.
- Fixed structural coverage percentages in Executive Summary and
Structural Coverage sections showing ~24% instead of ~93%.
_coverage_totals now aggregates from individual own-project files
rather than Devel::Cover's pre-aggregated Total key which
includes all instrumented CPAN dependencies.
- Fixed cyclomatic complexity badge colour and tooltip inverted.
High complexity now correctly shows red/Needs improvement;
low complexity shows green/Good. Second condition also fixed
to use $complexity rather than $score.
0.31 Fri Apr 10 08:07:40 EDT 2026
- Added TER3 (LCSAJ path coverage) column to mutation files table in
the test dashboard index, showing percentage and raw fraction
(e.g. "71.8% (352/491)")
- Added TER1, TER2 and TER3 metrics to per-file mutation report pages,
replacing the plain "Statement" and "Branch" labels with their
formal Test Effectiveness Ratio names
- Renamed the LCSAJ column header in the mutation files table to TER3
- With min set to zero on an integer, sometimes negative or floats could be created - added abs() and int() calls as needed
- Added --generate_mutant_tests=DIR option to generate_index.pl to
produce a timestamped test stub file (t/mutant_YYYYMMDD_HHMMSS.t)
for surviving mutants. High/Medium difficulty survivors get TODO
test stubs; Low difficulty survivors get comment-only hints.
Multiple mutations on the same source line are deduplicated into
one stub listing all variants — one good test kills them all.
Boundary value suggestions are generated for numeric mutations,
clamped and deduplicated for non-negative contexts such as
scalar() and length(). Environment variable hints are added where
the source line references $ENV{...}. The enclosing subroutine
name is shown in each stub for navigation context. File is skipped
if there are no survivors or low-difficulty hints to report.
- LCSAJ path dots on per-file mutation pages are now coloured blue
(covered) or red (not covered), based on whether any line in the
path range was executed during testing. Uncovered paths show
[NOT COVERED] in the hover tooltip. The LCSAJ legend is updated
to explain both colours.
- Replaced TER3-only column in mutation files table with a
TER1 / TER2 / TER3 triple, each component shown as a
colour-coded badge (green/yellow/red). TER1=Statement,
TER2=Branch, TER3=LCSAJ path coverage. Any component
without data shows a grey n/a badge. Column header carries
a tooltip defining all three metrics.
- Added --generate_test=mutant option to generate_index.pl (used
alongside --generate_mutant_tests=DIR). For NUM_BOUNDARY survivors,
attempts to produce a runnable YAML schema file in t/conf/ using
App::Test::Generator::SchemaExtractor rather than a TODO stub.
The schema is augmented with boundary values from the surviving
mutant (the exact boundary value plus one either side) and picked
up automatically by t/fuzz.t on the next test run. Falls back to
a TODO stub if SchemaExtractor fails, the enclosing sub cannot be
found, or confidence in the extracted schema is too low. The
--generate_test option is designed to accept future classes beyond
'mutant'. Updated generate_test_dashboard Step 7 to pass
--generate_test=mutant by default.
- Made output_dir optional in App::Test::Generator::SchemaExtractor
new() -- it is now only required if schema files will be written.
extract_all() gains a no_write option to suppress file output and
return schemas only, for use by callers that want to inspect or
augment schemas before deciding where to write them.
- Added --generate_fuzz flag to generate_index.pl. Scans t/conf/
for existing YAML schema files and writes timestamped augmented
copies (mutant_fuzz_YYYYMMDD_HHMMSS_FUNCTION.yml) with boundary
values from surviving NUM_BOUNDARY mutants merged in. The original
schema is never modified. Augmented schemas are picked up
automatically by t/fuzz.t. Schemas with no matching survivors are
skipped (with a verbose note). Boundary values are merged into
whichever edge key already exists in the schema (edge_case_array
or edge_cases), with deduplication. Schemas already prefixed with
mutant_fuzz_ are skipped to prevent cascading augmentation.
Updated generate_test_dashboard Step 7 to pass --generate_fuzz.
0.30 Thu Apr 2 07:17:16 EDT 2026
Added mutation levels.
Setting to fast will reduce the number of mutants by deduping and removing unnecessary mutants.
See App::Test::Generator::Mutator::_is_redundant_level for a list of those optimised out
Added basic LSCAJ data to the test dashboard.
Added simple security string testing.
Don't push too hard on builtins as they don't have good parameter validation.
Ensure adding only the byte order marker honours $min
More use of _DESCRIPTION
0.29 Thu Feb 26 12:57:59 EST 2026
Added mutation testing
Added guided testing to extract-schemas
Some routines were incorrectly labelled as getter routines
Added more edge cases
Added fallback to extract parameters from classic Perl body styles
Added Type::Param support (https://github.com/nigelhorne/App-Test-Generator/issues/4)
Getter routines take no arguments
Fixed string testing when both min and max are given
Don't give $class or $self as parameters
No input/output no longer croaks, because there are now a few tests that can be run
Add a basic hashref to mandatory args
Use UUID::Tiny and Readonly::Values::Boolean
( run in 0.405 second using v1.01-cache-2.11-cpan-bbe5e583499 )