view release on metacpan or  search on metacpan

doc/getting-started-blog.md  view on Meta::CPAN

# Stop Writing Test Cases By Hand: An Introduction to Specification-Driven Testing in Perl

**TL;DR:** Imagine describing what your function *should do* once, and automatically generating hundreds of test cases that probe edge cases, boundary conditions, and error handling. That's `App::Test::Generator`.

## The Problem

You've just written a function that validates email addresses. You know you should test it thoroughly:

```perl
sub validate_email {
    my ($email) = @_;
    # ... validation logic ...
    return 1 if valid, dies otherwise
}
```

So you start writing tests:

```perl
ok(validate_email('user@example.com'), 'basic email works');
ok(validate_email('user+tag@example.com'), 'plus addressing works');
dies_ok { validate_email('') } 'empty string dies';
dies_ok { validate_email('not-an-email') } 'invalid format dies';
# ... 50 more test cases you need to think of ...
```

**The questions that keep you up at night:**
- Did I test empty strings? What about strings with only whitespace?
- What about null bytes? Unicode? Emoji?
- What if someone passes `undef`? An array reference?
- Did I check the boundary between valid and invalid lengths?
- Am I testing the same cases I tested last month, or did I miss something new?

## The Solution: Write a Specification, Get Tests for Free

Instead of writing individual test cases, describe what your function *should accept*:

```yaml
---
module: Email::Validator
function: validate_email

input:
  email:
    type: string
    min: 3
    max: 254  # RFC 5321 limit
    matches: "^[\\w.+-]+@[\\w.-]+\\.[a-zA-Z]{2,}$"

output:
  type: boolean

config:
  test_undef: yes
  test_empty: yes
  test_nuls: yes

seed: 42
iterations: 100
```

Save this as `t/conf/validate_email.yml`, then run:

```bash
$ fuzz-harness-generator t/conf/validate_email.yml > t/validate_email_fuzz.t
$ prove -v t/validate_email_fuzz.t
```

**What just happened?**

The generator created a test file with:
- ✅ 100 random valid emails (fuzzing)
- ✅ Edge cases: exactly 3 chars, exactly 254 chars, 2 chars (too short), 255 chars (too long)
- ✅ Regex boundary tests: strings that almost match, strings with special chars
- ✅ Invalid inputs: `undef`, empty string, null bytes, emoji, full-width characters
- ✅ Wrong types: arrays, hashes, numbers passed as email
- ✅ Reproducible: Same seed = same random tests every time

That's **200+ test cases** from a 15-line YAML file.

## Real-World Example: Testing a Math Function

Let's test something more complex. Here's a function that normalizes numbers to a 0-1 range:

```perl
package Math::Utils;

sub normalize {
    my ($value, $min, $max) = @_;
    die "min must be less than max" unless $min < $max;
    die "value out of range" if $value < $min || $value > $max;

    return ($value - $min) / ($max - $min);
}
```

The specification captures both valid inputs **and** the transformation rules:

```yaml
---
module: Math::Utils
function: normalize

input:
  value:
    type: number
    position: 0
  min:
    type: number
    position: 1
  max:
    type: number
    position: 2

output:
  type: number
  min: 0
  max: 1

transforms:
  min_value_returns_zero:
    input:
      value: { type: number, value: 0 }
      min: { type: number, value: 0 }
      max: { type: number, value: 100 }
    output:
      type: number
      value: 0

  max_value_returns_one:
    input:
      value: { type: number, value: 100 }
      min: { type: number, value: 0 }
      max: { type: number, value: 100 }
    output:
      type: number
      value: 1

  midpoint_returns_half:
    input:
      value: { type: number, value: 50 }
      min: { type: number, value: 0 }
      max: { type: number, value: 100 }
    output:
      type: number
      value: 0.5

  inverted_range_dies:
    input:
      value: { type: number, value: 50 }
      min: { type: number, value: 100 }
      max: { type: number, value: 0 }
    output:
      _STATUS: DIES

iterations: 50
seed: 42
```

This generates tests that verify:
1. The math is correct (transforms)
2. Boundary conditions work (min=0 returns 0, max=100 returns 1)
3. Invalid inputs are rejected (inverted range dies)
4. Random inputs within range work correctly

## The Five-Minute Quick Start

### 1. Install the module

```bash
cpanm App::Test::Generator
```

### 2. Create a configuration file

`t/conf/my_function.yml`:

```yaml
---
module: My::Module
function: my_function

input:
  name:
    type: string
    min: 1
    max: 100
  age:
    type: integer
    min: 0
    max: 150
    optional: true

output:
  type: string

seed: 12345
iterations: 50
```

### 3. Generate and run tests

```bash
# Generate the test file
fuzz-harness-generator t/conf/my_function.yml > t/my_function_fuzz.t

# Run it
prove -v t/my_function_fuzz.t
```

### 4. Add to your CI/CD

The best part? Add this to GitHub Actions and it runs automatically:

`.github/workflows/fuzz.yml`:

```yaml
name: Fuzz Testing

on:
  push:
    branches: [main, master]
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM

jobs:
  fuzz-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
 
      - name: Setup Perl
        uses: shogo82148/actions-setup-perl@v1
        with:
          perl-version: '5.38'

      - name: Install dependencies
        run: |
          cpanm App::Test::Generator
          cpanm --installdeps .

      - name: Generate and run fuzz tests
        run: |
          mkdir -p t/fuzz
          for config in t/conf/*.yml; do
            test_name=$(basename "$config" .yml)
            fuzz-harness-generator "$config" > "t/fuzz/${test_name}_fuzz.t"
          done
          prove -lr t/fuzz/
```

Now you get **continuous fuzzing** - your code is tested against hundreds of edge cases every day.

## Common Patterns

### Pattern 1: Security-Critical Input Validation

```yaml

doc/getting-started-blog.md  view on Meta::CPAN

```yaml
type_edge_cases:
  string:
    - ''
    - ' '
    - "\t\n"
    - 'x' x 10000
    - "😊🎉"  # emoji
  integer:
    - 0
    - -1
    - 2147483647   # 32-bit max
    - -2147483648  # 32-bit min
```

## Getting Help

- 📚 **Full documentation**: `perldoc App::Test::Generator`
- 🐛 **Issues/Questions**: https://github.com/nigelhorne/App-Test-Generator/issues
- 💬 **Examples**: See `t/conf/` in the repository
- 📊 **Coverage reports**: https://nigelhorne.github.io/App-Test-Generator/coverage/

## Try It Today

Pick one function in your codebase that:
1. Has clear input requirements
2. You're nervous about (parsing, validation, critical path)
3. Could use more test coverage

Write a 10-line YAML file describing it. Generate 200+ tests. Find bugs you didn't know existed.

That's the power of specification-driven testing.

---

*App::Test::Generator is open source and available on CPAN. Contributions welcome!*

---

## Appendix: Complete Example

Here's a complete example testing CGI::Info's `script_path` function (which takes no arguments and returns an absolute path):

`t/conf/script_path.yml`:
```yaml
---
module: CGI::Info
function: script_path

input: undef  # Takes no arguments

output:
  type: string
  min: 1
  matches: "^(?:[A-Za-z]:[/\\\\]|/)"  # Windows or Unix absolute path

config:
  test_undef: yes

seed: 42
iterations: 50
```

Generated test output:
```
ok 1 - use CGI::Info;
ok 2 - script_path survives
ok 3 - output validates
ok 4 - script_path survives
ok 5 - output validates
# ... 50+ tests of calling with no arguments ...
1..52
```

Simple, clear, comprehensive.