Property-Based Testing: Finding Bugs Your Unit Tests Miss

The Limits of Example-Based Testing

Unit tests are example-based. You pick an input, specify the expected output, and verify the function produces it. This approach works well, but it has a blind spot: you can only test cases you think of.

Bugs live in the cases you didn't think of. The empty string. The negative number. The string with a null byte in the middle. The timezone edge case at midnight on a daylight saving transition. Your tests pass because they never encounter these inputs.

Property-based testing inverts this. Instead of specifying inputs, you specify properties that must hold true for all inputs, and let the testing framework generate hundreds of random inputs to try to violate those properties.

The Core Idea: Properties, Not Examples

A property is a rule that should hold for any valid input. Here are properties for a sorting function:

• The output length equals the input length
• Every element in the output was in the input
• Each element in the output is less than or equal to the next element

Instead of testing sort([3, 1, 2]) equals [1, 2, 3], you test that for any array of integers, the three properties above hold.

Here's the difference in practice:

// Example-based test
public function test_sorts_integers(): void
{
    $this->assertEquals([1, 2, 3], sort_ints([3, 1, 2]));
    $this->assertEquals([1], sort_ints([1]));
    $this->assertEquals([], sort_ints([]));
}

// Property-based test (using Eris library)
public function test_sort_preserves_elements(): void
{
    $this->forAll(
        Generator\vector(Generator\int())
    )->then(function (array $input) {
        $output = sort_ints($input);

        // Property 1: Same length
        $this->assertCount(count($input), $output);

        // Property 2: Same elements
        $inputSorted = $input;
        sort($inputSorted);
        $this->assertEquals($inputSorted, $output);

        // Property 3: Each element <= next
        for ($i = 0; $i < count($output) - 1; $i++) {
            $this->assertLessThanOrEqual($output[$i + 1], $output[$i]);
        }
    });
}

The property-based version will generate 100+ random integer arrays and verify all three properties hold for each.

Eris: Property-Based Testing for PHP

Eris is the main property-based testing library for PHP. Install it:

composer require --dev giorgiosironi/eris

Eris provides generators for common data types:

use Eris\TestTrait;
use Eris\Generator;

class MoneyCalculatorTest extends TestCase
{
    use TestTrait;

    public function test_addition_is_commutative(): void
    {
        $this->forAll(
            Generator\pos(),   // positive integer
            Generator\pos()
        )->then(function (int $a, int $b) {
            $moneyA = Money::fromCents($a, 'USD');
            $moneyB = Money::fromCents($b, 'USD');

            // a + b should equal b + a
            $this->assertEquals(
                $moneyA->add($moneyB)->cents(),
                $moneyB->add($moneyA)->cents()
            );
        });
    }

    public function test_adding_zero_is_identity(): void
    {
        $this->forAll(
            Generator\pos()
        )->then(function (int $amount) {
            $money = Money::fromCents($amount, 'USD');
            $zero = Money::fromCents(0, 'USD');

            $this->assertEquals($amount, $money->add($zero)->cents());
        });
    }
}

These are mathematical properties (commutativity, identity element) that your arithmetic must satisfy. If your Money::add() implementation has a bug that only manifests for certain input sizes, Eris will find it.

Shrinking: Finding the Minimal Failing Case

One of the most powerful features of property-based testing is shrinking. When a randomly generated input fails a property, the framework automatically tries to find the smallest input that still causes the failure.

Suppose Eris finds that your invoice calculation fails for the input [487, 23, 991, 4, 128]. It will then try:

• [487, 23, 991, 4]
• [487, 23, 991]
• [487, 991]
• [991]

Until it finds the minimal case. You get a bug report like "this fails with input [991]" instead of "this fails with some random 50-element array."

Shrinking turns property-based testing failures into actionable bug reports.

Practical Generators

Generators are composable. Build complex generators from simple ones:

// Generate a valid email address
$email = Generator\map(
    Generator\tuple(
        Generator\elements(['user', 'admin', 'test']),
        Generator\elements(['example.com', 'test.org', 'mail.net'])
    ),
    fn($parts) => implode('@', $parts)
);

// Generate a valid invoice line item
$lineItem = Generator\bind(
    Generator\choose(1, 100),          // quantity
    fn($qty) => Generator\bind(
        Generator\choose(100, 100000), // unit price in cents
        fn($price) => Generator\constant([
            'quantity'   => $qty,
            'unit_price' => $price,
            'total'      => $qty * $price,
        ])
    )
);

$this->forAll($lineItem)->then(function (array $item) {
    $this->assertEquals($item['quantity'] * $item['unit_price'], $item['total']);
});

Properties Worth Testing in Business Applications

Property-based testing is often associated with pure functions and algorithms. But it's equally valuable in business applications. Properties to look for:

Invariants: Things that must always be true regardless of input.

// An invoice's total must always equal the sum of its line items
$this->forAll(
    Generator\vector(Generator\pos(), Generator\choose(1, 10))
)->then(function (array $amounts) {
    $invoice = Invoice::factory()->create();
    foreach ($amounts as $amount) {
        $invoice->addLineItem(amount: $amount);
    }
    $this->assertEquals(array_sum($amounts), $invoice->fresh()->total);
});

Idempotence: Operations that can be applied multiple times with the same result.

// Applying the same discount code twice should have the same effect as once
$this->forAll(
    Generator\elements(['SUMMER10', 'VIP20', 'FIRST50'])
)->then(function (string $code) {
    $invoice = Invoice::factory()->create(['total' => 1000]);

    $invoice->applyDiscount($code);
    $firstTotal = $invoice->total;

    $invoice->applyDiscount($code);
    $secondTotal = $invoice->total;

    $this->assertEquals($firstTotal, $secondTotal);
});

Round-trip properties: Serialize then deserialize should return the original value.

$this->forAll(
    Generator\string()
)->then(function (string $content) {
    $document = Document::fromMarkdown($content);
    $recovered = $document->toMarkdown();

    // The round-trip should preserve the content
    $this->assertEquals($content, $recovered);
});

Seeding for Reproducibility

Property-based tests use random input generation, which means failures can be hard to reproduce. Most frameworks support seeding:

// Eris: run with a fixed seed to reproduce a specific failure
$this->minimumEvaluationRatio(1.0)
    ->seed(42)  // Use the seed from a previous failure output
    ->forAll(
        Generator\pos()
    )->then(function (int $amount) {
        // ...
    });

When a property-based test fails, the framework reports the seed it used. Save that seed and rerun to reproduce the exact failure every time.

Integrating with Your CI Pipeline

Property-based tests are slower than example-based tests because they run each property many times. Configure the number of runs based on your CI budget:

// Reduce runs in CI for speed; increase locally for thoroughness
$runs = env('APP_ENV') === 'testing' ? 50 : 500;

$this->limitTo($runs)
    ->forAll(Generator\pos())
    ->then(function (int $amount) {
        // property test
    });

Run property-based tests in a dedicated test suite that runs less frequently:

<!-- phpunit.xml -->
<testsuite name="property">
    <directory>tests/Property</directory>
</testsuite>

# Run nightly in CI, not on every PR
php artisan test --testsuite=property

When Property-Based Testing Shines

Invest in property-based testing for:

• Financial calculations: Money arithmetic, tax calculations, discount stacking
• Data transformations: Parsers, formatters, encoders, serializers
• Sorting and filtering: Any function that rearranges or selects data
• State machines: Workflow transitions that must maintain invariants
• Cryptographic operations: Encoding, hashing, signing

Don't reach for property-based testing for:

• Testing UI rendering
• Testing third-party API behavior
• Simple CRUD operations with no business logic

Practical Takeaways

• Property-based testing generates hundreds of random inputs to find edge cases you didn't think to test for
• Eris is the standard PHP library; use it alongside your existing PHPUnit or Pest tests
• Shrinking automatically finds the minimal failing input, making failures actionable
• Focus on invariants, idempotence, and round-trip properties in business applications
• Run property tests in a separate suite on a schedule due to their longer runtime

Need help building reliable systems? We help teams architect software that scales. scopeforged.com