[Bug]: [mutate] Mutations are reported as tested when no test ran (--processes, filters past PCRE's limits, timeouts)

Aperta
#2,018 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
php
Ambito
testing, tooling

Direzione di ricerca

Start by tracing MutationTest::hasFinished(), MutationTest::start(), and ProcessesOption::remove(); the issue identifies these as the main entry points. Read the reproduction steps and compare child process arguments, filters, and PHPUnit event logs to understand when a mutation should count as tested. Done means the reproduced cases report untested mutations when no test starts, while covered mutations and timeouts after test preparation are classified correctly.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

Filing here because issues are disabled on pestphp/pest-plugin-mutate. Related to #1843, which hits the same root (a child that never ran a test is scored as killed) through a deadlock; the cases below reach it in other ways.

Pest 5.3.0 · pest-plugin-mutate 5.0.2 · PHPUnit 13.3.6 · PHP 8.4 with Xdebug · OS macOS 26 (arm64) and Ubuntu 26.04 (CI)

Summary

In v5.0.2 a mutation counts towards the score (tested, or timed out) whenever the child PHPUnit process exits non-zero or times out (MutationTest::hasFinished()), whatever the reason. Several ordinary situations make the child fail or hang before running a single test, so the mutation is reported as killed without having been tested and the score is inflated, up to 100 %. The project below reproduces each one from scratch; it needs PHP 8.4 with Xdebug (Pest 5.3.0, PHPUnit 13.3.6).

Run of repro.sh v5.0.2 with the changes below
1. --parallel --processes=2 8 tested, score 100 % 5 untested, 3 tested
2. A line covered by 400 definitions, in series 8 tested, score 100 % 5 untested, 3 tested
3. Dataset: does the case that never reaches the line run? yes no
4. A run that hangs before any test starts 1 timeout, score 100 % 1 untested, score 0 %

Numbers::label() has 5 mutations that no test kills (the tests only check a word both labels share), so 1 and 2 must report 5 untested.

Reproduction

composer.json

{
    "require-dev": {
        "pestphp/pest": "5.3.0",
        "pestphp/pest-plugin-mutate": "5.0.2",
        "phpunit/phpunit": "13.3.6"
    },
    "autoload": { "psr-4": { "App\\": "src/" } },
    "config": { "allow-plugins": { "pestphp/pest-plugin": true } }
}

phpunit.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" bootstrap="vendor/autoload.php" colors="false">
    <testsuites>
        <testsuite name="Repro">
            <directory>tests</directory>
        </testsuite>
    </testsuites>
    <source>
        <include>
            <directory>src</directory>
        </include>
    </source>
</phpunit>

src/Numbers.php

<?php

namespace App;

final class Numbers
{
    // Every mutation of this line is killed by the tests.
    public static function double(int $number): int
    {
        return $number * 2;
    }

    // No mutation of this line is killed: the tests only check a word both labels share.
    public static function label(int $number): string
    {
        return $number > 0 ? 'positive' : 'not positive';
    }
}

src/Discount.php

<?php

namespace App;

final class Discount
{
    public static function price(int $price, bool $member): int
    {
        if ($member) {
            return $price - 10; // only the "member" case reaches this line
        }

        return $price;
    }
}

src/Flag.php

<?php

namespace App;

final class Flag
{
    public static function enabled(): bool
    {
        return false;
    }
}

tests/ProcessesTest.php

<?php

use App\Numbers;

test('double and label', function () {
    expect(Numbers::double(3))->toBe(6)
        ->and(Numbers::label(3))->toContain('positive');
});

tests/WideLineTest.php

<?php

use App\Numbers;

// 400 definitions cover the same lines: one filter alternative each is past PCRE's pattern size limit.
foreach (range(1, 400) as $n) {
    test("definition {$n} checks the doubled number exactly and only a word both labels share", function () use ($n) {
        expect(Numbers::double($n))->toBe($n * 2)
            ->and(Numbers::label($n))->toContain('positive');
    });
}

tests/DatasetTest.php

<?php

use App\Discount;

test('discount', function (int $price, bool $member, int $expected) {
    if (! $member && getenv('PEST_MUTATION_TESTING') !== false) {
        touch(__DIR__.'/../guest-case-ran-in-a-mutation-run');
    }

    expect(Discount::price($price, $member))->toBe($expected);
})->with([
    'guest' => [100, false, 100],
    'member' => [100, true, 90],
]);

tests/TimeoutTest.php

<?php

use App\Flag;

// Loaded before any test is prepared: under the mutation, the run hangs here and never starts a test.
if (Flag::enabled()) {
    sleep(60);
}

test('flag is off', function () {
    expect(Flag::enabled())->toBeFalse();
});

repro.sh

#!/usr/bin/env bash
# Runs the four reproductions of pest-plugin-mutate 5.0.2 and prints each summary line.
cd "$(dirname "$0")"
export XDEBUG_MODE=coverage
mutate() { vendor/bin/pest --mutate --no-cache --colors=never "$@" 2>&1 | grep -E 'Mutations:|Score:' | sed 's/^ */    /'; }

echo '1. --processes reaches every mutation run (label() has 5 mutations no test kills):'
mutate --class='App\Numbers' --parallel --processes=2 tests/ProcessesTest.php

echo '2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):'
mutate --class='App\Numbers' tests/WideLineTest.php

echo '3. The filter names the definition, not the dataset case:'
rm -f guest-case-ran-in-a-mutation-run
mutate --class='App\Discount' --mutator='Pest\Mutate\Mutators\Arithmetic\MinusToPlus' tests/DatasetTest.php
[ -f guest-case-ran-in-a-mutation-run ] && echo '    the "guest" case ran in the mutation run' || echo '    only the "member" case ran'

echo '4. A run that hangs before any test starts:'
mutate --class='App\Flag' --mutator='Pest\Mutate\Mutators\Logical\FalseToTrue' tests/TimeoutTest.php

Then composer install && bash repro.sh. Output with v5.0.2:

1. --processes reaches every mutation run (label() has 5 mutations no test kills):
    Mutations: 8 tested
    Score:     100.00%
2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):
    Mutations: 8 tested
    Score:     100.00%
3. The filter names the definition, not the dataset case:
    Mutations: 1 tested
    Score:     100.00%
    the "guest" case ran in the mutation run
4. A run that hangs before any test starts:
    Mutations: 1 timeout, 0 tested
    Score:     100.00%
1. --processes is passed on to every mutation's run

ProcessesOption::remove() returns false, so --processes=N stays in the arguments that MutationTest::start() hands to each mutation's child, while --parallel is removed. PHPUnit rejects the leftover option:

'vendor/bin/pest' '--processes=2' 'tests/ProcessesTest.php' '--bail' '--filter="…"'
Unknown option "--processes". Most similar options are --covers, --colors, …

The child exits non-zero, so every covered mutation is reported as tested: a 100 % score. The option is still needed by the initial (paratest) run, so the fix is to drop it from the child's arguments, next to the --coverage* ones. Related: --processes 2 (with a space) is read as a bare --processes, which means every core, and 2 stays in each child's arguments as a test path.

2. The --filter of a widely covered line exceeds PCRE's size limit

The filter has one alternative per test definition that covers the mutated line. Past roughly 32 KB (in WideLineTest.php, 400 definitions with names of about 80 characters) the pattern no longer compiles. PHPUnit evaluates it with @preg_match (NameFilterIterator::accept()), so the error is silent: no test is selected, the child prints No tests found., exits 1, and the mutation counts as tested. This happens in series too, and it hits the lines most of a suite covers: models, middleware, shared helpers.

Suggested fix: check that the filter compiles before using it, with the delimiters PHPUnit will use, and otherwise fall back to one alternative per test class, then to no filter. Each step selects a superset of the previous one, so no covering test is left out.

3. The filter names the definition, not the dataset case

The id Class::__pest_evaluable_name#dataset "case" is cut at #, so a mutation runs every case of a dataset whose definition covers the line, even the cases that never reach it (guest in DatasetTest.php). Correct, but slow with large datasets. Appending the case's words to its alternative, separated by [^A-Za-z0-9]+, runs only the covering cases. Joining them with .* instead can exhaust PCRE's backtracking limit on long case names, and the covering case is then silently left out. We checked the [^A-Za-z0-9]+ form against the 10,070 test names of a real suite: every one is selected by its own filter.

4. The root of 1, 2 and 4: a failing or timed-out run is not proof of a killed mutation

Whatever goes wrong in a child before it runs a test (an unknown option, a filter that selects nothing, a bootstrap that throws or hangs), it should never count towards the score. We pass --log-events-text=<tmp file> to each child and look for a Test Preparation Started event in it:

  • the child failed and a test started: tested;
  • the child timed out and a test started: timeout, as today;
  • no test started, whether the child failed, passed or timed out: untested, so the problem shows up instead of inflating the score.

PHPUnit writes each event to that file as it happens, so the line is there even when the child is killed at the timeout.

5. In parallel mode the timeout is too short

The timeout of a mutation's run is the initial run's wall-clock time plus 20 %. With --parallel the initial run took a fraction of its tests' time, while each mutation runs its tests in one process. A mutation that escapes on a widely covered line can therefore time out, and timeouts count towards the score. We multiply the initial duration by the number of processes when running in parallel.

Smaller things found on the way
  • The children inherit XDEBUG_MODE=coverage from the initial run, which makes each mutation run about three times slower. They only need to pass or fail, so XDEBUG_MODE=off in their environment helps.
  • The filter is passed as --filter="…". With Process and an argument array, the quotes reach PHPUnit, which takes them as the pattern's delimiters. The filter is therefore case-sensitive and has no i flag. It works, but probably by accident.
  • The id regex \\([a-zA-Z0-9]*):: drops test classes whose short name has _ or non-ASCII letters, and classes without a namespace.

We can open a PR with these changes, a small TestFilter class plus changes to MutationTest. With them applied, repro.sh prints:

1. --processes reaches every mutation run (label() has 5 mutations no test kills):
    Mutations: 5 untested, 3 tested
    Score:     37.50%
2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):
    Mutations: 5 untested, 3 tested
    Score:     37.50%
3. The filter names the definition, not the dataset case:
    Mutations: 1 tested
    Score:     100.00%
    only the "member" case ran
4. A run that hangs before any test starts:
    Mutations: 1 untested, 0 tested
    Score:     0.00%
Lingua principale
PHP
Stelle
11.8k
Fork
559
Merge medio
5g 23h
PR unite (30g)
24

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pestphp/pest

Tutte le issue di pestphp/pest

Issue simili

Altre issue su PHP

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.