[Bug]: [mutate] Mutations are reported as tested when no test ran (--processes, filters past PCRE's limits, timeouts)

Đang mở
#2,018 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 5 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
55/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
php
Lĩnh vực
testing, tooling

Hướng nghiên cứu

Start by tracing MutationTest::hasFinished(), MutationTest::start(), and ProcessesOption::remove(); the issue identifies these as the main entry points. Read the reproduction steps and compare child process arguments, filters, and PHPUnit event logs to understand when a mutation should count as tested. Done means the reproduced cases report untested mutations when no test starts, while covered mutations and timeouts after test preparation are classified correctly.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug

Filing here because issues are disabled on pestphp/pest-plugin-mutate. Related to #1843, which hits the same root (a child that never ran a test is scored as killed) through a deadlock; the cases below reach it in other ways.

Pest 5.3.0 · pest-plugin-mutate 5.0.2 · PHPUnit 13.3.6 · PHP 8.4 with Xdebug · OS macOS 26 (arm64) and Ubuntu 26.04 (CI)

Summary

In v5.0.2 a mutation counts towards the score (tested, or timed out) whenever the child PHPUnit process exits non-zero or times out (MutationTest::hasFinished()), whatever the reason. Several ordinary situations make the child fail or hang before running a single test, so the mutation is reported as killed without having been tested and the score is inflated, up to 100 %. The project below reproduces each one from scratch; it needs PHP 8.4 with Xdebug (Pest 5.3.0, PHPUnit 13.3.6).

Run of repro.sh v5.0.2 with the changes below
1. --parallel --processes=2 8 tested, score 100 % 5 untested, 3 tested
2. A line covered by 400 definitions, in series 8 tested, score 100 % 5 untested, 3 tested
3. Dataset: does the case that never reaches the line run? yes no
4. A run that hangs before any test starts 1 timeout, score 100 % 1 untested, score 0 %

Numbers::label() has 5 mutations that no test kills (the tests only check a word both labels share), so 1 and 2 must report 5 untested.

Reproduction

composer.json

{
    "require-dev": {
        "pestphp/pest": "5.3.0",
        "pestphp/pest-plugin-mutate": "5.0.2",
        "phpunit/phpunit": "13.3.6"
    },
    "autoload": { "psr-4": { "App\\": "src/" } },
    "config": { "allow-plugins": { "pestphp/pest-plugin": true } }
}

phpunit.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" bootstrap="vendor/autoload.php" colors="false">
    <testsuites>
        <testsuite name="Repro">
            <directory>tests</directory>
        </testsuite>
    </testsuites>
    <source>
        <include>
            <directory>src</directory>
        </include>
    </source>
</phpunit>

src/Numbers.php

<?php

namespace App;

final class Numbers
{
    // Every mutation of this line is killed by the tests.
    public static function double(int $number): int
    {
        return $number * 2;
    }

    // No mutation of this line is killed: the tests only check a word both labels share.
    public static function label(int $number): string
    {
        return $number > 0 ? 'positive' : 'not positive';
    }
}

src/Discount.php

<?php

namespace App;

final class Discount
{
    public static function price(int $price, bool $member): int
    {
        if ($member) {
            return $price - 10; // only the "member" case reaches this line
        }

        return $price;
    }
}

src/Flag.php

<?php

namespace App;

final class Flag
{
    public static function enabled(): bool
    {
        return false;
    }
}

tests/ProcessesTest.php

<?php

use App\Numbers;

test('double and label', function () {
    expect(Numbers::double(3))->toBe(6)
        ->and(Numbers::label(3))->toContain('positive');
});

tests/WideLineTest.php

<?php

use App\Numbers;

// 400 definitions cover the same lines: one filter alternative each is past PCRE's pattern size limit.
foreach (range(1, 400) as $n) {
    test("definition {$n} checks the doubled number exactly and only a word both labels share", function () use ($n) {
        expect(Numbers::double($n))->toBe($n * 2)
            ->and(Numbers::label($n))->toContain('positive');
    });
}

tests/DatasetTest.php

<?php

use App\Discount;

test('discount', function (int $price, bool $member, int $expected) {
    if (! $member && getenv('PEST_MUTATION_TESTING') !== false) {
        touch(__DIR__.'/../guest-case-ran-in-a-mutation-run');
    }

    expect(Discount::price($price, $member))->toBe($expected);
})->with([
    'guest' => [100, false, 100],
    'member' => [100, true, 90],
]);

tests/TimeoutTest.php

<?php

use App\Flag;

// Loaded before any test is prepared: under the mutation, the run hangs here and never starts a test.
if (Flag::enabled()) {
    sleep(60);
}

test('flag is off', function () {
    expect(Flag::enabled())->toBeFalse();
});

repro.sh

#!/usr/bin/env bash
# Runs the four reproductions of pest-plugin-mutate 5.0.2 and prints each summary line.
cd "$(dirname "$0")"
export XDEBUG_MODE=coverage
mutate() { vendor/bin/pest --mutate --no-cache --colors=never "$@" 2>&1 | grep -E 'Mutations:|Score:' | sed 's/^ */    /'; }

echo '1. --processes reaches every mutation run (label() has 5 mutations no test kills):'
mutate --class='App\Numbers' --parallel --processes=2 tests/ProcessesTest.php

echo '2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):'
mutate --class='App\Numbers' tests/WideLineTest.php

echo '3. The filter names the definition, not the dataset case:'
rm -f guest-case-ran-in-a-mutation-run
mutate --class='App\Discount' --mutator='Pest\Mutate\Mutators\Arithmetic\MinusToPlus' tests/DatasetTest.php
[ -f guest-case-ran-in-a-mutation-run ] && echo '    the "guest" case ran in the mutation run' || echo '    only the "member" case ran'

echo '4. A run that hangs before any test starts:'
mutate --class='App\Flag' --mutator='Pest\Mutate\Mutators\Logical\FalseToTrue' tests/TimeoutTest.php

Then composer install && bash repro.sh. Output with v5.0.2:

1. --processes reaches every mutation run (label() has 5 mutations no test kills):
    Mutations: 8 tested
    Score:     100.00%
2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):
    Mutations: 8 tested
    Score:     100.00%
3. The filter names the definition, not the dataset case:
    Mutations: 1 tested
    Score:     100.00%
    the "guest" case ran in the mutation run
4. A run that hangs before any test starts:
    Mutations: 1 timeout, 0 tested
    Score:     100.00%
1. --processes is passed on to every mutation's run

ProcessesOption::remove() returns false, so --processes=N stays in the arguments that MutationTest::start() hands to each mutation's child, while --parallel is removed. PHPUnit rejects the leftover option:

'vendor/bin/pest' '--processes=2' 'tests/ProcessesTest.php' '--bail' '--filter="…"'
Unknown option "--processes". Most similar options are --covers, --colors, …

The child exits non-zero, so every covered mutation is reported as tested: a 100 % score. The option is still needed by the initial (paratest) run, so the fix is to drop it from the child's arguments, next to the --coverage* ones. Related: --processes 2 (with a space) is read as a bare --processes, which means every core, and 2 stays in each child's arguments as a test path.

2. The --filter of a widely covered line exceeds PCRE's size limit

The filter has one alternative per test definition that covers the mutated line. Past roughly 32 KB (in WideLineTest.php, 400 definitions with names of about 80 characters) the pattern no longer compiles. PHPUnit evaluates it with @preg_match (NameFilterIterator::accept()), so the error is silent: no test is selected, the child prints No tests found., exits 1, and the mutation counts as tested. This happens in series too, and it hits the lines most of a suite covers: models, middleware, shared helpers.

Suggested fix: check that the filter compiles before using it, with the delimiters PHPUnit will use, and otherwise fall back to one alternative per test class, then to no filter. Each step selects a superset of the previous one, so no covering test is left out.

3. The filter names the definition, not the dataset case

The id Class::__pest_evaluable_name#dataset "case" is cut at #, so a mutation runs every case of a dataset whose definition covers the line, even the cases that never reach it (guest in DatasetTest.php). Correct, but slow with large datasets. Appending the case's words to its alternative, separated by [^A-Za-z0-9]+, runs only the covering cases. Joining them with .* instead can exhaust PCRE's backtracking limit on long case names, and the covering case is then silently left out. We checked the [^A-Za-z0-9]+ form against the 10,070 test names of a real suite: every one is selected by its own filter.

4. The root of 1, 2 and 4: a failing or timed-out run is not proof of a killed mutation

Whatever goes wrong in a child before it runs a test (an unknown option, a filter that selects nothing, a bootstrap that throws or hangs), it should never count towards the score. We pass --log-events-text=<tmp file> to each child and look for a Test Preparation Started event in it:

  • the child failed and a test started: tested;
  • the child timed out and a test started: timeout, as today;
  • no test started, whether the child failed, passed or timed out: untested, so the problem shows up instead of inflating the score.

PHPUnit writes each event to that file as it happens, so the line is there even when the child is killed at the timeout.

5. In parallel mode the timeout is too short

The timeout of a mutation's run is the initial run's wall-clock time plus 20 %. With --parallel the initial run took a fraction of its tests' time, while each mutation runs its tests in one process. A mutation that escapes on a widely covered line can therefore time out, and timeouts count towards the score. We multiply the initial duration by the number of processes when running in parallel.

Smaller things found on the way
  • The children inherit XDEBUG_MODE=coverage from the initial run, which makes each mutation run about three times slower. They only need to pass or fail, so XDEBUG_MODE=off in their environment helps.
  • The filter is passed as --filter="…". With Process and an argument array, the quotes reach PHPUnit, which takes them as the pattern's delimiters. The filter is therefore case-sensitive and has no i flag. It works, but probably by accident.
  • The id regex \\([a-zA-Z0-9]*):: drops test classes whose short name has _ or non-ASCII letters, and classes without a namespace.

We can open a PR with these changes, a small TestFilter class plus changes to MutationTest. With them applied, repro.sh prints:

1. --processes reaches every mutation run (label() has 5 mutations no test kills):
    Mutations: 5 untested, 3 tested
    Score:     37.50%
2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):
    Mutations: 5 untested, 3 tested
    Score:     37.50%
3. The filter names the definition, not the dataset case:
    Mutations: 1 tested
    Score:     100.00%
    only the "member" case ran
4. A run that hangs before any test starts:
    Mutations: 1 untested, 0 tested
    Score:     0.00%
Ngôn ngữ chính
PHP
Star
11.8k
Fork
559
Merge trung bình
5 ngày 23 giờ
Pull request đã merge (30 ngày)
24

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của pestphp/pest

Tất cả issue của pestphp/pest

Issue tương tự

Thêm issue về PHP

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.