[Bug]: [mutate] Mutations are reported as tested when no test ran (--processes, filters past PCRE's limits, timeouts)
メンテナーはふだん 5 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 55/100
調査の方向性
Start by tracing MutationTest::hasFinished(), MutationTest::start(), and ProcessesOption::remove(); the issue identifies these as the main entry points. Read the reproduction steps and compare child process arguments, filters, and PHPUnit event logs to understand when a mutation should count as tested. Done means the reproduced cases report untested mutations when no test starts, while covered mutations and timeouts after test preparation are classified correctly.
索引モデルが issue の本文から書いたものです。
説明
Filing here because issues are disabled on pestphp/pest-plugin-mutate. Related to #1843, which hits the same root (a child that never ran a test is scored as killed) through a deadlock; the cases below reach it in other ways.
Pest 5.3.0 · pest-plugin-mutate 5.0.2 · PHPUnit 13.3.6 · PHP 8.4 with Xdebug · OS macOS 26 (arm64) and Ubuntu 26.04 (CI)
Summary
In v5.0.2 a mutation counts towards the score (tested, or timed out) whenever the child PHPUnit process exits non-zero or times out (MutationTest::hasFinished()), whatever the reason. Several ordinary situations make the child fail or hang before running a single test, so the mutation is reported as killed without having been tested and the score is inflated, up to 100 %. The project below reproduces each one from scratch; it needs PHP 8.4 with Xdebug (Pest 5.3.0, PHPUnit 13.3.6).
Run of repro.sh |
v5.0.2 | with the changes below |
|---|---|---|
1. --parallel --processes=2 |
8 tested, score 100 % | 5 untested, 3 tested |
| 2. A line covered by 400 definitions, in series | 8 tested, score 100 % | 5 untested, 3 tested |
| 3. Dataset: does the case that never reaches the line run? | yes | no |
| 4. A run that hangs before any test starts | 1 timeout, score 100 % | 1 untested, score 0 % |
Numbers::label() has 5 mutations that no test kills (the tests only check a word both labels share), so 1 and 2 must report 5 untested.
Reproduction
composer.json
{
"require-dev": {
"pestphp/pest": "5.3.0",
"pestphp/pest-plugin-mutate": "5.0.2",
"phpunit/phpunit": "13.3.6"
},
"autoload": { "psr-4": { "App\\": "src/" } },
"config": { "allow-plugins": { "pestphp/pest-plugin": true } }
}
phpunit.xml
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" bootstrap="vendor/autoload.php" colors="false">
<testsuites>
<testsuite name="Repro">
<directory>tests</directory>
</testsuite>
</testsuites>
<source>
<include>
<directory>src</directory>
</include>
</source>
</phpunit>
src/Numbers.php
<?php
namespace App;
final class Numbers
{
// Every mutation of this line is killed by the tests.
public static function double(int $number): int
{
return $number * 2;
}
// No mutation of this line is killed: the tests only check a word both labels share.
public static function label(int $number): string
{
return $number > 0 ? 'positive' : 'not positive';
}
}
src/Discount.php
<?php
namespace App;
final class Discount
{
public static function price(int $price, bool $member): int
{
if ($member) {
return $price - 10; // only the "member" case reaches this line
}
return $price;
}
}
src/Flag.php
<?php
namespace App;
final class Flag
{
public static function enabled(): bool
{
return false;
}
}
tests/ProcessesTest.php
<?php
use App\Numbers;
test('double and label', function () {
expect(Numbers::double(3))->toBe(6)
->and(Numbers::label(3))->toContain('positive');
});
tests/WideLineTest.php
<?php
use App\Numbers;
// 400 definitions cover the same lines: one filter alternative each is past PCRE's pattern size limit.
foreach (range(1, 400) as $n) {
test("definition {$n} checks the doubled number exactly and only a word both labels share", function () use ($n) {
expect(Numbers::double($n))->toBe($n * 2)
->and(Numbers::label($n))->toContain('positive');
});
}
tests/DatasetTest.php
<?php
use App\Discount;
test('discount', function (int $price, bool $member, int $expected) {
if (! $member && getenv('PEST_MUTATION_TESTING') !== false) {
touch(__DIR__.'/../guest-case-ran-in-a-mutation-run');
}
expect(Discount::price($price, $member))->toBe($expected);
})->with([
'guest' => [100, false, 100],
'member' => [100, true, 90],
]);
tests/TimeoutTest.php
<?php
use App\Flag;
// Loaded before any test is prepared: under the mutation, the run hangs here and never starts a test.
if (Flag::enabled()) {
sleep(60);
}
test('flag is off', function () {
expect(Flag::enabled())->toBeFalse();
});
repro.sh
#!/usr/bin/env bash
# Runs the four reproductions of pest-plugin-mutate 5.0.2 and prints each summary line.
cd "$(dirname "$0")"
export XDEBUG_MODE=coverage
mutate() { vendor/bin/pest --mutate --no-cache --colors=never "$@" 2>&1 | grep -E 'Mutations:|Score:' | sed 's/^ */ /'; }
echo '1. --processes reaches every mutation run (label() has 5 mutations no test kills):'
mutate --class='App\Numbers' --parallel --processes=2 tests/ProcessesTest.php
echo '2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):'
mutate --class='App\Numbers' tests/WideLineTest.php
echo '3. The filter names the definition, not the dataset case:'
rm -f guest-case-ran-in-a-mutation-run
mutate --class='App\Discount' --mutator='Pest\Mutate\Mutators\Arithmetic\MinusToPlus' tests/DatasetTest.php
[ -f guest-case-ran-in-a-mutation-run ] && echo ' the "guest" case ran in the mutation run' || echo ' only the "member" case ran'
echo '4. A run that hangs before any test starts:'
mutate --class='App\Flag' --mutator='Pest\Mutate\Mutators\Logical\FalseToTrue' tests/TimeoutTest.php
Then composer install && bash repro.sh. Output with v5.0.2:
1. --processes reaches every mutation run (label() has 5 mutations no test kills):
Mutations: 8 tested
Score: 100.00%
2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):
Mutations: 8 tested
Score: 100.00%
3. The filter names the definition, not the dataset case:
Mutations: 1 tested
Score: 100.00%
the "guest" case ran in the mutation run
4. A run that hangs before any test starts:
Mutations: 1 timeout, 0 tested
Score: 100.00%
1. --processes is passed on to every mutation's run
ProcessesOption::remove() returns false, so --processes=N stays in the arguments that MutationTest::start() hands to each mutation's child, while --parallel is removed. PHPUnit rejects the leftover option:
'vendor/bin/pest' '--processes=2' 'tests/ProcessesTest.php' '--bail' '--filter="…"'
Unknown option "--processes". Most similar options are --covers, --colors, …
The child exits non-zero, so every covered mutation is reported as tested: a 100 % score. The option is still needed by the initial (paratest) run, so the fix is to drop it from the child's arguments, next to the --coverage* ones. Related: --processes 2 (with a space) is read as a bare --processes, which means every core, and 2 stays in each child's arguments as a test path.
2. The --filter of a widely covered line exceeds PCRE's size limit
The filter has one alternative per test definition that covers the mutated line. Past roughly 32 KB (in WideLineTest.php, 400 definitions with names of about 80 characters) the pattern no longer compiles. PHPUnit evaluates it with @preg_match (NameFilterIterator::accept()), so the error is silent: no test is selected, the child prints No tests found., exits 1, and the mutation counts as tested. This happens in series too, and it hits the lines most of a suite covers: models, middleware, shared helpers.
Suggested fix: check that the filter compiles before using it, with the delimiters PHPUnit will use, and otherwise fall back to one alternative per test class, then to no filter. Each step selects a superset of the previous one, so no covering test is left out.
3. The filter names the definition, not the dataset case
The id Class::__pest_evaluable_name#dataset "case" is cut at #, so a mutation runs every case of a dataset whose definition covers the line, even the cases that never reach it (guest in DatasetTest.php). Correct, but slow with large datasets. Appending the case's words to its alternative, separated by [^A-Za-z0-9]+, runs only the covering cases. Joining them with .* instead can exhaust PCRE's backtracking limit on long case names, and the covering case is then silently left out. We checked the [^A-Za-z0-9]+ form against the 10,070 test names of a real suite: every one is selected by its own filter.
4. The root of 1, 2 and 4: a failing or timed-out run is not proof of a killed mutation
Whatever goes wrong in a child before it runs a test (an unknown option, a filter that selects nothing, a bootstrap that throws or hangs), it should never count towards the score. We pass --log-events-text=<tmp file> to each child and look for a Test Preparation Started event in it:
- the child failed and a test started: tested;
- the child timed out and a test started: timeout, as today;
- no test started, whether the child failed, passed or timed out: untested, so the problem shows up instead of inflating the score.
PHPUnit writes each event to that file as it happens, so the line is there even when the child is killed at the timeout.
5. In parallel mode the timeout is too short
The timeout of a mutation's run is the initial run's wall-clock time plus 20 %. With --parallel the initial run took a fraction of its tests' time, while each mutation runs its tests in one process. A mutation that escapes on a widely covered line can therefore time out, and timeouts count towards the score. We multiply the initial duration by the number of processes when running in parallel.
Smaller things found on the way
- The children inherit
XDEBUG_MODE=coveragefrom the initial run, which makes each mutation run about three times slower. They only need to pass or fail, soXDEBUG_MODE=offin their environment helps. - The filter is passed as
--filter="…". WithProcessand an argument array, the quotes reach PHPUnit, which takes them as the pattern's delimiters. The filter is therefore case-sensitive and has noiflag. It works, but probably by accident. - The id regex
\\([a-zA-Z0-9]*)::drops test classes whose short name has_or non-ASCII letters, and classes without a namespace.
We can open a PR with these changes, a small TestFilter class plus changes to MutationTest. With them applied, repro.sh prints:
1. --processes reaches every mutation run (label() has 5 mutations no test kills):
Mutations: 5 untested, 3 tested
Score: 37.50%
2. A line covered by 400 definitions, in series (same class, same 5 mutations no test kills):
Mutations: 5 untested, 3 tested
Score: 37.50%
3. The filter names the definition, not the dataset case:
Mutations: 1 tested
Score: 100.00%
only the "member" case ran
4. A run that hangs before any test starts:
Mutations: 1 untested, 0 tested
Score: 0.00%
- 主要言語
- PHP
- スター
- 11.8k
- フォーク
- 559
- 平均マージ
- 5日 23時間
- マージ済み PR(30日)
- 24
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
pestphp/pest のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 5 日以内に返信
-
[Bug]: --mutate hides unreadable files, so mutants behind a failed read can never be killed再び着手できるかも このイシューのプルリクエストはマージされずにクローズされました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
メンテナーはふだん 5 日以内に返信
-
[Bug]: [mutate] Windows: @pest-mutate-ignore comments land on line 1 in files with LF line endings対応中かも @webdevsamran が 57 日前に担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 5 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 80/100
メンテナーはふだん 5 日以内に返信
-
[pest-plugin-browser] Page::waitForLoadState() never sends its message — waitForEvent() is a silent no-op対応中かも @asllanmaciel が 70 日前に担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
pestphp/pest#1892 · コメント 2 件 ·
メンテナーはふだん 5 日以内に返信
似ている issue
-
nginx update required due to NGINX2Root対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープン
難易度 1/5 1時間未満 初心者へのやさしさ 65/100
mailcow/mailcow-dockerized#7668 · コメント 5 件 · リアクション 2 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
lara-zeus/spatie-translatable#80 ·
メンテナーはふだん 1 日以内に返信
-
small
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
uiowa/uiowa#10296 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
asyncapi/modelina#2709 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
My-Little-Forum/mylittleforum#937 ·