[Bug]: Browser plugin: a test that fails on its first check is reported as risky, and an earlier page's server exception reaches later pages

Aperta
#2,009 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
laravel, php
Ambito
testing-qa

Direzione di ricerca

The fixes sit in the browser plugin's sources (the issue says its tracker is closed, so the work lands upstream): src/Api/Concerns/MakesElementAssertions.php for assertSee(), src/Drivers/LaravelHttpServer.php for $lastThrowable, and src/Plugin.php for afterEach. Start by running the two reproduction tests from the issue on a Laravel app with the plugin installed. Done means the first-check failure reports 1 assertion with no risky event, and the second test fails on Goodbye without the earlier server exception in its message.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

pestphp/pest-plugin-browser has issues turned off, so I file this here.
Each case below reproduces in a fresh laravel/laravel app with only Pest and the browser plugin added, on plugin 5.0.1 and on 5.2.0 (the 5.x head).
Line numbers are for 5.2.0, with 5.0.1 in brackets where it differs.

Versions: pestphp/pest-plugin-browser 5.0.1 and 5.2.0, pestphp/pest 5.2.1, phpunit/phpunit 13.3.4, PHP 8.5.8, Playwright 1.63.0 (Chromium), laravel/framework 13.39.0, macOS.
Routes used by the tests:

Route::get('/ok', fn () => 'Hello');
Route::get('/fail', fn () => throw new RuntimeException('Thrown by /fail.'));
1. A test that fails on its first browser check is also risky
it('fails on its first check', function () {
    visit('/ok')->assertSee('Goodbye');
});

Run: vendor/bin/pest tests/Browser/FirstCheckFailsTest.php --log-junit junit.xml --log-events-text events.txt
Expected: 1 failed test with 1 assertion and no risky event, as for a failed PHPUnit, Pest or Laravel assertion.
Actual: Tests: 1 failed (0 assertions), assertions="0" in junit.xml, and in events.txt:

Test Considered Risky (P\Tests\Browser\FirstCheckFailsTest::__pest_evaluable_it_fails_on_its_first_check)
This test did not perform any assertions

laravel/pao 1.1.5 reports "assertions":0,"failed":1,"risky":1.
A failed first action does the same: visit('/ok')->click('No such button') fails with Timeout 5000ms exceeded., 0 assertions, risky.
A test that passes one check before it fails is not affected.

Cause: these calls throw ExpectationFailedException directly, with no assertion counted: assertSee() and assertDontSee() (src/Api/Concerns/MakesElementAssertions.php:61, :79), which count one only when they pass, a failed action in Client::execute() (src/Playwright/Client.php:127 [106]), GuessLocator::for() (src/Support/GuessLocator.php:75 [65]) and ArrayablePendingAwaitablePage::ensureOnWebpage() (src/Api/ArrayablePendingAwaitablePage.php:151).
PHPUnit's own assertions count before they throw, and PHPUnit marks a test with 0 assertions as risky unless it errored, was skipped or is incomplete (src/Framework/TestRunner/TestRunner.php:224-234), so a failed test is not exempt.

Suggested fix: count one assertion before each of these throws, as assertPortIs() already does (src/Api/Concerns/MakesUrlAssertions.php:109-110).
With expect(true)->toBeTrue(); before the throw in assertSee(), the test above gives 1 failed (1 assertions) and no risky event.

2. The server exception of an earlier page reaches later pages of the same test
it('does not throw the exception of an earlier page again', function () {
    $this->withoutExceptionHandling();

    expect(fn () => visit('/fail')->assertSee('Server Error'))
        ->toThrow(RuntimeException::class, 'Thrown by /fail.');

    visit('/ok')->assertSee('Hello');
});

it('names only the exception of the page that failed', function () {
    visit('/no-such-page')->assertSee('Not Found');

    visit('/ok')->assertSee('Goodbye');
});

Expected: the first test passes, and the second fails on Goodbye with no server exception in its message, because /ok throws nothing.
Actual: the first test fails with RuntimeException: Thrown by /fail., thrown again by visit('/ok')->assertSee('Hello'); the same call passes in a test of its own.
The failure message of the second test contains:

The following exception was thrown by the HTTP server:
Symfony\Component\HttpKernel\Exception\NotFoundHttpException: The route no-such-page could not be found.

Cause: LaravelHttpServer keeps the last request exception in $lastThrowable (src/Drivers/LaravelHttpServer.php:352, :368 [281, 293]) and clears it only in flush(), which runs in the plugin's afterEach (src/Plugin.php:55 [53]).
Until then, AwaitableWebpage::__call() throws it again after each page call under withoutExceptionHandling() (src/Api/AwaitableWebpage.php:89, :100 [61, 72]), and each failed check adds it to its message (src/Exceptions/BrowserExpectationFailedException.php:60 [51]).

Suggested fix: clear $lastThrowable when throwLastThrowableIfNeeded() throws it, and when Page::goto() starts a new page load.
With both changes, both tests behave as expected, and visit('/fail')->assertSee('Goodbye') still names RuntimeException: Thrown by /fail. in its message.

3. Request: keep the plugin's frames out of failure traces

The failure of the test in case 1 is located in the plugin: the trace in junit.xml starts at vendor/pestphp/pest-plugin-browser/src/Exceptions/BrowserExpectationFailedException.php:69 [60], and laravel/pao reports that file and line.
Pest puts its own src on PHPUnit's exclude list (src/Bootstrappers/BootExcludeList.php), but the plugin does not.
With PHPUnit\Util\ExcludeList::addDirectory() for the plugin's src, the trace is only at tests/Browser/FirstCheckFailsTest.php:4.
PHPUnit then also ignores the warnings that the plugin suppresses with @, such as the rmdir() warnings that pestphp/pest-plugin-browser#250 and pestphp/pest-plugin-browser#278 fixed in 5.2.0.

Lingua principale
PHP
Stelle
11.8k
Fork
559
Merge medio
5g 23h
PR unite (30g)
24

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pestphp/pest

Tutte le issue di pestphp/pest

Issue simili

Altre issue su PHP

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.