Tests that keep passing when the code they cover is broken
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- bash, react, typescript
- Domain
- testing-qa
Research direction
Start with test/auth.test.tsx and the instances linked to #779, #782, #785, and #766; run the relevant tests and review the described deliberate mutations in isolation. Done means documenting an agreed convention for validating failure-guarding tests, including isolated mutation checks, or recording whether Stryker is worth evaluating.
Written by the indexing model from the issue text.
Description
Several checks in this repo have turned out to pass whether or not the thing they name works. They were found one at a time during review, so this issue collects them and proposes a convention.
A test that asserts nothing is worse than a missing one, because it reads as coverage.
Instances, and who verified each
The code never executes. In test/auth.test.tsx, a test rendered <AuthCheck> while beforeEach had signed the user out, so the fallback rendered, UserDetails never mounted, and its two expect calls never ran. Armando re-confirmed on v5 by putting an assertion that cannot pass inside UserDetails: the suite still went green. Fixed in #782.
The test passes when the thing it names is broken. Three, all confirmed by mutation:
- The
initialDatabranch ofgetServerSnapshotwas dead under test. Neuter it to always returnloadingand all 22 tests still passed. Found by Armando on #779, which is still open. does not show a logged-out user after navigating awaysits indescribe('useUser')but stopped callinguseUserwhen #782 replaced its wrapper. MakinguseUserthrow left it passing on that branch while the same mutation failed it onv5. Fixed in #782.- The packed-artifact load check only exercised
import(). Breaking therequire()half deliberately left every test green. Found on #766. That code has since been removed from the PR, so this one never landed.
The harness cannot report a failure at all. The flake probe ran its test command under bash -e without a set +e guard, so the first failing iteration killed the step before the result was recorded. It could only ever produce a clean table, and the first run that genuinely reproduced the flake would have reported least. Fixed in #785.
The test catches a mutation for the wrong reason. On the startWithValue removal branch, a test appeared to catch a deliberate break but passed under that same mutation when run in isolation: the failure came from another test's warning in the shared suite. It would have surfaced as an order-dependent CI flake.
What would catch these
A convention rather than a framework: any test whose purpose is to guard a specific failure should be shown to fail against a deliberate break of that failure, and the PR should say so. That is what caught four of the five above, and it costs one run.
Two caveats worth stating. Mutating in the suite is not enough on its own, since coupling between tests can produce the failure for an unrelated reason, so run the mutation in isolation as well. And this only covers tests written deliberately as guards; it says nothing about coverage that quietly evaporates when a wrapper changes, which is what happened in #782.
If we want a tool rather than a convention, mutation testing is off-the-shelf for TS (Stryker), and that is worth pricing before writing anything bespoke.
- Dominant language
- TypeScript
- Stars
- 3.6k
- Forks
- 403
- Avg merge
- 4d 19h
- Merged PRs (30d)
- 11
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from FirebaseExtended/reactfire
-
Difficulty 5/5 Over a week Newbie friendliness 38/100
FirebaseExtended/reactfire#801 ·
Maintainers usually reply within 2 days
-
v5
Difficulty 4/5 3-5 days Newbie friendliness 48/100
FirebaseExtended/reactfire#793 ·
Maintainers usually reply within 2 days
-
v5
Difficulty 4/5 3-5 days Newbie friendliness 35/100
FirebaseExtended/reactfire#790 · 2 comments ·
Maintainers usually reply within 2 days
-
v5
Difficulty 4/5 3-5 days Newbie friendliness 55/100
FirebaseExtended/reactfire#789 ·
Maintainers usually reply within 2 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
FirebaseExtended/reactfire#784 ·
Maintainers usually reply within 2 days
All issues in FirebaseExtended/reactfire
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
lichess-org/api#678 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
PostHog/posthog.com#20628 ·
Maintainers usually reply within 1 day
-
bug status:Needs Triage
Difficulty 1/5 Under an hour Newbie friendliness 92/100
jupyterlab/jupyterlab#19964 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
agentscope-ai/QwenPaw#8064 · 1 comment ·
Maintainers usually reply within 1 day
-
area: notebooks-jupyter bug theme: new notebook frontend
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
posit-dev/positron#16347 · 1 comment ·
Maintainers usually reply within 1 day