Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Tests that keep passing when the code they cover is broken

Aberta
  • Algo errado?
#788 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 2 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Funcionalidade
Clareza
Precisa de esclarecimento
Status de atividade
Pouca atividade
Stack de tecnologia
bash, react, typescript
Domínio
testing-qa

Direção de pesquisa

Comece com test/auth.test.tsx e as instâncias vinculadas a #779, #782, #785 e #766; execute os testes relevantes e revise isoladamente as mutações deliberadas descritas. Considera-se concluído quando uma convenção acordada para validar testes que protegem contra falhas, incluindo verificações de mutações isoladas, estiver documentada, ou quando for registrado se vale a pena avaliar o Stryker.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Several checks in this repo have turned out to pass whether or not the thing they name works. They were found one at a time during review, so this issue collects them and proposes a convention.

A test that asserts nothing is worse than a missing one, because it reads as coverage.

Instances, and who verified each

The code never executes. In test/auth.test.tsx, a test rendered <AuthCheck> while beforeEach had signed the user out, so the fallback rendered, UserDetails never mounted, and its two expect calls never ran. Armando re-confirmed on v5 by putting an assertion that cannot pass inside UserDetails: the suite still went green. Fixed in #782.

The test passes when the thing it names is broken. Three, all confirmed by mutation:

  • The initialData branch of getServerSnapshot was dead under test. Neuter it to always return loading and all 22 tests still passed. Found by Armando on #779, which is still open.
  • does not show a logged-out user after navigating away sits in describe('useUser') but stopped calling useUser when #782 replaced its wrapper. Making useUser throw left it passing on that branch while the same mutation failed it on v5. Fixed in #782.
  • The packed-artifact load check only exercised import(). Breaking the require() half deliberately left every test green. Found on #766. That code has since been removed from the PR, so this one never landed.

The harness cannot report a failure at all. The flake probe ran its test command under bash -e without a set +e guard, so the first failing iteration killed the step before the result was recorded. It could only ever produce a clean table, and the first run that genuinely reproduced the flake would have reported least. Fixed in #785.

The test catches a mutation for the wrong reason. On the startWithValue removal branch, a test appeared to catch a deliberate break but passed under that same mutation when run in isolation: the failure came from another test's warning in the shared suite. It would have surfaced as an order-dependent CI flake.

What would catch these

A convention rather than a framework: any test whose purpose is to guard a specific failure should be shown to fail against a deliberate break of that failure, and the PR should say so. That is what caught four of the five above, and it costs one run.

Two caveats worth stating. Mutating in the suite is not enough on its own, since coupling between tests can produce the failure for an unrelated reason, so run the mutation in isolation as well. And this only covers tests written deliberately as guards; it says nothing about coverage that quietly evaporates when a wrapper changes, which is what happened in #782.

If we want a tool rather than a convention, mutation testing is off-the-shelf for TS (Stryker), and that is worth pricing before writing anything bespoke.

Linguagem predominante
TypeScript
Estrelas
3.6k
Forks
402
Merge médio
2d 18h
PRs com merge (30d)
17

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de FirebaseExtended/reactfire

Todas as issues de FirebaseExtended/reactfire

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.