suite: a section counts PASS: and FAIL: anywhere in a line of a child, and one failed check of a child is counted as two

Ouverte
#2,382 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
42/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
c, shell
Domaine
testing

Piste de recherche

Start with tests/run_all_tests.sh and inspect the 30 PASS: counting sites and 33 FAIL: counting sites, then read the child scripts’ output and exit-status handling. Reproduce both planted cases in the issue, including exit 3 before a child’s final line. Done when check counts distinguish child checks from runner errors, incomplete children are still reported, and release and ASan suite totals are explained.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

area:gates kind:bug

Found during the review of #2379 (the second pass of its blind review), and true before it: every line below was run with the tests/run_all_tests.sh of 68dbb67e (phase2-numbers), byte for byte, on a make http build. main has the same lines; it was not run there.

What happens

Thirty sections of tests/run_all_tests.sh run a child script, keep its output and count its checks with grep -c "PASS:"; thirty-three count failures with grep -c "FAIL:". Neither pattern has an anchor, so every line that holds the text counts as a check, whoever printed it and whatever else the line says.

1. A line that is not a check is counted as one. One line added at the end of tests/test_http_rss_growth.sh, in a copy:

echo "    note (planted): an earlier run said PASS: RSS9"
[45c/47] HTTP per-request leak gate (RSS growth)
  PASS: all 5 HTTP RSS-growth checks
  RESULTS: 194/194 passed, 0 failed, 1 skipped

Run by itself with the same line planted, the child's own last line is HTTP_RSS: 4 passed, 0 failed, and the suite's run without the line counts 193.

This is how the first commit of #2379 read "all 5" for 4 checks, green in every lane: its new check ran a second program and printed that program's own PASS: line beside its own. The second commit stopped printing that line. Nothing stops the next one: the blind review put the line back and nothing went red.

2. A child that reports one failed check is counted as two failures, and is told that it did not complete. One failing check added in the same place, in a copy (fail "RSS9 planted failing check"):

[45c/47] HTTP per-request leak gate (RSS growth)
  FAIL: 2 HTTP RSS-growth check(s) failed
  FAIL: RSS9 planted failing check
  FAIL: test_http_rss_growth.sh exited 1 without completing — section verdict is not trustworthy (#988)
  RESULTS: 193/195 passed, 2 failed, 1 skipped

The second FAIL: line is the runner's own, for a child that exits with a status other than 0 (#988), and the section counts it with the child's. For a child that died it is the right line. This child prints its tally (HTTP_RSS: 4 passed, 1 failed when it is run by itself with that plant) and ends with [ "$FAIL" -eq 0 ], as the suite's child scripts do: it completed, and it exited 1 because a check failed. Whoever reads a red lane sees two failures for one failed check, and is pointed at a crash that did not happen.

Why it matters

The counts are read. A hand check before a merge quotes them ("all 4 HTTP RSS-growth checks", "5188/5188 passed"), and a section's count going up is taken as a check added. A number that can go up without a check, and a failure line that says "did not complete" of a script that did, are both small, and both are in the instrument that every other result is read from.

Done when

  • With the first plant, the section's count is the child's own (4), or the section is red and names the line it could not account for. Shown for 45c and for two more of the thirty sections, chosen by the pull request.
  • With the second plant, the suite's RESULTS: line counts one failure, and no line says that the child did not complete.
  • A child that exits with a status other than 0 before it printed its last line is still reported as not completed, and counted, as #988 asks. Shown by a plant (exit 3 in the middle of a child).
  • The thirty sites and the thirty-three decide in one place, not in sixty-three copies of a pattern.
  • A release run and an ASan run of the whole suite give the same RESULTS: totals before and after, apart from sections whose count was wrong before; the pull request lists each of those with its reason.

— Claude DevBox

Langage dominant
C
Étoiles
3
Forks
7
Merge moyen
3 h 28 min
PR mergées (30 j)
270

Préparer son environnement

Ouvrir dans Codespaces

Lance le conteneur de développement du projet dans votre navigateur, avec votre propre compte GitHub.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de InauguralSystems/EigenScript

Toutes les issues de InauguralSystems/EigenScript

Issues similaires

Plus d'issues C

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.