A run that ends with exit code 2 leaves the reports of an earlier run in `build/`

Open Beginner friendly
#20 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
cli, testing-qa

Research direction

Reproduce first: run cobalto test --filter TSTBATCH in examples/, then add the EXPECT WS-T-COUNT TO BE FROBNICATED line to tests/LOANBTCH.cblt and rerun to see exit code 2 leave build/results.json and build/junit.xml untouched. Read the test and report command entry points to find where they stop before writing reports (the same place cobalto suggest deletes results.json each round), and clear the report files there. Add a case to tests/cli/selftest.py asserting no earlier report survives exit code 2, and update the Output section of docs/reference.md and docs/coding-agents.md.

Written by the indexing model from the issue text.

Description

bug

When cobalto test stops with exit code 2 before it reads any results (a configuration error, a DSL syntax error or a compile error), it writes none of its reports, and it does not remove the ones already in build/. results.json, junit.xml, sonar-tests.xml and, with --coverage, lcov.info, coverage.xml and sonar-coverage.xml stay as the last run that got that far left them.

Why it matters

This is a silent pass waiting to happen:

  • A CI step that publishes build/junit.xml whatever the exit code, for example with if: always() in GitHub Actions, shows the earlier run while the suite does not even compile. If the earlier run was green, so is the report.
  • A script or a coding agent that reads build/results.json without checking the exit code reads the "ok" of an earlier run.

cobalto suggest already works around it: it deletes results.json before each round, so that a round that does not compile is never read as the one before it.

How to reproduce it

In examples/, with the default profile:

cobalto test --filter TSTBATCH

It ends with exit code 1, because of the intentional failure, and writes build/results.json and build/junit.xml. Then add this line right after the first TESTCASE of tests/LOANBTCH.cblt:

           EXPECT WS-T-COUNT TO BE FROBNICATED

and run the same command again. It ends with exit code 2:

CBLT-E201: compile error in tests/LOANBTCH.cblt:67 (generated at build/TSTBATCH.cbl:162): 'FROBNICATED' is not defined
CBLT-E202: the build failed (exit 1)

build/results.json and build/junit.xml keep the content and the modification time of the first run.

One possible shape

Remove the reports at the start of cobalto test and cobalto report, before anything can fail, so that after exit code 2 there is no report instead of an old one. Writing a report that records the error is the other option. Either way, a scenario in tests/cli/selftest.py should check that no report from an earlier run survives an exit code 2.

The documentation added for #15 describes the current behavior, in the Output section of docs/reference.md and in docs/coding-agents.md. Both need updating with the fix.

Versions

  • Cobalto 0.7.0 (main at 4dcb631)
  • GnuCOBOL 3.2
  • macOS
  • Profile gnucobol
Dominant language
Python
Stars
2
Forks
0
Avg merge
4h 51m
Merged PRs (30d)
9

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from abstracta/cobalto

All issues in abstracta/cobalto

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.