A run that ends with exit code 2 leaves the reports of an earlier run in `build/`
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- cli, testing-qa
Research direction
Reproduce first: run cobalto test --filter TSTBATCH in examples/, then add the EXPECT WS-T-COUNT TO BE FROBNICATED line to tests/LOANBTCH.cblt and rerun to see exit code 2 leave build/results.json and build/junit.xml untouched. Read the test and report command entry points to find where they stop before writing reports (the same place cobalto suggest deletes results.json each round), and clear the report files there. Add a case to tests/cli/selftest.py asserting no earlier report survives exit code 2, and update the Output section of docs/reference.md and docs/coding-agents.md.
Written by the indexing model from the issue text.
Description
When cobalto test stops with exit code 2 before it reads any results (a configuration error, a DSL syntax error or a compile error), it writes none of its reports, and it does not remove the ones already in build/. results.json, junit.xml, sonar-tests.xml and, with --coverage, lcov.info, coverage.xml and sonar-coverage.xml stay as the last run that got that far left them.
Why it matters
This is a silent pass waiting to happen:
- A CI step that publishes
build/junit.xmlwhatever the exit code, for example withif: always()in GitHub Actions, shows the earlier run while the suite does not even compile. If the earlier run was green, so is the report. - A script or a coding agent that reads
build/results.jsonwithout checking the exit code reads the"ok"of an earlier run.
cobalto suggest already works around it: it deletes results.json before each round, so that a round that does not compile is never read as the one before it.
How to reproduce it
In examples/, with the default profile:
cobalto test --filter TSTBATCH
It ends with exit code 1, because of the intentional failure, and writes build/results.json and build/junit.xml. Then add this line right after the first TESTCASE of tests/LOANBTCH.cblt:
EXPECT WS-T-COUNT TO BE FROBNICATED
and run the same command again. It ends with exit code 2:
CBLT-E201: compile error in tests/LOANBTCH.cblt:67 (generated at build/TSTBATCH.cbl:162): 'FROBNICATED' is not defined
CBLT-E202: the build failed (exit 1)
build/results.json and build/junit.xml keep the content and the modification time of the first run.
One possible shape
Remove the reports at the start of cobalto test and cobalto report, before anything can fail, so that after exit code 2 there is no report instead of an old one. Writing a report that records the error is the other option. Either way, a scenario in tests/cli/selftest.py should check that no report from an earlier run survives an exit code 2.
The documentation added for #15 describes the current behavior, in the Output section of docs/reference.md and in docs/coding-agents.md. Both need updating with the fix.
Versions
- Cobalto 0.7.0 (
mainat 4dcb631) - GnuCOBOL 3.2
- macOS
- Profile
gnucobol
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 4h 51m
- Merged PRs (30d)
- 9
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from abstracta/cobalto
-
extension help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
compiler help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
bug
Difficulty 3/5 Half a day Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
release
Difficulty 3/5 1-2 days Newbie friendliness 52/100
Maintainers usually reply within 1 day
-
help wanted
Difficulty 4/5 3-5 days Newbie friendliness 50/100
Maintainers usually reply within 1 day
All issues in abstracta/cobalto
Similar issues
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
juandresrodca/AdminForge#66 ·
Maintainers usually reply within 1 day
-
backend/local comp/tools P2 tool/terminal type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NousResearch/hermes-agent#158788 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
public-apis/public-apis#9330 · 1 reaction ·
Maintainers usually reply within 12 days
-
approved author-page correction move-papers
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
acl-org/acl-anthology#11043 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day