Validate artifact contracts after an expected nonzero program exit
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
Research direction
The bug is in tools/run_tests.py in the classify_program function. Start by reading the function to see where validate_compiled_artifact is called. The fix is to ensure validation runs before returning for expected-nonzero exits. Run the provided reproduction script to confirm the issue, then write a unit test for the skipped branch. Check tests/cases/shared/test1.wave for an example test case.
Written by the indexing model from the issue text.
Description
Problem
In classify_program in tools/run_tests.py, the expected-nonzero success branch returns before validate_compiled_artifact. A matching exit code can therefore pass a case whose assembly/object requirements fail.
Reproduction
Run this focused unit reproduction from the repository root:
import sys
from dataclasses import replace
from unittest.mock import patch
from tools import run_tests as r
metadata = replace(
r.parse_test_metadata("tests/cases/shared/test1.wave"), expected_exit=42
)
with patch.object(r, "validate_compiled_artifact", return_value="missing assembly marker") as check:
result = r.classify_program(
"audit", "tests/cases/shared/test1.wave",
[sys.executable, "-c", "raise SystemExit(42)"], metadata, None,
)
print(result[0], check.call_count)
Save as /tmp/repro.py; run PYTHONPATH=. python3 /tmp/repro.py. The mock represents a failing artifact requirement.
Tested at 01955d715dda275de993948037091140bb859174 on Fedora Linux 43, x86_64, with Python 3.14.
Actual: PASS (expected exit=42) followed by 3 0, indicating a pass and no validator call. Expected: failure classification 0 and one validator call. Setting both exit values to 0 already produces that expected result.
Completion criteria
- Validate artifact requirements before accepting either expected-zero or expected-nonzero execution.
- Retain the runner's distinct success classifications and actual/expected exit diagnostics.
- Add a unit regression for the skipped branch and an integration case with a real missing assembly marker.
- Keep matching-exit cases with valid artifacts passing.
Compile-only object-presence checking in #514 is separate.
Difficulty: easy.
- Dominant language
- Rust
- Stars
- 53
- Forks
- 16
- Avg merge
- 3h 38m
- Merged PRs (30d)
- 48
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from wavefnd/Wave
-
enhancement good first issue needs testing
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
enhancement good first issue needs testing
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
bug ci good first issue needs testing
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
bug good first issue needs testing
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
bug good first issue os compatibility
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
TheLarkInn/aipm#2413 ·
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 90/100
alexgorbatchev/simple-ptt#15 ·
-
tooling
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
todo:ticket
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
taikoxyz/taiko-mono#22168 · 1 comment ·