Stress test failures are non-blocking and lack triage
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- build-system, ci-cd, testing
Research direction
Start with the Makefile check-stress target and scripts/run-stress.sh, then run the five tests individually on x86_64 and aarch64 as described. Record deterministic, timing-sensitive, and environmental failures, review scripts/lsan-suppressions.txt, and finish by documenting the triage and applying selective blocking, expected-failure, and retry decisions.
Written by the indexing model from the issue text.
Description
Rationale
Five stress tests (fd-exhaust, rapid-fork, concurrent-io, signal-race, long-running) run via make check-stress but failures are swallowed with an informational message. Permanently non-blocking status means regressions go undetected and the tests provide no CI value.
Proposed Changes
- Triage existing failures: run each stress test in isolation on both x86_64 and aarch64, document which tests fail, under what conditions, and whether failures are kbox bugs or environmental noise.
- Selective promotion: promote deterministic tests (fd-exhaust and rapid-fork are candidates) to blocking status. Keep timing-dependent tests (signal-race) as non-blocking with expected-failure annotations.
- Retry after triage: add retry logic (3 attempts) only for tests proven to be timing-sensitive, to avoid masking real instability.
- Revisit LSAN suppressions:
lsan-suppressions.txtalready suppresses LKL semaphore leaks inposix-host.c. Determine if these are upstream LKL issues or kbox integration bugs, and update suppressions as needed.
Considerations
STRESS_TIMEOUTis already configurable via environment variable; consider per-test timeouts or CI-specific defaults rather than a global increase- Stress binaries are built with
-O2 -static(no ASAN/UBSAN), testing different codepaths than unit tests - signal-race is the most likely to be flaky (depends on SIGALRM delivery timing)
- Blocking on all stress tests would slow CI; selective promotion is the pragmatic path
- Triage must come before retries -- adding retries first risks masking real instability
References
tests/stress/: stress test source filesscripts/run-stress.sh: test runner with per-test timeout and LSAN configurationMakefile:check-stresstarget swallows failuresscripts/lsan-suppressions.txt: existing LSAN suppressions
- Dominant language
- C
- Stars
- 95
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from sysprog21/kbox
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Similar issues
-
level/task module/gcp type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
hapostgres/pg_auto_failover#1190 ·
-
docs
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
P3 sonic-vpp
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
sonic-net/sonic-buildimage#29662 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 94/100
spack/spack-packages#6586 ·