fix(benchmark): curl/wget tests are non-deterministic (live HTTP requests)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- shell
- Domain
- cli, testing-qa
Research direction
Start in scripts/benchmark.sh, especially the command execution around lines 44-45 and the curl/wget benchmark entries around lines 349-350 and 358; compare them with the local fixture patterns in the json and log sections. Run the benchmark before and after the change, and verify the curl/wget cases use stable local data without live HTTP requests or intermittent token-savings failures.
Written by the indexing model from the issue text.
Description
Problem
The benchmark CI gate (scripts/benchmark.sh) intermittently fails on curl json, curl text, and wget tests with negative token savings (e.g., 35 → 38 (-8%) or 8 → 40 (-400%)).
This blocks unrelated PRs — e.g., #2836 (grep context separator fix) showed a curl json failure despite touching no curl code.
Root Cause
scripts/benchmark.sh runs the raw command and RTK command as separate invocations (lines 44-45):
unix_out=$(eval "$unix_cmd" 2>/dev/null || true)
rtk_out=$(eval "$rtk_cmd" 2>/dev/null || true)
For curl/wget tests (lines 349-350, 358), this means two separate HTTP requests to httpbin.org:
bench "curl json" "curl -s https://httpbin.org/json" "$RTK curl https://httpbin.org/json"
bench "curl text" "curl -s https://httpbin.org/robots.txt" "$RTK curl https://httpbin.org/robots.txt"
bench "wget" "wget -qO- https://httpbin.org/json" "$RTK wget https://httpbin.org/json"
Response variations between the two calls (timestamps, headers, content-length, server-side changes) cause token count mismatches. The never-worse guard inside RTK (#2554) works correctly within a single execution, but the benchmark's cross-execution comparison is inherently non-deterministic for live endpoints.
Prior Art
Same class of issue as:
- #2551 — identified
curlemitting more tokens than raw on tiny responses - #2554 — added
core::guard::never_worse()to solve the within-RTK problem - #2751 — parser fix PR that also had this exact benchmark flakiness; commenter noted the curl test is non-deterministic
The never-worse guard (#2554) solved the within-RTK problem. But the benchmark itself still compares outputs from two separate HTTP requests, making it structurally flaky.
Proposed Fix
Replace live httpbin.org calls with local fixtures, following the pattern already used by the json and log benchmark sections (lines 228-246, 286-302):
# Create fixture (matching httpbin.org/json response)
cat > /tmp/rtk_bench_curl.json << 'CURLEOF'
{"slideshow":{"author":"Yours Truly","date":"date of publication","slides":[...]}}
CURLEOF
bench "curl json" "cat /tmp/rtk_bench_curl.json" "$RTK curl file:///tmp/rtk_bench_curl.json"
This makes the test deterministic while still exercising the curl filter logic.
- Dominant language
- Rust
- Stars
- 81.1k
- Forks
- 5.1k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 42
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from rtk-ai/rtk
-
area:cli bug filter-quality good first issue priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
bug core output-formatting
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
uv_cmd discards print_with_hint's return value, so the tee hint is not counted and savings read 100% Openanalytics bug python
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
area:cli bug good first issue platform:windows priority:medium resolved-pending-close
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
area:cli bug good first issue priority:high
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
gitbutlerapp/gitbutler#15998 · 1 comment ·
-
bug triage:deciding
Difficulty 1/5 Under an hour Newbie friendliness 88/100
open-telemetry/otel-arrow#4132 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100