fix(benchmark): curl/wget tests are non-deterministic (live HTTP requests)

Open Beginner friendly
#2,844 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
shell
Domain
cli, testing-qa

Research direction

Start in scripts/benchmark.sh, especially the command execution around lines 44-45 and the curl/wget benchmark entries around lines 349-350 and 358; compare them with the local fixture patterns in the json and log sections. Run the benchmark before and after the change, and verify the curl/wget cases use stable local data without live HTTP requests or intermittent token-savings failures.

Written by the indexing model from the issue text.

Description

area:ci area:testing bug good first issue priority:high

Problem

The benchmark CI gate (scripts/benchmark.sh) intermittently fails on curl json, curl text, and wget tests with negative token savings (e.g., 35 → 38 (-8%) or 8 → 40 (-400%)).

This blocks unrelated PRs — e.g., #2836 (grep context separator fix) showed a curl json failure despite touching no curl code.

Root Cause

scripts/benchmark.sh runs the raw command and RTK command as separate invocations (lines 44-45):

unix_out=$(eval "$unix_cmd" 2>/dev/null || true)
rtk_out=$(eval "$rtk_cmd" 2>/dev/null || true)

For curl/wget tests (lines 349-350, 358), this means two separate HTTP requests to httpbin.org:

bench "curl json" "curl -s https://httpbin.org/json" "$RTK curl https://httpbin.org/json"
bench "curl text" "curl -s https://httpbin.org/robots.txt" "$RTK curl https://httpbin.org/robots.txt"
bench "wget"      "wget -qO- https://httpbin.org/json"     "$RTK wget https://httpbin.org/json"

Response variations between the two calls (timestamps, headers, content-length, server-side changes) cause token count mismatches. The never-worse guard inside RTK (#2554) works correctly within a single execution, but the benchmark's cross-execution comparison is inherently non-deterministic for live endpoints.

Prior Art

Same class of issue as:

  • #2551 — identified curl emitting more tokens than raw on tiny responses
  • #2554 — added core::guard::never_worse() to solve the within-RTK problem
  • #2751 — parser fix PR that also had this exact benchmark flakiness; commenter noted the curl test is non-deterministic

The never-worse guard (#2554) solved the within-RTK problem. But the benchmark itself still compares outputs from two separate HTTP requests, making it structurally flaky.

Proposed Fix

Replace live httpbin.org calls with local fixtures, following the pattern already used by the json and log benchmark sections (lines 228-246, 286-302):

# Create fixture (matching httpbin.org/json response)
cat > /tmp/rtk_bench_curl.json << 'CURLEOF'
{"slideshow":{"author":"Yours Truly","date":"date of publication","slides":[...]}}
CURLEOF

bench "curl json" "cat /tmp/rtk_bench_curl.json" "$RTK curl file:///tmp/rtk_bench_curl.json"

This makes the test deterministic while still exercising the curl filter logic.

Dominant language
Rust
Stars
81.1k
Forks
5.1k
Avg merge
4d 11h
Merged PRs (30d)
42

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from rtk-ai/rtk

All issues in rtk-ai/rtk

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.