Measure the hard context budget on real models (bench: --context-tokens small vs default)

Open
  • Is something wrong?
#137 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
machine-learning, testing

Research direction

Start with bench/run.py to see how it forwards --param, then trace how bench/ arguments reach tether run and where sessions.peak_tokens and scores are recorded. Run the specified workloads and models at each token budget for two trials, then write the results in docs/benchmarks.md; completion means every non-overflow cell meets the budget limit and scores are within 0.1 of the 8192 run, or has a linked issue.

Written by the indexing model from the issue text.

Description

#81 was verified with scripted e2e scenarios and unit tests; it was never measured on a real model with a small --context-tokens (the original report was GLM-5.3 at 16384 reaching ~24k). Need the evidence that the fold/drop/shrink order costs little quality.

Path

Add bench/ flags --context-tokens and --fold-after passed through to tether run (bench/run.py already forwards --param), run focus, code-medium and qa on glm-5.3, nemotron-3.5-lightning and qwen3.5:9b at 2048, 4096 and 8192, 2 trials each, and record peak tokens from sessions.peak_tokens against the budget plus the score. Write the table into docs/benchmarks.md.

Acceptance

peak_tokens <= budget * 1.15 for every cell where the stderr overflow warning did not fire; scores within 0.1 of the 8192 run, or an issue per cell that is not. No product code change expected; if a cell breaks the budget, file the bug with the session id.

Dominant language
Python
Stars
3
Forks
1
Avg merge
52m
Merged PRs (30d)
166

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from igormcsouza/tether

All issues in igormcsouza/tether

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.