Eval scenario: long-context decode TPS decay — half-speed point as a compaction-threshold anchor
Maintainers usually reply within 1 day
@BiBoyang is already working on this.
Since Oct 8, 2026.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Read the issue’s linked dsh-eval-harness reference, especially eval_tps_run and the README measurement-discipline section, then inspect skill-up’s evaluator and run-configuration entry points. The issue leaves scope, hook shape, cost controls, and regression behavior open, so first establish whether this metric belongs in skill-up before implementation. Done would require an agreed integration shape and a measured long-context performance report; no tests or target files are specified.
Written by the indexing model from the issue text.
Description
Context. Long-context agent sessions pay a bandwidth tax: instantaneous decode TPS falls as context grows, because every decoded token re-reads the entire KV cache from HBM. I measured this in a real coding session — ~200 tok/s at the start, ~80 at 100k context, ~50 at 300k — and wrote up the analysis here (Chinese): Compaction:保真还是速度,这是一个问题. The actionable takeaway: a compaction threshold should be anchored on the model's half-speed point (the context length where instantaneous TPS drops to half of baseline — ≈67k for the GPT-class deployment I measured, far earlier than the folklore 200k), and that point differs per model/architecture. skill-up evaluates behavioral quality of skills and agents; there is currently no axis for performance under long context, which is where real agent runs live.
What I built. A working implementation of this measurement in dsh-eval-harness (eval_tps_run, e90c255): for each model, fire controlled streaming requests at nominal context steps (8k/16k/32k/…), record per-chunk timing plus raw usage, fit tps(n) = 1/(a + b·n), and report baseline speed (1/a), the KV-efficiency slope (b), and the half-speed point (n½). Optional anchor-recall requests yield a fidelity-decay second curve from the same run. The discipline that makes numbers comparable across providers: per-request nonce forcing prefix-cache misses, pinned output side (fixed temperature / max_tokens), medians over repeats, mandatory token budget with dry-run estimate, strictly serial execution. Every report carries an explicit caveat: numbers describe the measured deployment (provider + endpoint + date), not the architecture.
Why it might fit skill-up. Behavioral evals degrade differently when the model is 4× slower at 300k: timeouts, truncated retries, and judge latency all shift. A decay-curve capability would let users (a) pick compaction thresholds from evidence instead of defaults, (b) regression-check provider deployments over time (a silent downgrade shows up as movement in b or n½), and (c) compare architectures (e.g. MLA vs GQA) on the axis that actually gates long-context usability.
Open questions (not a proposal yet):
- Scope. skill-up evaluators judge agent behavior on tasks; this measurement needs raw API-level streaming control with no agent loop. Is a non-behavioral performance metric in scope for skill-up, or better kept as a companion tool whose reports skill-up can reference?
- Hook shape. If in scope: a new evaluator type, or a
metrics:block in the run config producing a sidecar report alongside behavioral results? - Cost control. A full curve is hundreds of thousands of tokens per model. Should scenario configs declare budgets the way CI declares timeouts (refuse to run past them)?
- Cross-time comparability. Provider deployments drift silently. Is there room for baseline curves (golden-file style) with tolerance bands, so a deployment regression fails CI?
Reference implementation and report format: https://github.com/BiBoyang/dsh-eval-harness (see eval_tps_run; the README section lists the measurement discipline). Happy to port the core loop to Go and prototype whichever shape you prefer.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day