Eval scenario: long-context decode TPS decay — half-speed point as a compaction-threshold anchor

Open
#296 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@BiBoyang is already working on this.

Since Oct 8, 2026.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
go
Domain
ai

Research direction

Read the issue’s linked dsh-eval-harness reference, especially eval_tps_run and the README measurement-discipline section, then inspect skill-up’s evaluator and run-configuration entry points. The issue leaves scope, hook shape, cost controls, and regression behavior open, so first establish whether this metric belongs in skill-up before implementation. Done would require an agreed integration shape and a measured long-context performance report; no tests or target files are specified.

Written by the indexing model from the issue text.

Description

question

Context. Long-context agent sessions pay a bandwidth tax: instantaneous decode TPS falls as context grows, because every decoded token re-reads the entire KV cache from HBM. I measured this in a real coding session — ~200 tok/s at the start, ~80 at 100k context, ~50 at 300k — and wrote up the analysis here (Chinese): Compaction:保真还是速度,这是一个问题. The actionable takeaway: a compaction threshold should be anchored on the model's half-speed point (the context length where instantaneous TPS drops to half of baseline — ≈67k for the GPT-class deployment I measured, far earlier than the folklore 200k), and that point differs per model/architecture. skill-up evaluates behavioral quality of skills and agents; there is currently no axis for performance under long context, which is where real agent runs live.

What I built. A working implementation of this measurement in dsh-eval-harness (eval_tps_run, e90c255): for each model, fire controlled streaming requests at nominal context steps (8k/16k/32k/…), record per-chunk timing plus raw usage, fit tps(n) = 1/(a + b·n), and report baseline speed (1/a), the KV-efficiency slope (b), and the half-speed point (n½). Optional anchor-recall requests yield a fidelity-decay second curve from the same run. The discipline that makes numbers comparable across providers: per-request nonce forcing prefix-cache misses, pinned output side (fixed temperature / max_tokens), medians over repeats, mandatory token budget with dry-run estimate, strictly serial execution. Every report carries an explicit caveat: numbers describe the measured deployment (provider + endpoint + date), not the architecture.

Why it might fit skill-up. Behavioral evals degrade differently when the model is 4× slower at 300k: timeouts, truncated retries, and judge latency all shift. A decay-curve capability would let users (a) pick compaction thresholds from evidence instead of defaults, (b) regression-check provider deployments over time (a silent downgrade shows up as movement in b or n½), and (c) compare architectures (e.g. MLA vs GQA) on the axis that actually gates long-context usability.

Open questions (not a proposal yet):

  1. Scope. skill-up evaluators judge agent behavior on tasks; this measurement needs raw API-level streaming control with no agent loop. Is a non-behavioral performance metric in scope for skill-up, or better kept as a companion tool whose reports skill-up can reference?
  2. Hook shape. If in scope: a new evaluator type, or a metrics: block in the run config producing a sidecar report alongside behavioral results?
  3. Cost control. A full curve is hundreds of thousands of tokens per model. Should scenario configs declare budgets the way CI declares timeouts (refuse to run past them)?
  4. Cross-time comparability. Provider deployments drift silently. Is there room for baseline curves (golden-file style) with tolerance bands, so a deployment regression fails CI?

Reference implementation and report format: https://github.com/BiBoyang/dsh-eval-harness (see eval_tps_run; the README section lists the measurement discipline). Happy to port the core loop to Go and prototype whichever shape you prefer.

Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.