P4: measure real model think-time distribution to size the sandbox latency budget

Aperta
  • Qualcosa non va?
#4 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
shell
Ambito
performance

Direzione di ricerca

Start with the E11 section in deploy/microvm/EXPERIMENTS.md and inspect the harness/ for existing real Pi-agent sessions or traces. Decide what workload population and gap definition the measurement can support, then report the distribution and fractions below 15 ms, 60 ms, and 100 ms; done means the evidence and its limitations are clearly documented.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Get an actual distribution of the time a real agent spends between tool calls ("thinking time" — model inference + planning), rather than the vendor-documented "hundreds of ms" folklore the original P4 design leaned on. This is the cheapest item in the epic (#256) and gates the others: it decides whether relaxing the sandbox's warm-hot-path bar from 15 ms to ~60 ms is actually safe, or whether some models/workloads have thinking gaps short enough that 52.74 ms is still user-visible overhead.

Why this matters

  • The original spec's pre-build model assumed replenishment/resume cost would be "hidden" behind model think-time. That arithmetic has never been measured — E11 only ever drove synthetic Execs (true, cat >/dev/null), never a real agent loop.
  • The metal run measured warm hot path at 52.74 ms (STOP against the 15 ms bar). Before deciding a 60 ms bar is fine, we need to know: what fraction of real tool-call gaps are shorter than 60 ms? If a non-trivial fraction of turns have near-zero think time (e.g., a chained sequence of fast tool calls with no model call in between, or a fast/small model), the sandbox overhead is not actually hidden for those turns.
  • This doesn't require touching the microVM stack at all — it's the cheapest possible experiment in this epic and should run first.

What's already known / where to look

  • E11's own driving model is "closed-loop-per-slot" (see deploy/microvm/EXPERIMENTS.md, E11 section, and the drivingModel field in the e11-ladder-*.json records) — it issues the next Exec as soon as the previous one completes, with no think-time gap modeled at all. That's fine for measuring the sandbox's own ceiling, but it means none of the current throughput numbers say anything about a realistic inter-Exec gap.
  • The harness (harness/) already runs real Pi-agent sessions elsewhere in the repo. The P6 density work used a model stub with a declared latency profile (300ms TTFT, 12ms/token) rather than real model latency — also not a real measurement, but a useful precedent for how to calibrate a stub once this issue produces a distribution.
  • Candidate data sources: existing eval/production traces if any are captured; a small live-model harness run instrumented to log time between successive tool-call completions and the next tool-call dispatch; or literature/vendor numbers as a fallback, clearly labeled as unverified.

Open questions for the spec

  1. What population of workloads/models should this represent — one reference model, a range, or whatever the harness is actually deployed against in production?
  2. Is "think time" measured strictly as model-inference latency, or does it include agent-side planning/tool-selection overhead between the model response and the next Exec dispatch?
  3. What's the right statistic to report — p50 is likely to be misleading here; the tail (p10, or fraction below a candidate threshold) is what actually determines whether sandbox overhead is ever visible.
  4. Does this vary meaningfully by tool-call type (e.g., a call right after a fast follow-up vs. a call after a long model turn)?

Suggested scope for a first cut

A read-only measurement task: instrument or query existing traces for inter-tool-call gaps, produce a distribution (not just a mean), and report what fraction falls under candidate bars (15 ms, 60 ms, 100 ms). No new infrastructure should be required for a first pass.

Part of rossoctl/moca-experiments#3 (P4 epic: next experiments after the metal STOP). See PR rossoctl/moca#251 for the metal-run numbers this decision depends on.

Lingua principale
Shell
Stelle
0
Fork
1
Merge medio
6h 16m
PR unite (30g)
7

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di rossoctl/moca-experiments

Tutte le issue di rossoctl/moca-experiments

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.