E9: run the deployment-tier comparison — VM supervisor+mux vs Knative one-session-per-pod

Aperta
  • Qualcosa non va?
#12 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
kubernetes, shell
Ambito
infrastructure, performance

Direzione di ricerca

First verify the listed prerequisites, especially the E8 run and required merges; then read P6 §5.3 and the E8 run records for the shared rungs, stub, and knee-detection. Use relay-leaf-smoke.sh as the transport reference and inspect deploy/knative/model-stub/ and the E6 driver’s ksvc environment handling. Done means both arms use the same stub and relay + gRPC transport, record matching connections per session and per-arm knees, and publish results in deploy/vm/EXPERIMENTS.md with a pointer from docs/experiment-results.md.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Run E9 — deployment-tier comparison (P6 §5.3): the VM supervisor+mux arm against the Knative
one-session-per-pod arm, same host class and same workload, to isolate what the deployment tier
costs. No issue tracks this today; the epic lists it as outstanding and it is the only P6 experiment
with neither a spec-amendment issue nor a driver issue.

E9 compares capacity, not cost. E1-style economics on the VM is round two (§7).

The two pins — and the second one is easy to get wrong

Isolating one tier means pinning every other. E9 has two such constraints:

1. The model tier. E6's existing numbers were taken against a real model and are not comparable
to stub-driven ones. The Knative arm must be re-run against the same stub — cheap, via
ANTHROPIC_BASE_URL on the ksvc, and E6's driver already manipulates and restores ksvc env
(restore_ksvc_env on its EXIT trap).

2. The tool tier. Left to their defaults the arms do not match: the Knative arm resolves pods
and gets persistentExecInPod's fast channel, while the VM arm runs over gRPC and has none (§3.1a).
The comparison would then charge the deployment tier for a transport difference, biased against the
VM. So both arms run the relay + gRPC transport — SH_REMOTE_SANDBOX=1 against the in-cluster
relay, which relay-leaf-smoke.sh already exercises. Configuration, not code.

The invariant a reviewer should check: E9 varies the deployment tier and nothing else. Any future arm
inherits both pins.

Preconditions

# Gate Why
1 E8 run first E9 reuses E8's rungs, stub and knee-detection; comparing two tiers before either arm's own knee is understood produces a ratio nobody can attribute
2 rossoctl/moca-experiments#10 merged Same two-arm methodology and the same reporting discipline apply here
3 PR rossoctl/moca#253 merged Both arms need /turn tool calls actually reaching the sandbox pool
4 Model stub deployable to the cluster It lives at deploy/knative/model-stub/ — not under deploy/vm/ — precisely because both arms must drive the same stub (§5.4). The VM arm runs the same image locally
5 Same duty basis row on both arms §2.3, named in each run record

Step 7 of P6 §3's build order is exactly this: "E9 second arm on a cluster — both arms against the
same stub and the same gRPC transport."

Acceptance

  • Both arms run on the same host class, same workload, same duty-basis row
  • Both arms recorded as running the relay + gRPC transport; neither arm on persistentExecInPod
  • Both arms against the same stub, with the profile recorded and pinned per arm
  • Connections per session recorded on every rung of both arms and required to match across
    arms     (§3.4) — a mismatch charges one policy for the other's connection budget
  • The reported figure is a deployment-tier comparison, with the model and tool tiers shown pinned
  • Per-arm knee attribution, not just a ratio
  • No cost/resource-seconds claim (out of scope per §7)
  • Results in deploy/vm/EXPERIMENTS.md, pointer from docs/experiment-results.md

Note on P5

Like E8, E9 is measurable single-subject (§3.6). Mixed-subject rungs and the isolation claim need
rossoctl/moca#239; the comparison itself does not. State which was run.

Related: E8 run, rossoctl/moca-experiments#10, PR rossoctl/moca#253, rossoctl/moca#240 (ScaledJob → elastic pod pool, which changes what the Knative
arm is — if rossoctl/moca#240 lands first, record which deployment model the cluster arm measured).

Lingua principale
Shell
Stelle
0
Fork
1
Merge medio
6h 16m
PR unite (30g)
7

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di rossoctl/moca-experiments

Tutte le issue di rossoctl/moca-experiments

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.