perf: live tick 50-160 ms - census starves the host (fixed, undeployed) and the tick's own 40 ms of CPU (profiled 11-13)

Aperta
#3,427 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
12/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go

Direzione di ricerca

The tick cost lives in Ticker.Run, with the monster leg (RunMonsterLeg, RunMonsterAction, advanceInstanceStep), per-session EntryActionSpeed under prepareTick/SnapshotSessions, and characterChanged in the character commit path. The census starvation is already fixed in #3312, so do not start there; read the profile tables and the release plan first. Done means a live capture at 100+ sessions with tick p50 at or under 30 ms and p99 under 50 ms.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: server enhancement priority: high

Part of #2150. Owner target: 10-30 ms per world tick. Today live p50 is 50-60 ms and p99 is 100-160 ms, with spikes of 250-585 ms.

Live capture, 2026-11-13 18:53Z, release a57ea55e6, about 120 sessions

/transport/metrics was diffed over 60 s, and the CPU profile was taken over 30 s on the loopback profiler.

  • Tick: mean 46.6 ms. In the minute: p50 bucket 50-60 ms, p90 80 ms, p99 100-160 ms. 69 overruns of the 100 ms interval in the minute; 16,770 in total.
  • Phase p50 / p99: hooks 25-50 / 100 ms, divisions 16-25 / 50 ms, before_hooks 5 / 16 ms, commit 2 / 10 ms.
  • Host: 4 cores, 2% idle. The GameWorld uses about 1.8 cores. GOGC=200, GOMEMLIMIT=2304MiB, live heap 1.5 GB. SRO_STORE_ASYNC_COMMIT=1 is already on.

Cause 1: the census sampler starves the tick (fixed on main, not deployed)

census.minGap is 41.7% flat (28.4 s of 68.2 s) of the whole process. It runs on the publicstats status sampler (sampleCensus → PublicCounts → buildIndex), not on the tick. It takes about one of the four cores, so the tick waits for a free core, and the 100-585 ms ticks come from that wait. In the 11-12 profile cited by #3302 it was 19.7%; it grows with the ring account's span count.

#3312 (merged 22f9945e8, 11-12 13:47 +0800) removes it. Production runs a57ea55e6 (11-12 00:30 +0800), which predates it. The fix ships with the next release.

Cause 2: the tick's own CPU is about 40 ms per tick

The Ticker.Run stacks are 12.06 s over about 300 ticks. Per tick:

Work ms/tick Top frames
Monster leg (RunMonsterLeg) ~17 RunMonsterAction 3.27 s, advanceAndPublish 2.69 s, advanceInstanceStep 2.17 s
Store reads (Deps.Read → ReadState) ~7 characterSnapshotCast 21%, PopulationPlayers 12%, questMarkerTick 9%, characterSnapshot 8%
Session snapshots (prepareTick → SnapshotSessions) ~5.7 appearanceFor → EntryActionSpeed 0.82 s: every session, every tick
Skill objects (advanceSkillObjectCommits) ~5 character commits
Basic attacks, projectiles ~4.8, ~4.2
Character commits (upsertTimedTx) ~4 characterChanged 0.82 s (section compare), CloneCharacterStaged, encode
Pets, player abnormals ~3, ~2.8

GC: gcBgMarkWorker is 5.8 s (8.5%) of process CPU. On a host with no idle core, that time comes out of the tick's core budget too.

Plan

  1. Release main with #3312. This is the owner's call; the release is held until the owner's localhost test. Re-capture afterwards: the expected p50 is about 40 ms, with the starvation spikes gone.
  2. Cut the tick's own CPU, largest first, one PR each, with a before/after capture:
    • the monster leg;
    • per-session EntryActionSpeed (hand it out from a revision-keyed cache, as captureFor already does);
    • characterChanged's reflective section compare (per-section dirty marks from the doors);
    • repeat character snapshot reads inside one tick.

Done when a live capture at 100+ sessions shows tick p50 at or under 30 ms and p99 under 50 ms.

Lingua principale
Go
Stelle
97
Fork
61
Merge medio
4h 14m
PR unite (30g)
913

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di opensro-dev/opensro

Tutte le issue di opensro-dev/opensro

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.