perf: live tick 50-160 ms - census starves the host (fixed, undeployed) and the tick's own 40 ms of CPU (profiled 11-13)

Offen
#3,427 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
12/100
Issue-Typ
Refactoring
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
go
Bereich
backend, performance

Rechercherichtung

The tick cost lives in Ticker.Run, with the monster leg (RunMonsterLeg, RunMonsterAction, advanceInstanceStep), per-session EntryActionSpeed under prepareTick/SnapshotSessions, and characterChanged in the character commit path. The census starvation is already fixed in #3312, so do not start there; read the profile tables and the release plan first. Done means a live capture at 100+ sessions with tick p50 at or under 30 ms and p99 under 50 ms.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

area: server enhancement priority: high

Part of #2150. Owner target: 10-30 ms per world tick. Today live p50 is 50-60 ms and p99 is 100-160 ms, with spikes of 250-585 ms.

Live capture, 2026-11-13 18:53Z, release a57ea55e6, about 120 sessions

/transport/metrics was diffed over 60 s, and the CPU profile was taken over 30 s on the loopback profiler.

  • Tick: mean 46.6 ms. In the minute: p50 bucket 50-60 ms, p90 80 ms, p99 100-160 ms. 69 overruns of the 100 ms interval in the minute; 16,770 in total.
  • Phase p50 / p99: hooks 25-50 / 100 ms, divisions 16-25 / 50 ms, before_hooks 5 / 16 ms, commit 2 / 10 ms.
  • Host: 4 cores, 2% idle. The GameWorld uses about 1.8 cores. GOGC=200, GOMEMLIMIT=2304MiB, live heap 1.5 GB. SRO_STORE_ASYNC_COMMIT=1 is already on.

Cause 1: the census sampler starves the tick (fixed on main, not deployed)

census.minGap is 41.7% flat (28.4 s of 68.2 s) of the whole process. It runs on the publicstats status sampler (sampleCensus → PublicCounts → buildIndex), not on the tick. It takes about one of the four cores, so the tick waits for a free core, and the 100-585 ms ticks come from that wait. In the 11-12 profile cited by #3302 it was 19.7%; it grows with the ring account's span count.

#3312 (merged 22f9945e8, 11-12 13:47 +0800) removes it. Production runs a57ea55e6 (11-12 00:30 +0800), which predates it. The fix ships with the next release.

Cause 2: the tick's own CPU is about 40 ms per tick

The Ticker.Run stacks are 12.06 s over about 300 ticks. Per tick:

Work ms/tick Top frames
Monster leg (RunMonsterLeg) ~17 RunMonsterAction 3.27 s, advanceAndPublish 2.69 s, advanceInstanceStep 2.17 s
Store reads (Deps.Read → ReadState) ~7 characterSnapshotCast 21%, PopulationPlayers 12%, questMarkerTick 9%, characterSnapshot 8%
Session snapshots (prepareTick → SnapshotSessions) ~5.7 appearanceFor → EntryActionSpeed 0.82 s: every session, every tick
Skill objects (advanceSkillObjectCommits) ~5 character commits
Basic attacks, projectiles ~4.8, ~4.2
Character commits (upsertTimedTx) ~4 characterChanged 0.82 s (section compare), CloneCharacterStaged, encode
Pets, player abnormals ~3, ~2.8

GC: gcBgMarkWorker is 5.8 s (8.5%) of process CPU. On a host with no idle core, that time comes out of the tick's core budget too.

Plan

  1. Release main with #3312. This is the owner's call; the release is held until the owner's localhost test. Re-capture afterwards: the expected p50 is about 40 ms, with the starvation spikes gone.
  2. Cut the tick's own CPU, largest first, one PR each, with a before/after capture:
    • the monster leg;
    • per-session EntryActionSpeed (hand it out from a revision-keyed cache, as captureFor already does);
    • characterChanged's reflective section compare (per-section dirty marks from the doors);
    • repeat character snapshot reads inside one tick.

Done when a live capture at 100+ sessions shows tick p50 at or under 30 ms and p99 under 50 ms.

Vorherrschende Sprache
Go
Sterne
97
Forks
61
Ø Merge
4 Std. 14 Min.
Gemergte PRs (30 T.)
913

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus opensro-dev/opensro

Alle Issues in opensro-dev/opensro

Ähnliche Issues

Weitere Issues zu Go

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.