perf: live tick 50-160 ms - census starves the host (fixed, undeployed) and the tick's own 40 ms of CPU (profiled 11-13)

未关闭
#3,427 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
12/100
Issue 类型
重构
描述清晰度
基本清楚
活跃度
活跃
技术栈
go

调研方向

The tick cost lives in Ticker.Run, with the monster leg (RunMonsterLeg, RunMonsterAction, advanceInstanceStep), per-session EntryActionSpeed under prepareTick/SnapshotSessions, and characterChanged in the character commit path. The census starvation is already fixed in #3312, so do not start there; read the profile tables and the release plan first. Done means a live capture at 100+ sessions with tick p50 at or under 30 ms and p99 under 50 ms.

由索引模型根据 Issue 内容生成。

描述

area: server enhancement priority: high

Part of #2150. Owner target: 10-30 ms per world tick. Today live p50 is 50-60 ms and p99 is 100-160 ms, with spikes of 250-585 ms.

Live capture, 2026-11-13 18:53Z, release a57ea55e6, about 120 sessions

/transport/metrics was diffed over 60 s, and the CPU profile was taken over 30 s on the loopback profiler.

  • Tick: mean 46.6 ms. In the minute: p50 bucket 50-60 ms, p90 80 ms, p99 100-160 ms. 69 overruns of the 100 ms interval in the minute; 16,770 in total.
  • Phase p50 / p99: hooks 25-50 / 100 ms, divisions 16-25 / 50 ms, before_hooks 5 / 16 ms, commit 2 / 10 ms.
  • Host: 4 cores, 2% idle. The GameWorld uses about 1.8 cores. GOGC=200, GOMEMLIMIT=2304MiB, live heap 1.5 GB. SRO_STORE_ASYNC_COMMIT=1 is already on.

Cause 1: the census sampler starves the tick (fixed on main, not deployed)

census.minGap is 41.7% flat (28.4 s of 68.2 s) of the whole process. It runs on the publicstats status sampler (sampleCensus → PublicCounts → buildIndex), not on the tick. It takes about one of the four cores, so the tick waits for a free core, and the 100-585 ms ticks come from that wait. In the 11-12 profile cited by #3302 it was 19.7%; it grows with the ring account's span count.

#3312 (merged 22f9945e8, 11-12 13:47 +0800) removes it. Production runs a57ea55e6 (11-12 00:30 +0800), which predates it. The fix ships with the next release.

Cause 2: the tick's own CPU is about 40 ms per tick

The Ticker.Run stacks are 12.06 s over about 300 ticks. Per tick:

Work ms/tick Top frames
Monster leg (RunMonsterLeg) ~17 RunMonsterAction 3.27 s, advanceAndPublish 2.69 s, advanceInstanceStep 2.17 s
Store reads (Deps.Read → ReadState) ~7 characterSnapshotCast 21%, PopulationPlayers 12%, questMarkerTick 9%, characterSnapshot 8%
Session snapshots (prepareTick → SnapshotSessions) ~5.7 appearanceFor → EntryActionSpeed 0.82 s: every session, every tick
Skill objects (advanceSkillObjectCommits) ~5 character commits
Basic attacks, projectiles ~4.8, ~4.2
Character commits (upsertTimedTx) ~4 characterChanged 0.82 s (section compare), CloneCharacterStaged, encode
Pets, player abnormals ~3, ~2.8

GC: gcBgMarkWorker is 5.8 s (8.5%) of process CPU. On a host with no idle core, that time comes out of the tick's core budget too.

Plan

  1. Release main with #3312. This is the owner's call; the release is held until the owner's localhost test. Re-capture afterwards: the expected p50 is about 40 ms, with the starvation spikes gone.
  2. Cut the tick's own CPU, largest first, one PR each, with a before/after capture:
    • the monster leg;
    • per-session EntryActionSpeed (hand it out from a revision-keyed cache, as captureFor already does);
    • characterChanged's reflective section compare (per-section dirty marks from the doors);
    • repeat character snapshot reads inside one tick.

Done when a live capture at 100+ sessions shows tick p50 at or under 30 ms and p99 under 50 ms.

主要语言
Go
星标
97
派生
61
平均合并
4 小时 14 分钟
30 天内合并 PR
913

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

opensro-dev/opensro 的其他 Issue

查看 opensro-dev/opensro 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。