perf: live tick 50-160 ms - census starves the host (fixed, undeployed) and the tick's own 40 ms of CPU (profiled 11-13)

Đang mở
#3,427 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
12/100
Loại issue
Tái cấu trúc
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go
Lĩnh vực
backend, performance

Hướng nghiên cứu

The tick cost lives in Ticker.Run, with the monster leg (RunMonsterLeg, RunMonsterAction, advanceInstanceStep), per-session EntryActionSpeed under prepareTick/SnapshotSessions, and characterChanged in the character commit path. The census starvation is already fixed in #3312, so do not start there; read the profile tables and the release plan first. Done means a live capture at 100+ sessions with tick p50 at or under 30 ms and p99 under 50 ms.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

area: server enhancement priority: high

Part of #2150. Owner target: 10-30 ms per world tick. Today live p50 is 50-60 ms and p99 is 100-160 ms, with spikes of 250-585 ms.

Live capture, 2026-11-13 18:53Z, release a57ea55e6, about 120 sessions

/transport/metrics was diffed over 60 s, and the CPU profile was taken over 30 s on the loopback profiler.

  • Tick: mean 46.6 ms. In the minute: p50 bucket 50-60 ms, p90 80 ms, p99 100-160 ms. 69 overruns of the 100 ms interval in the minute; 16,770 in total.
  • Phase p50 / p99: hooks 25-50 / 100 ms, divisions 16-25 / 50 ms, before_hooks 5 / 16 ms, commit 2 / 10 ms.
  • Host: 4 cores, 2% idle. The GameWorld uses about 1.8 cores. GOGC=200, GOMEMLIMIT=2304MiB, live heap 1.5 GB. SRO_STORE_ASYNC_COMMIT=1 is already on.

Cause 1: the census sampler starves the tick (fixed on main, not deployed)

census.minGap is 41.7% flat (28.4 s of 68.2 s) of the whole process. It runs on the publicstats status sampler (sampleCensus → PublicCounts → buildIndex), not on the tick. It takes about one of the four cores, so the tick waits for a free core, and the 100-585 ms ticks come from that wait. In the 11-12 profile cited by #3302 it was 19.7%; it grows with the ring account's span count.

#3312 (merged 22f9945e8, 11-12 13:47 +0800) removes it. Production runs a57ea55e6 (11-12 00:30 +0800), which predates it. The fix ships with the next release.

Cause 2: the tick's own CPU is about 40 ms per tick

The Ticker.Run stacks are 12.06 s over about 300 ticks. Per tick:

Work ms/tick Top frames
Monster leg (RunMonsterLeg) ~17 RunMonsterAction 3.27 s, advanceAndPublish 2.69 s, advanceInstanceStep 2.17 s
Store reads (Deps.Read → ReadState) ~7 characterSnapshotCast 21%, PopulationPlayers 12%, questMarkerTick 9%, characterSnapshot 8%
Session snapshots (prepareTick → SnapshotSessions) ~5.7 appearanceFor → EntryActionSpeed 0.82 s: every session, every tick
Skill objects (advanceSkillObjectCommits) ~5 character commits
Basic attacks, projectiles ~4.8, ~4.2
Character commits (upsertTimedTx) ~4 characterChanged 0.82 s (section compare), CloneCharacterStaged, encode
Pets, player abnormals ~3, ~2.8

GC: gcBgMarkWorker is 5.8 s (8.5%) of process CPU. On a host with no idle core, that time comes out of the tick's core budget too.

Plan

  1. Release main with #3312. This is the owner's call; the release is held until the owner's localhost test. Re-capture afterwards: the expected p50 is about 40 ms, with the starvation spikes gone.
  2. Cut the tick's own CPU, largest first, one PR each, with a before/after capture:
    • the monster leg;
    • per-session EntryActionSpeed (hand it out from a revision-keyed cache, as captureFor already does);
    • characterChanged's reflective section compare (per-section dirty marks from the doors);
    • repeat character snapshot reads inside one tick.

Done when a live capture at 100+ sessions shows tick p50 at or under 30 ms and p99 under 50 ms.

Ngôn ngữ chính
Go
Star
97
Fork
61
Merge trung bình
4 giờ 14 phút
Pull request đã merge (30 ngày)
913

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của opensro-dev/opensro

Tất cả issue của opensro-dev/opensro

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.