policer: adaptive pacing instead of a fixed 10 obj/s default
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 20/100
- Issue 类型
- 功能
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- go
调研方向
Start with pkg/services/policer/policer.go and process.go to trace the current batch pacing, then read check.go and pkg/services/replicator/replicator.go to understand how replication is dispatched. Review the related issues listed in the proposal before choosing an approach. Done means configurable pacing and in-flight limits, slow PUTs no longer stall scans, pending replicas are retained safely, and the stated throughput and reload criteria are met.
由索引模型根据 Issue 内容生成。
描述
Summary
The default policer rate is 10 objects/s (object_batch_size=10, replication_cooldown=1s ticker), 30/s in boost. That was a deliberate choice (#2617: "10 objects per second is OK..."), and #2590 even proposed making the policer slower to protect load. The problem is not the constant by itself; it is that the rate is fixed and replication runs inside the scan goroutine, so the effective rate is batch / max(cooldown, slowest object).
At 1 PB / REP 3 / 1 MB objects, each node holds ~125M local objects. A full pass is ~145 days at 10/s, ~48 days at 30/s. A lost 16 TB drive takes months to be fully repaired while using <1% of a 10 GbE link. #3762 (boost mode) is closed and helps ~40M-object networks, but 3× on 125M is still ~48 days, and it does not change the trigger.
Current behavior
pkg/services/policer/policer.go:158-160:batchSize=10,repCooldown=1s.pkg/services/policer/process.go:88,118: ticker gates every batch;process.go:136: boost multiplies the batch.pkg/services/policer/check.go:436:tryToReplicatecallsreplicator.HandleTaskinline.HandleTaskdoes a fullGetBytes+ remote PUT withput_timeout=1m(pkg/services/replicator/process.go:23), and the batch waits (process.go:164). One slow PUT stalls the batch.- The 8-worker/4096-slot replicator queue (
pkg/services/replicator/replicator.go:13-15) is bypassed by the policer; it only serves PUT-time post-placement. - No load feedback: boost reacts to missing replicas, not to client or drive load.
Impact
| objects/node | 10/s | 30/s | 100/s | 1000/s |
|---|---|---|---|---|
| 125M (1 PB, REP3, 1 MB) | 145 d | 48 d | 14.5 d | 1.45 d |
At 10/s and 1 MB objects, repair bandwidth is 10 MB/s; at 1000/s it is 1 GB/s. A 10 GbE link caps ~1000/s at 1 MB; 25 GbE is needed for a 24 h pass (~1500/s).
Proposal
- Replace the ticker with a token bucket + in-flight semaphore. Add
objects_per_secondandmax_inflight; keepreplication_cooldownback-compatible (derive rate from batch/cooldown when the new field is unset). - Move replication off the scan goroutine to the existing replicator queue with an in-flight cap. Record "replication pending" and do not drop the local copy until a remote PUT is confirmed. This is the larger lever; pacing alone will not help if a batch stalls on a 1-minute PUT.
- Optional second step: feed client in-flight request pressure into the pacer (MinIO-style
max_io), so the default is "as fast as the hardware allows without hurting clients" rather than a constant. - Metrics: cycle duration, objects/s, boost state, peer HEAD failure reasons, backlog/ETA.
Non-goals
- Raising
object_batch_sizeby 25× before (1) and (2): the scan also makes destructive decisions (dropRedundantLocalObject,deleteLocalObject, shard relocation), and a 25× rate multiplies the blast radius. See #4019, #3918, #2178. - Fixed erasure sets / protocol changes.
Acceptance criteria
- A 1 PB / 1 MB-object cluster can complete a full pass in ≤24 h with documented hardware (~1500 obj/s, 25 GbE).
- A slow replication PUT does not stall the scan batch.
- No local copy is dropped while a replication for it is pending.
- Rate/in-flight config reloads via SIGHUP; existing tests pass.
- Bench or test shows ≥100 obj/s against a mock network with the new pacer.
Related
-
#3762 (closed): boost mode; why 3× is not enough at PB scale.
-
#2590 (closed), #2617 (merged): why 1 s / 10 obj/s was chosen; this issue is about adaptive pacing, not a bigger constant.
-
#4019, #3918, #2178: why raising the rate blindly is dangerous.
-
#2021 (open): replicator metrics.
-
#1764 (open): config defaults/docs.
-
#4336 (open): per-node health state for unreachable holders.
- 主要语言
- Go
- 星标
- 37
- 派生
- 56
- 平均合并
- 1 天 21 小时
- 30 天内合并 PR
- 46
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
nspcc-dev/neofs-node 的其他 Issue
-
bug hal:proposed neofs-storage
难度 2/5 1-3 小时 新手友好度 78/100
nspcc-dev/neofs-node#4325 · 3 条评论 ·
维护者通常 1 天内回复
-
hal:proposed I4 neofs-storage S3 security U4
难度 2/5 1-3 小时 新手友好度 69/100
nspcc-dev/neofs-node#4280 · 2 条评论 ·
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 45/100
nspcc-dev/neofs-node#4339 ·
维护者通常 1 天内回复
-
bug neofs-storage
难度 4/5 3-5 天 新手友好度 45/100
nspcc-dev/neofs-node#4336 ·
维护者通常 1 天内回复
-
bug neofs-storage
难度 3/5 1-2 天 新手友好度 68/100
nspcc-dev/neofs-node#4333 ·
维护者通常 1 天内回复
查看 nspcc-dev/neofs-node 的全部 Issue
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 86/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 64/100
projectdiscovery/cdncheck#554 ·
维护者通常 1 天内回复
-
triage needed
难度 2/5 1-3 小时 新手友好度 66/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 78/100
OpenSlides/openslides-go#342 ·
维护者通常 4 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 77/100