policer: adaptive pacing instead of a fixed 10 obj/s default

未关闭
#4,337 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
20/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
go

调研方向

Start with pkg/services/policer/policer.go and process.go to trace the current batch pacing, then read check.go and pkg/services/replicator/replicator.go to understand how replication is dispatched. Review the related issues listed in the proposal before choosing an approach. Done means configurable pacing and in-flight limits, slow PUTs no longer stall scans, pending replicas are retained safely, and the stated throughput and reload criteria are met.

由索引模型根据 Issue 内容生成。

描述

enhancement I3 neofs-storage performance S3 U3

Summary

The default policer rate is 10 objects/s (object_batch_size=10, replication_cooldown=1s ticker), 30/s in boost. That was a deliberate choice (#2617: "10 objects per second is OK..."), and #2590 even proposed making the policer slower to protect load. The problem is not the constant by itself; it is that the rate is fixed and replication runs inside the scan goroutine, so the effective rate is batch / max(cooldown, slowest object).

At 1 PB / REP 3 / 1 MB objects, each node holds ~125M local objects. A full pass is ~145 days at 10/s, ~48 days at 30/s. A lost 16 TB drive takes months to be fully repaired while using <1% of a 10 GbE link. #3762 (boost mode) is closed and helps ~40M-object networks, but 3× on 125M is still ~48 days, and it does not change the trigger.

Current behavior

  • pkg/services/policer/policer.go:158-160: batchSize=10, repCooldown=1s.
  • pkg/services/policer/process.go:88,118: ticker gates every batch; process.go:136: boost multiplies the batch.
  • pkg/services/policer/check.go:436: tryToReplicate calls replicator.HandleTask inline. HandleTask does a full GetBytes + remote PUT with put_timeout=1m (pkg/services/replicator/process.go:23), and the batch waits (process.go:164). One slow PUT stalls the batch.
  • The 8-worker/4096-slot replicator queue (pkg/services/replicator/replicator.go:13-15) is bypassed by the policer; it only serves PUT-time post-placement.
  • No load feedback: boost reacts to missing replicas, not to client or drive load.

Impact

objects/node 10/s 30/s 100/s 1000/s
125M (1 PB, REP3, 1 MB) 145 d 48 d 14.5 d 1.45 d

At 10/s and 1 MB objects, repair bandwidth is 10 MB/s; at 1000/s it is 1 GB/s. A 10 GbE link caps ~1000/s at 1 MB; 25 GbE is needed for a 24 h pass (~1500/s).

Proposal

  1. Replace the ticker with a token bucket + in-flight semaphore. Add objects_per_second and max_inflight; keep replication_cooldown back-compatible (derive rate from batch/cooldown when the new field is unset).
  2. Move replication off the scan goroutine to the existing replicator queue with an in-flight cap. Record "replication pending" and do not drop the local copy until a remote PUT is confirmed. This is the larger lever; pacing alone will not help if a batch stalls on a 1-minute PUT.
  3. Optional second step: feed client in-flight request pressure into the pacer (MinIO-style max_io), so the default is "as fast as the hardware allows without hurting clients" rather than a constant.
  4. Metrics: cycle duration, objects/s, boost state, peer HEAD failure reasons, backlog/ETA.

Non-goals

  • Raising object_batch_size by 25× before (1) and (2): the scan also makes destructive decisions (dropRedundantLocalObject, deleteLocalObject, shard relocation), and a 25× rate multiplies the blast radius. See #4019, #3918, #2178.
  • Fixed erasure sets / protocol changes.

Acceptance criteria

  • A 1 PB / 1 MB-object cluster can complete a full pass in ≤24 h with documented hardware (~1500 obj/s, 25 GbE).
  • A slow replication PUT does not stall the scan batch.
  • No local copy is dropped while a replication for it is pending.
  • Rate/in-flight config reloads via SIGHUP; existing tests pass.
  • Bench or test shows ≥100 obj/s against a mock network with the new pacer.

Related

  • #3762 (closed): boost mode; why 3× is not enough at PB scale.

  • #2590 (closed), #2617 (merged): why 1 s / 10 obj/s was chosen; this issue is about adaptive pacing, not a bigger constant.

  • #4019, #3918, #2178: why raising the rate blindly is dangerous.

  • #2021 (open): replicator metrics.

  • #1764 (open): config defaults/docs.

  • #4336 (open): per-node health state for unreachable holders.

主要语言
Go
星标
37
派生
56
平均合并
1 天 21 小时
30 天内合并 PR
46

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

nspcc-dev/neofs-node 的其他 Issue

查看 nspcc-dev/neofs-node 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。