policer: adaptive pacing instead of a fixed 10 obj/s default

Đang mở
#4,337 2 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
20/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go

Hướng nghiên cứu

Start with pkg/services/policer/policer.go and process.go to trace the current batch pacing, then read check.go and pkg/services/replicator/replicator.go to understand how replication is dispatched. Review the related issues listed in the proposal before choosing an approach. Done means configurable pacing and in-flight limits, slow PUTs no longer stall scans, pending replicas are retained safely, and the stated throughput and reload criteria are met.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement I3 neofs-storage performance S3 U3

Summary

The default policer rate is 10 objects/s (object_batch_size=10, replication_cooldown=1s ticker), 30/s in boost. That was a deliberate choice (#2617: "10 objects per second is OK..."), and #2590 even proposed making the policer slower to protect load. The problem is not the constant by itself; it is that the rate is fixed and replication runs inside the scan goroutine, so the effective rate is batch / max(cooldown, slowest object).

At 1 PB / REP 3 / 1 MB objects, each node holds ~125M local objects. A full pass is ~145 days at 10/s, ~48 days at 30/s. A lost 16 TB drive takes months to be fully repaired while using <1% of a 10 GbE link. #3762 (boost mode) is closed and helps ~40M-object networks, but 3× on 125M is still ~48 days, and it does not change the trigger.

Current behavior

  • pkg/services/policer/policer.go:158-160: batchSize=10, repCooldown=1s.
  • pkg/services/policer/process.go:88,118: ticker gates every batch; process.go:136: boost multiplies the batch.
  • pkg/services/policer/check.go:436: tryToReplicate calls replicator.HandleTask inline. HandleTask does a full GetBytes + remote PUT with put_timeout=1m (pkg/services/replicator/process.go:23), and the batch waits (process.go:164). One slow PUT stalls the batch.
  • The 8-worker/4096-slot replicator queue (pkg/services/replicator/replicator.go:13-15) is bypassed by the policer; it only serves PUT-time post-placement.
  • No load feedback: boost reacts to missing replicas, not to client or drive load.

Impact

objects/node 10/s 30/s 100/s 1000/s
125M (1 PB, REP3, 1 MB) 145 d 48 d 14.5 d 1.45 d

At 10/s and 1 MB objects, repair bandwidth is 10 MB/s; at 1000/s it is 1 GB/s. A 10 GbE link caps ~1000/s at 1 MB; 25 GbE is needed for a 24 h pass (~1500/s).

Proposal

  1. Replace the ticker with a token bucket + in-flight semaphore. Add objects_per_second and max_inflight; keep replication_cooldown back-compatible (derive rate from batch/cooldown when the new field is unset).
  2. Move replication off the scan goroutine to the existing replicator queue with an in-flight cap. Record "replication pending" and do not drop the local copy until a remote PUT is confirmed. This is the larger lever; pacing alone will not help if a batch stalls on a 1-minute PUT.
  3. Optional second step: feed client in-flight request pressure into the pacer (MinIO-style max_io), so the default is "as fast as the hardware allows without hurting clients" rather than a constant.
  4. Metrics: cycle duration, objects/s, boost state, peer HEAD failure reasons, backlog/ETA.

Non-goals

  • Raising object_batch_size by 25× before (1) and (2): the scan also makes destructive decisions (dropRedundantLocalObject, deleteLocalObject, shard relocation), and a 25× rate multiplies the blast radius. See #4019, #3918, #2178.
  • Fixed erasure sets / protocol changes.

Acceptance criteria

  • A 1 PB / 1 MB-object cluster can complete a full pass in ≤24 h with documented hardware (~1500 obj/s, 25 GbE).
  • A slow replication PUT does not stall the scan batch.
  • No local copy is dropped while a replication for it is pending.
  • Rate/in-flight config reloads via SIGHUP; existing tests pass.
  • Bench or test shows ≥100 obj/s against a mock network with the new pacer.

Related

  • #3762 (closed): boost mode; why 3× is not enough at PB scale.

  • #2590 (closed), #2617 (merged): why 1 s / 10 obj/s was chosen; this issue is about adaptive pacing, not a bigger constant.

  • #4019, #3918, #2178: why raising the rate blindly is dangerous.

  • #2021 (open): replicator metrics.

  • #1764 (open): config defaults/docs.

  • #4336 (open): per-node health state for unreachable holders.

Ngôn ngữ chính
Go
Star
37
Fork
56
Merge trung bình
1 ngày 21 giờ
Pull request đã merge (30 ngày)
46

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của nspcc-dev/neofs-node

Tất cả issue của nspcc-dev/neofs-node

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.