policer: adaptive pacing instead of a fixed 10 obj/s default

Aperta
#4,337 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go

Direzione di ricerca

Start with pkg/services/policer/policer.go and process.go to trace the current batch pacing, then read check.go and pkg/services/replicator/replicator.go to understand how replication is dispatched. Review the related issues listed in the proposal before choosing an approach. Done means configurable pacing and in-flight limits, slow PUTs no longer stall scans, pending replicas are retained safely, and the stated throughput and reload criteria are met.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement I3 neofs-storage performance S3 U3

Summary

The default policer rate is 10 objects/s (object_batch_size=10, replication_cooldown=1s ticker), 30/s in boost. That was a deliberate choice (#2617: "10 objects per second is OK..."), and #2590 even proposed making the policer slower to protect load. The problem is not the constant by itself; it is that the rate is fixed and replication runs inside the scan goroutine, so the effective rate is batch / max(cooldown, slowest object).

At 1 PB / REP 3 / 1 MB objects, each node holds ~125M local objects. A full pass is ~145 days at 10/s, ~48 days at 30/s. A lost 16 TB drive takes months to be fully repaired while using <1% of a 10 GbE link. #3762 (boost mode) is closed and helps ~40M-object networks, but 3× on 125M is still ~48 days, and it does not change the trigger.

Current behavior

  • pkg/services/policer/policer.go:158-160: batchSize=10, repCooldown=1s.
  • pkg/services/policer/process.go:88,118: ticker gates every batch; process.go:136: boost multiplies the batch.
  • pkg/services/policer/check.go:436: tryToReplicate calls replicator.HandleTask inline. HandleTask does a full GetBytes + remote PUT with put_timeout=1m (pkg/services/replicator/process.go:23), and the batch waits (process.go:164). One slow PUT stalls the batch.
  • The 8-worker/4096-slot replicator queue (pkg/services/replicator/replicator.go:13-15) is bypassed by the policer; it only serves PUT-time post-placement.
  • No load feedback: boost reacts to missing replicas, not to client or drive load.

Impact

objects/node 10/s 30/s 100/s 1000/s
125M (1 PB, REP3, 1 MB) 145 d 48 d 14.5 d 1.45 d

At 10/s and 1 MB objects, repair bandwidth is 10 MB/s; at 1000/s it is 1 GB/s. A 10 GbE link caps ~1000/s at 1 MB; 25 GbE is needed for a 24 h pass (~1500/s).

Proposal

  1. Replace the ticker with a token bucket + in-flight semaphore. Add objects_per_second and max_inflight; keep replication_cooldown back-compatible (derive rate from batch/cooldown when the new field is unset).
  2. Move replication off the scan goroutine to the existing replicator queue with an in-flight cap. Record "replication pending" and do not drop the local copy until a remote PUT is confirmed. This is the larger lever; pacing alone will not help if a batch stalls on a 1-minute PUT.
  3. Optional second step: feed client in-flight request pressure into the pacer (MinIO-style max_io), so the default is "as fast as the hardware allows without hurting clients" rather than a constant.
  4. Metrics: cycle duration, objects/s, boost state, peer HEAD failure reasons, backlog/ETA.

Non-goals

  • Raising object_batch_size by 25× before (1) and (2): the scan also makes destructive decisions (dropRedundantLocalObject, deleteLocalObject, shard relocation), and a 25× rate multiplies the blast radius. See #4019, #3918, #2178.
  • Fixed erasure sets / protocol changes.

Acceptance criteria

  • A 1 PB / 1 MB-object cluster can complete a full pass in ≤24 h with documented hardware (~1500 obj/s, 25 GbE).
  • A slow replication PUT does not stall the scan batch.
  • No local copy is dropped while a replication for it is pending.
  • Rate/in-flight config reloads via SIGHUP; existing tests pass.
  • Bench or test shows ≥100 obj/s against a mock network with the new pacer.

Related

  • #3762 (closed): boost mode; why 3× is not enough at PB scale.

  • #2590 (closed), #2617 (merged): why 1 s / 10 obj/s was chosen; this issue is about adaptive pacing, not a bigger constant.

  • #4019, #3918, #2178: why raising the rate blindly is dangerous.

  • #2021 (open): replicator metrics.

  • #1764 (open): config defaults/docs.

  • #4336 (open): per-node health state for unreachable holders.

Lingua principale
Go
Stelle
37
Fork
56
Merge medio
1g 21h
PR unite (30g)
46

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di nspcc-dev/neofs-node

Tutte le issue di nspcc-dev/neofs-node

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.