policer: adaptive pacing instead of a fixed 10 obj/s default
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 20/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- go
- Ambito
- databases, distributed-systems
Direzione di ricerca
Start with pkg/services/policer/policer.go and process.go to trace the current batch pacing, then read check.go and pkg/services/replicator/replicator.go to understand how replication is dispatched. Review the related issues listed in the proposal before choosing an approach. Done means configurable pacing and in-flight limits, slow PUTs no longer stall scans, pending replicas are retained safely, and the stated throughput and reload criteria are met.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
The default policer rate is 10 objects/s (object_batch_size=10, replication_cooldown=1s ticker), 30/s in boost. That was a deliberate choice (#2617: "10 objects per second is OK..."), and #2590 even proposed making the policer slower to protect load. The problem is not the constant by itself; it is that the rate is fixed and replication runs inside the scan goroutine, so the effective rate is batch / max(cooldown, slowest object).
At 1 PB / REP 3 / 1 MB objects, each node holds ~125M local objects. A full pass is ~145 days at 10/s, ~48 days at 30/s. A lost 16 TB drive takes months to be fully repaired while using <1% of a 10 GbE link. #3762 (boost mode) is closed and helps ~40M-object networks, but 3× on 125M is still ~48 days, and it does not change the trigger.
Current behavior
pkg/services/policer/policer.go:158-160:batchSize=10,repCooldown=1s.pkg/services/policer/process.go:88,118: ticker gates every batch;process.go:136: boost multiplies the batch.pkg/services/policer/check.go:436:tryToReplicatecallsreplicator.HandleTaskinline.HandleTaskdoes a fullGetBytes+ remote PUT withput_timeout=1m(pkg/services/replicator/process.go:23), and the batch waits (process.go:164). One slow PUT stalls the batch.- The 8-worker/4096-slot replicator queue (
pkg/services/replicator/replicator.go:13-15) is bypassed by the policer; it only serves PUT-time post-placement. - No load feedback: boost reacts to missing replicas, not to client or drive load.
Impact
| objects/node | 10/s | 30/s | 100/s | 1000/s |
|---|---|---|---|---|
| 125M (1 PB, REP3, 1 MB) | 145 d | 48 d | 14.5 d | 1.45 d |
At 10/s and 1 MB objects, repair bandwidth is 10 MB/s; at 1000/s it is 1 GB/s. A 10 GbE link caps ~1000/s at 1 MB; 25 GbE is needed for a 24 h pass (~1500/s).
Proposal
- Replace the ticker with a token bucket + in-flight semaphore. Add
objects_per_secondandmax_inflight; keepreplication_cooldownback-compatible (derive rate from batch/cooldown when the new field is unset). - Move replication off the scan goroutine to the existing replicator queue with an in-flight cap. Record "replication pending" and do not drop the local copy until a remote PUT is confirmed. This is the larger lever; pacing alone will not help if a batch stalls on a 1-minute PUT.
- Optional second step: feed client in-flight request pressure into the pacer (MinIO-style
max_io), so the default is "as fast as the hardware allows without hurting clients" rather than a constant. - Metrics: cycle duration, objects/s, boost state, peer HEAD failure reasons, backlog/ETA.
Non-goals
- Raising
object_batch_sizeby 25× before (1) and (2): the scan also makes destructive decisions (dropRedundantLocalObject,deleteLocalObject, shard relocation), and a 25× rate multiplies the blast radius. See #4019, #3918, #2178. - Fixed erasure sets / protocol changes.
Acceptance criteria
- A 1 PB / 1 MB-object cluster can complete a full pass in ≤24 h with documented hardware (~1500 obj/s, 25 GbE).
- A slow replication PUT does not stall the scan batch.
- No local copy is dropped while a replication for it is pending.
- Rate/in-flight config reloads via SIGHUP; existing tests pass.
- Bench or test shows ≥100 obj/s against a mock network with the new pacer.
Related
-
#3762 (closed): boost mode; why 3× is not enough at PB scale.
-
#2590 (closed), #2617 (merged): why 1 s / 10 obj/s was chosen; this issue is about adaptive pacing, not a bigger constant.
-
#4019, #3918, #2178: why raising the rate blindly is dangerous.
-
#2021 (open): replicator metrics.
-
#1764 (open): config defaults/docs.
-
#4336 (open): per-node health state for unreachable holders.
- Lingua principale
- Go
- Stelle
- 37
- Fork
- 56
- Merge medio
- 1g 21h
- PR unite (30g)
- 46
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di nspcc-dev/neofs-node
-
bug hal:proposed neofs-storage
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
nspcc-dev/neofs-node#4325 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Check alpha scope in IRApertahal:proposed I4 neofs-storage S3 security U4
Difficoltà 2/5 1-3 ore Idoneità per principianti 69/100
nspcc-dev/neofs-node#4280 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
nspcc-dev/neofs-node#4339 ·
I maintainer di solito rispondono entro 1 giorno
-
bug neofs-storage
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
nspcc-dev/neofs-node#4336 ·
I maintainer di solito rispondono entro 1 giorno
-
bug neofs-storage
Difficoltà 3/5 1-2 giorni Idoneità per principianti 68/100
nspcc-dev/neofs-node#4333 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di nspcc-dev/neofs-node
Issue simili
-
Missing Readme InstructionsAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 78/100
OpenSlides/openslides-go#342 ·
I maintainer di solito rispondono entro 4 giorni
-
docs: link to the CLA in InditexTech/foss instead of keeping a local CLA.mdForse già presa @amagan l’ha presa oggi. Apertadocumentation
Difficoltà 1/5 1-3 ore Idoneità per principianti 88/100
InditexTech/redkey-operator#129 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 88/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
trufflesecurity/trufflehog#5573 ·
I maintainer di solito rispondono entro 1 giorno