policer: no per-node health state for unreachable holders

Aperta
#4,336 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go

Direzione di ricerca

Start with pkg/services/policer/check.go:284-336 and the head_timeout configuration in cmd/neofs-node/config/policer/policer.go; review related issues #1969 and #1757 for failure classification and backoff context. Trace how HEAD errors affect placement and backup repair, then add per-node failure tracking, expiry, configuration and metrics with tests for timeout, refused connections and 404. Done means repeated HEADs are limited per TTL, failures are observable, and backup repair still works without a request storm when a node returns.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug neofs-storage

Summary

processNodes HEADs every holder for every object with head_timeout (5 s default). There is no per-node negative cache: a node that is down or blackholed is retried for every object, and the only trace is a repeated log line. A blackholed node can make each object pay the full timeout, collapsing the scan rate; a node that refuses connections fails fast but still costs one attempt per object.

Note: with spare candidates (CBF > 1), a down node's copy is still repaired via a backup 404; the gap is that the policer cannot distinguish a persistently unreachable node from a transient error, cannot back off, and cannot tell operators which node is failing.

Current behavior

  • pkg/services/policer/check.go:284-336: the placement loop HEADs each remote node sequentially, one context.WithTimeout(ctx, headTimeout) per node.
  • check.go:320-336: only apistatus.ErrObjectNotFound creates a candidate; transport errors are logged and ignored.
  • No per-node cache or metric; the same node is HEADed for every object.
  • head_timeout default 5 s (cmd/neofs-node/config/policer/policer.go).

Impact

  • In a partition/blackhole, a 10-object batch can spend up to ~5 s per object waiting for the dead node, i.e. ~2 obj/s instead of 10. At 1 PB / 125M local objects, that is years per pass instead of months.
  • Operators get a log flood, not a per-node signal; #1988 is related.
  • Repair via backups still happens, but at the degraded rate.

Proposed fix

  • Per-node health cache: consecutive failures, last success, TTL (e.g. 30 s). Skip HEADs to known-unreachable nodes within the TTL, count their copy as missing, reset on success.
  • Config: head_failure_threshold, unreachable_ttl.
  • Metrics: policer_peer_head_failures_total{node,reason}, policer_peer_unavailable.
  • Do not treat a single failure as unavailable; require K consecutive failures (see also #1969 for which errors count).

Acceptance criteria

  • With one blackholed node, the policer issues at most one HEAD to it per TTL, not one per object.
  • The per-node failure is visible in metrics.
  • Repair via backup still happens; no storm when the node returns.
  • Unit tests with fake HEAD timeout/refused/404; existing policer tests pass.

Related

  • #4316 (open): read path maps unreachable holders to 404.

  • #2085 (closed, fixed by #3383): HEAD now reads storage.

  • #1757 (closed): reconnect backoff for failed nodes.

  • #1988 (open): duplicate policer error logs.

  • #1969 (open): which disk errors count.

  • #4337 (open): adaptive pacing; a down node also makes the scan slower.

Lingua principale
Go
Stelle
37
Fork
56
Merge medio
1g 21h
PR unite (30g)
46

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di nspcc-dev/neofs-node

Tutte le issue di nspcc-dev/neofs-node

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.