Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Storage radius oscillates every wakeup when the reserve is full of out-of-radius chunks, leaking pullsync streams on peers

Aperta
  • Qualcosa non va?
#5,659 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go
Ambito
backend, distributed-systems, networking

Direzione di ricerca

Start with reserveWorker in pkg/storer/reserve.go and the pullsync handler in pkg/pullsync/pullsync.go; trace how radius changes restart the puller and how a reset stream affects the handler waiting in collectAddrs. Reproduce with a full reserve whose within-radius count is below the threshold, then verify the radius stops oscillating and cancelled requests no longer leave handlers blocked. The issue links #5657 and #5658 for related context.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Context

bee v2.8.2 (master has the same code). Mainnet full nodes.

Summary

A node whose reserve is full but holds mostly chunks outside its storage radius flips its radius down and back up on every reserve-worker wakeup (~30 min). Each flip restarts the puller. The puller cancels its sync requests, and every cancelled request leaves an inbound pullsync stream stuck on the peer that served it. The affected peers climb steadily towards the libp2p 5000 inbound-stream limit, after which they refuse all new inbound streams.

1. Radius flip (pkg/storer/reserve.go, reserveWorker)

On each thresholdTicker tick:

if count < threshold(db.reserve.Capacity()) && db.syncer.SyncRate() == 0 && radius > minimumRadius {
    radius--

count is the number of chunks within radius. If a reserve is at capacity but most of its chunks have a lower proximity order, count stays below 50% of capacity while IsWithinCapacity() is false. The radius drops (e.g. 9 → 8), the reserve is now over capacity, unreserve raises it back to 9, and the next tick repeats.

Observed on 3 nodes (capacity 4,194,304):

reserveSize reserveSizeWithinRadius
4,192,842 618,428
4,193,523 1,600,989
4,193,023 2,065,927

Healthy nodes in the same neighbourhoods hold ~3.82M within radius 9. These 3 nodes recorded 803 radius changes in 7 days; all other nodes recorded 0. One of them started flipping when a batch expiry pushed its within-radius count from 2,141,002 to 2,065,638, just under the 2,097,152 threshold.

Suggestion: do not decrease the radius while the reserve is at or above capacity, or evict out-of-radius chunks before evaluating the threshold.

2. Each flip leaks streams on peers (pkg/pullsync/pullsync.go)

Each radius change triggers puller.manage/onChange, which cancels sync workers and resets intervals. That produces ~600–970 context canceled sync errors per flip on the flipping node.

On the serving peer, the handler for each cancelled request stays blocked in makeOffer → collectAddrs → SubscribeBin waiting for the first chunk of the interval. The remote stream reset is not observed, and the handler's context is only cancelled on peer disconnect. So each flip leaves ~25–30 stuck streams per serving peer.

Evidence from one affected peer (/debug/pprof/goroutine?debug=2):

  • 3783 pullsync handlers blocked in collectAddrs, 3590 of them on a single connection to the flipping node. The other peers hold a normal 23–27.
  • The stream count rises by +25–30 within the same minute as each of the flipping node's puller error bursts (6/6 matches over 3 hours).
  • The ~2-minute radius-8 window means peers at PO 8 (the sibling half of the neighbourhood) also serve the flipping node, so they are affected too, not only same-neighbourhood peers.

The same server-side behaviour was triggered by a third-party client between September and October, and several nodes hit the 5000 cap. #5657 described the handler not observing resets but was closed. #5658 (a per-peer stream limit with disconnect) would contain the damage, but does not fix the cause.

Suggestion: cancel the handler's context when the stream is reset (or bound the first-chunk wait), plus #5658 as defence in depth, and backoff in the puller.

Reproduce

Any full node whose reserve fills with chunks below its eventual radius (e.g. it synced initially at a lower radius) and whose within-radius count falls under 50% of capacity.

Lingua principale
Go
Stelle
1.5k
Fork
390
Merge medio
2g 11h
PR unite (30g)
23

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ethersphere/bee

Tutte le issue di ethersphere/bee

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.