Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Storage radius oscillates every wakeup when the reserve is full of out-of-radius chunks, leaking pullsync streams on peers

Open
  • Is something wrong?
#5,659 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
backend, distributed-systems, networking

Research direction

Start with reserveWorker in pkg/storer/reserve.go and the pullsync handler in pkg/pullsync/pullsync.go; trace how radius changes restart the puller and how a reset stream affects the handler waiting in collectAddrs. Reproduce with a full reserve whose within-radius count is below the threshold, then verify the radius stops oscillating and cancelled requests no longer leave handlers blocked. The issue links #5657 and #5658 for related context.

Written by the indexing model from the issue text.

Description

Context

bee v2.8.2 (master has the same code). Mainnet full nodes.

Summary

A node whose reserve is full but holds mostly chunks outside its storage radius flips its radius down and back up on every reserve-worker wakeup (~30 min). Each flip restarts the puller. The puller cancels its sync requests, and every cancelled request leaves an inbound pullsync stream stuck on the peer that served it. The affected peers climb steadily towards the libp2p 5000 inbound-stream limit, after which they refuse all new inbound streams.

1. Radius flip (pkg/storer/reserve.go, reserveWorker)

On each thresholdTicker tick:

if count < threshold(db.reserve.Capacity()) && db.syncer.SyncRate() == 0 && radius > minimumRadius {
    radius--

count is the number of chunks within radius. If a reserve is at capacity but most of its chunks have a lower proximity order, count stays below 50% of capacity while IsWithinCapacity() is false. The radius drops (e.g. 9 → 8), the reserve is now over capacity, unreserve raises it back to 9, and the next tick repeats.

Observed on 3 nodes (capacity 4,194,304):

reserveSize reserveSizeWithinRadius
4,192,842 618,428
4,193,523 1,600,989
4,193,023 2,065,927

Healthy nodes in the same neighbourhoods hold ~3.82M within radius 9. These 3 nodes recorded 803 radius changes in 7 days; all other nodes recorded 0. One of them started flipping when a batch expiry pushed its within-radius count from 2,141,002 to 2,065,638, just under the 2,097,152 threshold.

Suggestion: do not decrease the radius while the reserve is at or above capacity, or evict out-of-radius chunks before evaluating the threshold.

2. Each flip leaks streams on peers (pkg/pullsync/pullsync.go)

Each radius change triggers puller.manage/onChange, which cancels sync workers and resets intervals. That produces ~600–970 context canceled sync errors per flip on the flipping node.

On the serving peer, the handler for each cancelled request stays blocked in makeOffer → collectAddrs → SubscribeBin waiting for the first chunk of the interval. The remote stream reset is not observed, and the handler's context is only cancelled on peer disconnect. So each flip leaves ~25–30 stuck streams per serving peer.

Evidence from one affected peer (/debug/pprof/goroutine?debug=2):

  • 3783 pullsync handlers blocked in collectAddrs, 3590 of them on a single connection to the flipping node. The other peers hold a normal 23–27.
  • The stream count rises by +25–30 within the same minute as each of the flipping node's puller error bursts (6/6 matches over 3 hours).
  • The ~2-minute radius-8 window means peers at PO 8 (the sibling half of the neighbourhood) also serve the flipping node, so they are affected too, not only same-neighbourhood peers.

The same server-side behaviour was triggered by a third-party client between September and October, and several nodes hit the 5000 cap. #5657 described the handler not observing resets but was closed. #5658 (a per-peer stream limit with disconnect) would contain the damage, but does not fix the cause.

Suggestion: cancel the handler's context when the stream is reset (or bound the first-chunk wait), plus #5658 as defence in depth, and backoff in the puller.

Reproduce

Any full node whose reserve fills with chunks below its eventual radius (e.g. it synced initially at a lower radius) and whose within-radius count falls under 50% of capacity.

Dominant language
Go
Stars
1.5k
Forks
390
Avg merge
2d 11h
Merged PRs (30d)
23

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ethersphere/bee

All issues in ethersphere/bee

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.