Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

pusher: chunksWorker deadlocks on a deferred chunk that fails IdentityAddress (send on nil op.Err), halting all push-sync

Open
  • Is something wrong?
#5,641 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 2 days

@sbackend123 is already working on this.

Since Oct 6, 2026.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
backend, distributed-systems

Research direction

Start in pkg/pusher/pusher.go, in chunksWorker where storage.IdentityAddress errors are handled; compare deferred operations with direct ones and with pushDeferred. Add a regression test in pkg/pusher based on TestInvalidChunkDoesNotBlockPusher, then run the package tests. Done means an invalid deferred chunk is reported as ChunkCouldNotSync, and a following valid chunk still syncs without the worker blocking.

Written by the indexing model from the issue text.

Description

seen
Summary

pusher.(*Service).chunksWorker can block forever. Once that happens the node pushes nothing to the network again, and it stays that way across restarts.

For a queued (deferred) chunk on which storage.IdentityAddress fails, the worker sends the error on op.Err. For deferred ops op.Err is nil, so the send blocks forever:

// pkg/pusher/pusher.go (v2.8.0, v2.8.2 and master), chunksWorker
case op := <-cc:
    idAddress, err := storage.IdentityAddress(op.Chunk)
    if err != nil {
        op.Err <- err   // op.Err is nil for ops built from SubscribePush (Direct: false)
        continue
    }

The chunk stays in the push queue, so the worker blocks on it again after every restart.

What we saw (light node, bee v2.8.2)
  • The pusher stopped while a client was re-uploading ~4k single-owner chunks (a feed-chain re-stamp).
  • From then on, every upload tag showed sent 0.
  • Non-deferred uploads (POST /soc with Swarm-Deferred-Upload: false) hung until the client timed out.
  • Even at debug level, node/pusher and node/pushsync logged nothing.
  • /health, peer count (~100) and chain sync all looked normal.
  • A restart did not help. Neither did downgrading to v2.8.0, since that code is identical.
  • The goroutine dump (/debug/pprof/goroutine?debug=2) showed:
goroutine 427 [chan send (nil chan), 9 minutes]:
github.com/ethersphere/bee/v2/pkg/pusher.(*Service).chunksWorker(...)
	github.com/ethersphere/bee/v2/pkg/pusher/pusher.go:222 +0x6e5

It stalled for ~14 h until we deployed the patch below. The patched node logged one invalid chunk in push queue, dropping and then drained a ~35k-chunk backlog in a few minutes.

Suggested fix

Only send on op.Err for direct ops. Drop an invalid deferred chunk from the push queue the same way pushDeferred handles chunks it cannot sync:

 			idAddress, err := storage.IdentityAddress(op.Chunk)
 			if err != nil {
-				op.Err <- err
+				if op.Direct {
+					op.Err <- err
+				} else {
+					// A deferred op has no Err channel: sending on the nil
+					// channel blocked this worker forever and stalled all
+					// push-sync. Drop the invalid chunk from the push queue,
+					// as pushDeferred does for chunks it cannot sync.
+					s.logger.Error(err, "pusher: invalid chunk in push queue, dropping", "chunk_address", op.Chunk.Address())
+					if rerr := s.storer.Report(ctx, op.Chunk, storage.ChunkCouldNotSync); rerr != nil {
+						s.logger.Error(rerr, "pusher: failed to report invalid chunk", "chunk_address", op.Chunk.Address())
+					}
+				}
 				continue
 			}
Regression test

This test fails on v2.8.2 (pusher stalled behind an invalid chunk: timed out waiting for condition) and passes with the fix. The existing pkg/pusher tests still pass.

func TestInvalidChunkDoesNotBlockPusher(t *testing.T) {
	t.Parallel()

	key, _ := crypto.GenerateSecp256k1Key()
	signer := crypto.NewDefaultSigner(key)

	pushSyncService := pushsyncmock.New(func(ctx context.Context, chunk swarm.Chunk) (*pushsync.Receipt, error) {
		signature, _ := signer.Sign(chunk.Address().Bytes())
		return &pushsync.Receipt{Address: swarm.NewAddress(chunk.Address().Bytes()), Signature: signature, Nonce: block}, nil
	})

	storer := &mockStorer{chunks: make(chan swarm.Chunk)}
	_ = createPusher(t, storer, pushSyncService, defaultMockBatchStore, defaultRetryCount)

	valid := testingc.GenerateTestRandomChunk()
	addr := make([]byte, swarm.HashSize)
	for i := range addr {
		addr[i] = 0xab
	}
	// Shorter than a SOC header and not hashing to addr: neither CAC nor SOC.
	invalid := swarm.NewChunk(swarm.NewAddress(addr), []byte("not a chunk")).WithStamp(valid.Stamp())

	go func() {
		storer.chunks <- invalid
		storer.chunks <- valid
	}()

	if err := spinlock.Wait(spinTimeout, func() bool { return storer.isReported(valid, storage.ChunkSynced) }); err != nil {
		t.Fatalf("pusher stalled behind an invalid chunk: %v", err)
	}
	if !storer.isReported(invalid, storage.ChunkCouldNotSync) {
		t.Fatal("invalid chunk was not reported as could-not-sync")
	}
}
Open question

We don't know how the invalid chunk got into the push queue in the first place. The upload API is supposed to accept only valid CACs or SOCs. Hardening the worker fixes the stall either way. Happy to open a PR with the fix and test if that helps.

Dominant language
Go
Stars
1.5k
Forks
390
Avg merge
2d 12h
Merged PRs (30d)
18

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ethersphere/bee

All issues in ethersphere/bee

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.