Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

pusher: chunksWorker deadlocks on a deferred chunk that fails IdentityAddress (send on nil op.Err), halting all push-sync

Aperta
  • Qualcosa non va?
#5,641 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@sbackend123 ci sta già lavorando.

Dal 6/10/2026.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
75/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go
Ambito
backend, distributed-systems

Direzione di ricerca

Start in pkg/pusher/pusher.go, in chunksWorker where storage.IdentityAddress errors are handled; compare deferred operations with direct ones and with pushDeferred. Add a regression test in pkg/pusher based on TestInvalidChunkDoesNotBlockPusher, then run the package tests. Done means an invalid deferred chunk is reported as ChunkCouldNotSync, and a following valid chunk still syncs without the worker blocking.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

seen
Summary

pusher.(*Service).chunksWorker can block forever. Once that happens the node pushes nothing to the network again, and it stays that way across restarts.

For a queued (deferred) chunk on which storage.IdentityAddress fails, the worker sends the error on op.Err. For deferred ops op.Err is nil, so the send blocks forever:

// pkg/pusher/pusher.go (v2.8.0, v2.8.2 and master), chunksWorker
case op := <-cc:
    idAddress, err := storage.IdentityAddress(op.Chunk)
    if err != nil {
        op.Err <- err   // op.Err is nil for ops built from SubscribePush (Direct: false)
        continue
    }

The chunk stays in the push queue, so the worker blocks on it again after every restart.

What we saw (light node, bee v2.8.2)
  • The pusher stopped while a client was re-uploading ~4k single-owner chunks (a feed-chain re-stamp).
  • From then on, every upload tag showed sent 0.
  • Non-deferred uploads (POST /soc with Swarm-Deferred-Upload: false) hung until the client timed out.
  • Even at debug level, node/pusher and node/pushsync logged nothing.
  • /health, peer count (~100) and chain sync all looked normal.
  • A restart did not help. Neither did downgrading to v2.8.0, since that code is identical.
  • The goroutine dump (/debug/pprof/goroutine?debug=2) showed:
goroutine 427 [chan send (nil chan), 9 minutes]:
github.com/ethersphere/bee/v2/pkg/pusher.(*Service).chunksWorker(...)
	github.com/ethersphere/bee/v2/pkg/pusher/pusher.go:222 +0x6e5

It stalled for ~14 h until we deployed the patch below. The patched node logged one invalid chunk in push queue, dropping and then drained a ~35k-chunk backlog in a few minutes.

Suggested fix

Only send on op.Err for direct ops. Drop an invalid deferred chunk from the push queue the same way pushDeferred handles chunks it cannot sync:

 			idAddress, err := storage.IdentityAddress(op.Chunk)
 			if err != nil {
-				op.Err <- err
+				if op.Direct {
+					op.Err <- err
+				} else {
+					// A deferred op has no Err channel: sending on the nil
+					// channel blocked this worker forever and stalled all
+					// push-sync. Drop the invalid chunk from the push queue,
+					// as pushDeferred does for chunks it cannot sync.
+					s.logger.Error(err, "pusher: invalid chunk in push queue, dropping", "chunk_address", op.Chunk.Address())
+					if rerr := s.storer.Report(ctx, op.Chunk, storage.ChunkCouldNotSync); rerr != nil {
+						s.logger.Error(rerr, "pusher: failed to report invalid chunk", "chunk_address", op.Chunk.Address())
+					}
+				}
 				continue
 			}
Regression test

This test fails on v2.8.2 (pusher stalled behind an invalid chunk: timed out waiting for condition) and passes with the fix. The existing pkg/pusher tests still pass.

func TestInvalidChunkDoesNotBlockPusher(t *testing.T) {
	t.Parallel()

	key, _ := crypto.GenerateSecp256k1Key()
	signer := crypto.NewDefaultSigner(key)

	pushSyncService := pushsyncmock.New(func(ctx context.Context, chunk swarm.Chunk) (*pushsync.Receipt, error) {
		signature, _ := signer.Sign(chunk.Address().Bytes())
		return &pushsync.Receipt{Address: swarm.NewAddress(chunk.Address().Bytes()), Signature: signature, Nonce: block}, nil
	})

	storer := &mockStorer{chunks: make(chan swarm.Chunk)}
	_ = createPusher(t, storer, pushSyncService, defaultMockBatchStore, defaultRetryCount)

	valid := testingc.GenerateTestRandomChunk()
	addr := make([]byte, swarm.HashSize)
	for i := range addr {
		addr[i] = 0xab
	}
	// Shorter than a SOC header and not hashing to addr: neither CAC nor SOC.
	invalid := swarm.NewChunk(swarm.NewAddress(addr), []byte("not a chunk")).WithStamp(valid.Stamp())

	go func() {
		storer.chunks <- invalid
		storer.chunks <- valid
	}()

	if err := spinlock.Wait(spinTimeout, func() bool { return storer.isReported(valid, storage.ChunkSynced) }); err != nil {
		t.Fatalf("pusher stalled behind an invalid chunk: %v", err)
	}
	if !storer.isReported(invalid, storage.ChunkCouldNotSync) {
		t.Fatal("invalid chunk was not reported as could-not-sync")
	}
}
Open question

We don't know how the invalid chunk got into the push queue in the first place. The upload API is supposed to accept only valid CACs or SOCs. Hardening the worker fixes the stall either way. Happy to open a PR with the fix and test if that helps.

Lingua principale
Go
Stelle
1.5k
Fork
390
Merge medio
2g 23h
PR unite (30g)
21

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ethersphere/bee

Tutte le issue di ethersphere/bee

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.