Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

pusher: chunksWorker deadlocks on a deferred chunk that fails IdentityAddress (send on nil op.Err), halting all push-sync

Abierto
  • ¿Algo no está bien?
#5,641 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 2 días

@sbackend123 ya está trabajando en esto.

Desde el 6/10/2026.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
75/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go
Área
backend, distributed-systems

Línea de trabajo

Start in pkg/pusher/pusher.go, in chunksWorker where storage.IdentityAddress errors are handled; compare deferred operations with direct ones and with pushDeferred. Add a regression test in pkg/pusher based on TestInvalidChunkDoesNotBlockPusher, then run the package tests. Done means an invalid deferred chunk is reported as ChunkCouldNotSync, and a following valid chunk still syncs without the worker blocking.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

seen
Summary

pusher.(*Service).chunksWorker can block forever. Once that happens the node pushes nothing to the network again, and it stays that way across restarts.

For a queued (deferred) chunk on which storage.IdentityAddress fails, the worker sends the error on op.Err. For deferred ops op.Err is nil, so the send blocks forever:

// pkg/pusher/pusher.go (v2.8.0, v2.8.2 and master), chunksWorker
case op := <-cc:
    idAddress, err := storage.IdentityAddress(op.Chunk)
    if err != nil {
        op.Err <- err   // op.Err is nil for ops built from SubscribePush (Direct: false)
        continue
    }

The chunk stays in the push queue, so the worker blocks on it again after every restart.

What we saw (light node, bee v2.8.2)
  • The pusher stopped while a client was re-uploading ~4k single-owner chunks (a feed-chain re-stamp).
  • From then on, every upload tag showed sent 0.
  • Non-deferred uploads (POST /soc with Swarm-Deferred-Upload: false) hung until the client timed out.
  • Even at debug level, node/pusher and node/pushsync logged nothing.
  • /health, peer count (~100) and chain sync all looked normal.
  • A restart did not help. Neither did downgrading to v2.8.0, since that code is identical.
  • The goroutine dump (/debug/pprof/goroutine?debug=2) showed:
goroutine 427 [chan send (nil chan), 9 minutes]:
github.com/ethersphere/bee/v2/pkg/pusher.(*Service).chunksWorker(...)
	github.com/ethersphere/bee/v2/pkg/pusher/pusher.go:222 +0x6e5

It stalled for ~14 h until we deployed the patch below. The patched node logged one invalid chunk in push queue, dropping and then drained a ~35k-chunk backlog in a few minutes.

Suggested fix

Only send on op.Err for direct ops. Drop an invalid deferred chunk from the push queue the same way pushDeferred handles chunks it cannot sync:

 			idAddress, err := storage.IdentityAddress(op.Chunk)
 			if err != nil {
-				op.Err <- err
+				if op.Direct {
+					op.Err <- err
+				} else {
+					// A deferred op has no Err channel: sending on the nil
+					// channel blocked this worker forever and stalled all
+					// push-sync. Drop the invalid chunk from the push queue,
+					// as pushDeferred does for chunks it cannot sync.
+					s.logger.Error(err, "pusher: invalid chunk in push queue, dropping", "chunk_address", op.Chunk.Address())
+					if rerr := s.storer.Report(ctx, op.Chunk, storage.ChunkCouldNotSync); rerr != nil {
+						s.logger.Error(rerr, "pusher: failed to report invalid chunk", "chunk_address", op.Chunk.Address())
+					}
+				}
 				continue
 			}
Regression test

This test fails on v2.8.2 (pusher stalled behind an invalid chunk: timed out waiting for condition) and passes with the fix. The existing pkg/pusher tests still pass.

func TestInvalidChunkDoesNotBlockPusher(t *testing.T) {
	t.Parallel()

	key, _ := crypto.GenerateSecp256k1Key()
	signer := crypto.NewDefaultSigner(key)

	pushSyncService := pushsyncmock.New(func(ctx context.Context, chunk swarm.Chunk) (*pushsync.Receipt, error) {
		signature, _ := signer.Sign(chunk.Address().Bytes())
		return &pushsync.Receipt{Address: swarm.NewAddress(chunk.Address().Bytes()), Signature: signature, Nonce: block}, nil
	})

	storer := &mockStorer{chunks: make(chan swarm.Chunk)}
	_ = createPusher(t, storer, pushSyncService, defaultMockBatchStore, defaultRetryCount)

	valid := testingc.GenerateTestRandomChunk()
	addr := make([]byte, swarm.HashSize)
	for i := range addr {
		addr[i] = 0xab
	}
	// Shorter than a SOC header and not hashing to addr: neither CAC nor SOC.
	invalid := swarm.NewChunk(swarm.NewAddress(addr), []byte("not a chunk")).WithStamp(valid.Stamp())

	go func() {
		storer.chunks <- invalid
		storer.chunks <- valid
	}()

	if err := spinlock.Wait(spinTimeout, func() bool { return storer.isReported(valid, storage.ChunkSynced) }); err != nil {
		t.Fatalf("pusher stalled behind an invalid chunk: %v", err)
	}
	if !storer.isReported(invalid, storage.ChunkCouldNotSync) {
		t.Fatal("invalid chunk was not reported as could-not-sync")
	}
}
Open question

We don't know how the invalid chunk got into the push queue in the first place. The upload API is supposed to accept only valid CACs or SOCs. Hardening the worker fixes the stall either way. Happy to open a PR with the fix and test if that helps.

Lenguaje dominante
Go
Estrellas
1.5k
Forks
390
Merge medio
2 d 12 h
PR fusionados (30 d)
19

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ethersphere/bee

Todos los issues de ethersphere/bee

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.