EC Audit: C1: metabase finds an EC part by index prefix, so part 1 also matches parts 10–19

Abierto Apto para principiantes
#4,325 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
78/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go

Línea de trabajo

Empieza en pkg/local_object_storage/metabase/ec.go, cerca de resolveECPartInMetaBucket, y compara la búsqueda de partes con la búsqueda en el índice de reglas. Después, añade una prueba de regresión a las pruebas del motor de almacenamiento local de objetos usando las partes 1 y 10 y siguiendo la reproducción del issue. Ejecuta go test ./pkg/local_object_storage/engine/ -run TestECAuditC1 -v; se considera terminado cuando solicitar la parte 1 devuelve solo la parte 1 o indica que no se encontró si está ausente.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug hal:proposed neofs-storage

Expected Behavior

A request for part i of an EC rule returns the object whose __NEOFS__EC_PART_IDX is exactly i, or "not found".

Current Behavior

resolveECPartInMetaBucket builds the key prefix …__NEOFS__EC_PART_IDX<delim><i> and takes the first key that starts with it (metabase/ec.go#L189-L203). The value is the tail of the key and nothing ends it, so index 1 also matches 10…19, 2 matches 20…29, and so on up to 6 matching 60…63. When a particular index is requested the function returns at once, without comparing the stored value. The rule index a few lines above is compared with bytes.Equal and is not affected.

Children of the parent are walked in OID order. So the wrong part comes back

  • when the shard holds only the part with the longer index, and
  • when it holds both and the OID of the longer one sorts first (12 of 20 random cases in the test below).

Any rule with 11 or more parts is affected, for example EC 12/4.

Nothing after the lookup notices:

  • Shard.GetECPart, HeadECPart, GetECPartRange and the Read* variants read the resolved object and do not look at its part index (shard/ec.go#L91-L123).
  • A node that fetches a whole part from another node checks the EC attributes of the reply (get/ec.go#L685), so that case is caught. A range of a part comes without a header (get/ec.go#L1367-L1379) and is taken as is. Ranged GET, the streaming full GET (parts after the first are read as ranges) and the policer's part fetch (policer/ec.go#L440-L443) all read ranges.
  • The policer counts a local HeadECPart hit as "part present" (policer/ec.go#L229-L266).

What this means for EC 12/4, by reading the code: the node holding part 1 is down. NodeSequenceForPart(1, 16, …) asks nodes 1, 2, 3, … in turn. Nodes 2–9 say "not found". Node 10 answers with part 10. A ranged read then returns bytes of part 10 in place of part 1, with status OK, instead of rebuilding part 1 from parity.

Possible Solution

Compare the whole key, as is done for the rule index: the value is the last component, so bytes.Equal(k, partPref) is enough when pi.Index >= 0. A test with parts 1 and 10 of one rule would cover it; the existing "another part" case uses 34 and 35.

Steps to Reproduce (for bugs)

  1. Save the test below as pkg/local_object_storage/engine/ecaudit_c1_test.go.
  2. go test ./pkg/local_object_storage/engine/ -run TestECAuditC1 -v

It passes, that is, it asserts the wrong behaviour:

HeadECPart(index=1) returned part with index 10
GetECPart(index=1) returned payload of part 10 (64 bytes)
GetECPartRange(index=1, off=8, len=16) returned bytes of part 10
part 10 returned for index 1 in 12 of 20 cases (whenever OID(part 10) < OID(part 1))
ecaudit_c1_test.go
package engine

import (
	"bytes"
	"context"
	"io"
	"testing"

	iec "github.com/nspcc-dev/neofs-node/internal/ec"
	"github.com/nspcc-dev/neofs-node/internal/testutil"
	"github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor"
	cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
	neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
	"github.com/nspcc-dev/neofs-sdk-go/object"
	oidtest "github.com/nspcc-dev/neofs-sdk-go/object/id/test"
	"github.com/stretchr/testify/require"
)

func TestECAuditC1_PartIndexPrefix(t *testing.T) {
	ctx := context.Background()
	cnr := cidtest.ID()
	signer := neofscryptotest.Signer()

	newPart := func(parent object.Object, idx int) object.Object {
		obj, err := iec.FormObjectForECPart(signer, parent, testutil.RandByteSlice(64), iec.PartInfo{RuleIndex: 0, Index: idx})
		require.NoError(t, err)
		return obj
	}

	t.Run("only part 10 stored, part 1 requested", func(t *testing.T) {
		parent := *generateObjectWithCID(cnr)
		parent.SetID(oidtest.ID())

		e := testNewEngineWithShards(testNewShard(t, 1))
		p10 := newPart(parent, 10)
		require.NoError(t, e.Put(ctx, &p10, nil))

		hdr, err := e.HeadECPart(ctx, cnr, parent.GetID(), iec.PartInfo{RuleIndex: 0, Index: 1})
		require.NoError(t, err) // want: ErrObjectNotFound
		got, err := iec.GetPartInfo(hdr)
		require.NoError(t, err)
		t.Logf("HeadECPart(index=1) returned part with index %d, OID %s", got.Index, hdr.GetID())
		require.Equal(t, 10, got.Index)

		hdr, rc, err := e.GetECPart(ctx, cnr, parent.GetID(), iec.PartInfo{RuleIndex: 0, Index: 1}, false)
		require.NoError(t, err)
		b, err := io.ReadAll(rc)
		require.NoError(t, err)
		rc.Close()
		require.True(t, bytes.Equal(b, p10.Payload()))
		t.Logf("GetECPart(index=1) returned payload of part 10 (%d bytes)", len(b))

		_, _, rc, err = e.GetECPartRange(ctx, cnr, parent.GetID(), iec.PartInfo{RuleIndex: 0, Index: 1}, blobstor.NewPayloadRange(8, 16), false)
		require.NoError(t, err)
		b, err = io.ReadAll(rc)
		require.NoError(t, err)
		rc.Close()
		require.True(t, bytes.Equal(b, p10.Payload()[8:24]))
		t.Logf("GetECPartRange(index=1, off=8, len=16) returned bytes of part 10")
	})

	t.Run("parts 1 and 10 stored, part 1 requested", func(t *testing.T) {
		wrong := 0
		const n = 20
		for range n {
			parent := *generateObjectWithCID(cnr)
			parent.SetID(oidtest.ID())

			e := testNewEngineWithShards(testNewShard(t, 1))
			p1, p10 := newPart(parent, 1), newPart(parent, 10)
			require.NoError(t, e.Put(ctx, &p1, nil))
			require.NoError(t, e.Put(ctx, &p10, nil))

			hdr, err := e.HeadECPart(ctx, cnr, parent.GetID(), iec.PartInfo{RuleIndex: 0, Index: 1})
			require.NoError(t, err)
			if hdr.GetID() == p10.GetID() {
				wrong++
				require.Negative(t, bytes.Compare(p10.GetID().Marshal(), p1.GetID().Marshal()))
			}
		}
		t.Logf("part 10 returned for index 1 in %d of %d cases (whenever OID(part 10) < OID(part 1))", wrong, n)
		require.NotZero(t, wrong)
	})
}

Context

Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.

Regression

Not known.

Your Environment

  • Version used: 0.58.2 (1f0d51b5d)
  • Server setup and configuration: none, unit tests only
  • Operating System and version (uname -a): Darwin 25.6.0 arm64, Go 1.27.3
Lenguaje dominante
Go
Estrellas
37
Forks
56
Merge medio
1 d 18 h
PR fusionados (30 d)
42

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de nspcc-dev/neofs-node

Todos los issues de nspcc-dev/neofs-node

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.