EC Audit: C1: metabase finds an EC part by index prefix, so part 1 also matches parts 10–19

Aperta Adatta ai principianti
#4,325 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
78/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go

Direzione di ricerca

Inizia da pkg/local_object_storage/metabase/ec.go, vicino a resolveECPartInMetaBucket, e confronta la ricerca delle parti con quella dell’indice delle regole. Poi aggiungi un test di regressione ai test del motore di archiviazione locale degli oggetti, usando le parti 1 e 10 e seguendo la riproduzione nell’issue. Esegui go test ./pkg/local_object_storage/engine/ -run TestECAuditC1 -v; il lavoro è completato quando la richiesta della parte 1 restituisce solo la parte 1 oppure “non trovato” se è assente.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug hal:proposed neofs-storage

Expected Behavior

A request for part i of an EC rule returns the object whose __NEOFS__EC_PART_IDX is exactly i, or "not found".

Current Behavior

resolveECPartInMetaBucket builds the key prefix …__NEOFS__EC_PART_IDX<delim><i> and takes the first key that starts with it (metabase/ec.go#L189-L203). The value is the tail of the key and nothing ends it, so index 1 also matches 10…19, 2 matches 20…29, and so on up to 6 matching 60…63. When a particular index is requested the function returns at once, without comparing the stored value. The rule index a few lines above is compared with bytes.Equal and is not affected.

Children of the parent are walked in OID order. So the wrong part comes back

  • when the shard holds only the part with the longer index, and
  • when it holds both and the OID of the longer one sorts first (12 of 20 random cases in the test below).

Any rule with 11 or more parts is affected, for example EC 12/4.

Nothing after the lookup notices:

  • Shard.GetECPart, HeadECPart, GetECPartRange and the Read* variants read the resolved object and do not look at its part index (shard/ec.go#L91-L123).
  • A node that fetches a whole part from another node checks the EC attributes of the reply (get/ec.go#L685), so that case is caught. A range of a part comes without a header (get/ec.go#L1367-L1379) and is taken as is. Ranged GET, the streaming full GET (parts after the first are read as ranges) and the policer's part fetch (policer/ec.go#L440-L443) all read ranges.
  • The policer counts a local HeadECPart hit as "part present" (policer/ec.go#L229-L266).

What this means for EC 12/4, by reading the code: the node holding part 1 is down. NodeSequenceForPart(1, 16, …) asks nodes 1, 2, 3, … in turn. Nodes 2–9 say "not found". Node 10 answers with part 10. A ranged read then returns bytes of part 10 in place of part 1, with status OK, instead of rebuilding part 1 from parity.

Possible Solution

Compare the whole key, as is done for the rule index: the value is the last component, so bytes.Equal(k, partPref) is enough when pi.Index >= 0. A test with parts 1 and 10 of one rule would cover it; the existing "another part" case uses 34 and 35.

Steps to Reproduce (for bugs)

  1. Save the test below as pkg/local_object_storage/engine/ecaudit_c1_test.go.
  2. go test ./pkg/local_object_storage/engine/ -run TestECAuditC1 -v

It passes, that is, it asserts the wrong behaviour:

HeadECPart(index=1) returned part with index 10
GetECPart(index=1) returned payload of part 10 (64 bytes)
GetECPartRange(index=1, off=8, len=16) returned bytes of part 10
part 10 returned for index 1 in 12 of 20 cases (whenever OID(part 10) < OID(part 1))
ecaudit_c1_test.go
package engine

import (
	"bytes"
	"context"
	"io"
	"testing"

	iec "github.com/nspcc-dev/neofs-node/internal/ec"
	"github.com/nspcc-dev/neofs-node/internal/testutil"
	"github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor"
	cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
	neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
	"github.com/nspcc-dev/neofs-sdk-go/object"
	oidtest "github.com/nspcc-dev/neofs-sdk-go/object/id/test"
	"github.com/stretchr/testify/require"
)

func TestECAuditC1_PartIndexPrefix(t *testing.T) {
	ctx := context.Background()
	cnr := cidtest.ID()
	signer := neofscryptotest.Signer()

	newPart := func(parent object.Object, idx int) object.Object {
		obj, err := iec.FormObjectForECPart(signer, parent, testutil.RandByteSlice(64), iec.PartInfo{RuleIndex: 0, Index: idx})
		require.NoError(t, err)
		return obj
	}

	t.Run("only part 10 stored, part 1 requested", func(t *testing.T) {
		parent := *generateObjectWithCID(cnr)
		parent.SetID(oidtest.ID())

		e := testNewEngineWithShards(testNewShard(t, 1))
		p10 := newPart(parent, 10)
		require.NoError(t, e.Put(ctx, &p10, nil))

		hdr, err := e.HeadECPart(ctx, cnr, parent.GetID(), iec.PartInfo{RuleIndex: 0, Index: 1})
		require.NoError(t, err) // want: ErrObjectNotFound
		got, err := iec.GetPartInfo(hdr)
		require.NoError(t, err)
		t.Logf("HeadECPart(index=1) returned part with index %d, OID %s", got.Index, hdr.GetID())
		require.Equal(t, 10, got.Index)

		hdr, rc, err := e.GetECPart(ctx, cnr, parent.GetID(), iec.PartInfo{RuleIndex: 0, Index: 1}, false)
		require.NoError(t, err)
		b, err := io.ReadAll(rc)
		require.NoError(t, err)
		rc.Close()
		require.True(t, bytes.Equal(b, p10.Payload()))
		t.Logf("GetECPart(index=1) returned payload of part 10 (%d bytes)", len(b))

		_, _, rc, err = e.GetECPartRange(ctx, cnr, parent.GetID(), iec.PartInfo{RuleIndex: 0, Index: 1}, blobstor.NewPayloadRange(8, 16), false)
		require.NoError(t, err)
		b, err = io.ReadAll(rc)
		require.NoError(t, err)
		rc.Close()
		require.True(t, bytes.Equal(b, p10.Payload()[8:24]))
		t.Logf("GetECPartRange(index=1, off=8, len=16) returned bytes of part 10")
	})

	t.Run("parts 1 and 10 stored, part 1 requested", func(t *testing.T) {
		wrong := 0
		const n = 20
		for range n {
			parent := *generateObjectWithCID(cnr)
			parent.SetID(oidtest.ID())

			e := testNewEngineWithShards(testNewShard(t, 1))
			p1, p10 := newPart(parent, 1), newPart(parent, 10)
			require.NoError(t, e.Put(ctx, &p1, nil))
			require.NoError(t, e.Put(ctx, &p10, nil))

			hdr, err := e.HeadECPart(ctx, cnr, parent.GetID(), iec.PartInfo{RuleIndex: 0, Index: 1})
			require.NoError(t, err)
			if hdr.GetID() == p10.GetID() {
				wrong++
				require.Negative(t, bytes.Compare(p10.GetID().Marshal(), p1.GetID().Marshal()))
			}
		}
		t.Logf("part 10 returned for index 1 in %d of %d cases (whenever OID(part 10) < OID(part 1))", wrong, n)
		require.NotZero(t, wrong)
	})
}

Context

Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.

Regression

Not known.

Your Environment

  • Version used: 0.58.2 (1f0d51b5d)
  • Server setup and configuration: none, unit tests only
  • Operating System and version (uname -a): Darwin 25.6.0 arm64, Go 1.27.3
Lingua principale
Go
Stelle
37
Fork
56
Merge medio
1g 18h
PR unite (30g)
42

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di nspcc-dev/neofs-node

Tutte le issue di nspcc-dev/neofs-node

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.