EC Audit: C2: EC GET verifies neither the parts it reads nor the payload it assembles

Aperta
#4,326 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go

Direzione di ricerca

Start with the referenced paths in pkg/services/object/get/util.go and pkg/services/object/get/ec.go, then read pkg/core/object/ec.go to see where part hashes are stored. Run the reproduction described in the issue with go test ./pkg/services/object/get/ -run TestECAuditC2 -v. Done means GET detects corrupt parts and assembled payloads and returns correct data or an error, including the ranged case.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug hal:proposed neofs-storage

Expected Behavior

A node that assembles an EC object uses only parts whose payload matches the hash recorded for them in the parent's __NEOFS__EC_PART_HASHES (or the part's own payload checksum). A part that fails is treated as missing and rebuilt from parity or read from another node. The client gets the stored bytes or an error.

Current Behavior

One part with a wrong payload gives the client wrong bytes with status OK.

  • Parts are requested from other nodes with checksum verification switched off (get/util.go#L330-L344, the TODO there and #3566).
  • getECPart reads as many bytes as the part header declares and checks nothing else about them (get/ec.go#L577-L638).
  • iec.Decode and iec.ConcatDataParts do not hash anything.
  • __NEOFS__EC_PART_HASHES is read only when a part is stored (core/object/ec.go#L135-L147). Neither GET nor the policer uses it.
  • The assembled payload is not compared with the parent's payload checksum.

Results of the test below (rule 3/2, 4096-byte object, one flipped byte in one stored part):

Request Parts Result
full GET all present, part 1 corrupted OK, payload differs from the object
full GET part 2 missing, part 0 corrupted OK, payload differs in 2 bytes (the flipped one and the same position in the rebuilt part)
ranged GET over part 1 part 1 corrupted OK, range differs from the object

A client using the SDK notices the first two cases after the download: it checks the payload against the header checksum for a full GET. It does not for ranged or payload-only requests (verifyChecksums = !skipChecksumVerification && !hasRange && !payloadOnly in client/object_get.go), so those have no check anywhere. In all cases the node could have served the right data from the remaining parts and does not try.

#3566 asks to bring the checksum check back for the node-to-node part request. That covers whole-part reads. Ranges of parts and the assembled result need their own check.

Possible Solution

Check each whole part against its entry in __NEOFS__EC_PART_HASHES before use; on mismatch count it as unavailable and go on to parity. Drop SkipChecksumVerification for part requests as #3566 proposes. A range of a part cannot be checked against a whole-part hash without reading the whole part, so ranged GET needs a decision: read whole parts, or keep the gap and document it.

Steps to Reproduce (for bugs)

  1. Save the test below as pkg/services/object/get/ecaudit_test.go.
  2. go test ./pkg/services/object/get/ -run TestECAuditC2 -v

All three subtests pass, that is, GET returns no error and a payload that differs from the stored one.

ecaudit_test.go
package getsvc

import (
	"bytes"
	"context"
	"io"
	"testing"

	iec "github.com/nspcc-dev/neofs-node/internal/ec"
	"github.com/nspcc-dev/neofs-node/internal/testutil"
	cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
	cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
	neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
	"github.com/nspcc-dev/neofs-sdk-go/netmap"
	"github.com/nspcc-dev/neofs-sdk-go/object"
	oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
	usertest "github.com/nspcc-dev/neofs-sdk-go/user/test"
	"github.com/stretchr/testify/require"
)

type ecAuditScene struct {
	cnr           cid.ID
	rule          iec.Rule
	parentHdr     object.Object
	parentAddr    oid.Address
	parentPayload []byte
	partObjs      []object.Object
	nodeSvcs      []*Service
	nodeLists     [][]netmap.NodeInfo
	getNodesMap   map[oid.Address]getNodesForObjectValue
	tc            *testECServiceConn
}

func newECAuditScene(t *testing.T, rule iec.Rule, pldLen int) *ecAuditScene {
	s := &ecAuditScene{cnr: cidtest.ID(), rule: rule, parentPayload: testutil.RandByteSlice(pldLen)}
	s.parentHdr.SetContainerID(s.cnr)
	s.parentHdr.SetPayloadSize(uint64(pldLen))
	require.NoError(t, s.parentHdr.CalculateAndSetID())
	s.parentAddr = oid.NewAddress(s.cnr, s.parentHdr.GetID())

	parts, _, err := iec.Encode(rule, s.parentPayload)
	require.NoError(t, err)
	s.partObjs = make([]object.Object, len(parts))
	for i := range parts {
		s.partObjs[i], err = iec.FormObjectForECPart(usertest.User(), s.parentHdr, parts[i], iec.PartInfo{Index: i})
		require.NoError(t, err)
	}

	s.nodeLists, _ = testNodeMatrix(t, []int{len(parts)})
	s.getNodesMap = map[oid.Address]getNodesForObjectValue{
		s.parentAddr: {nodeSets: s.nodeLists, ecRules: []iec.Rule{rule}},
	}
	s.tc = &testECServiceConn{
		mockKeyStorage: mockKeyStorage{privKey: neofscryptotest.ECDSAPrivateKey()},
		parentHdr:      s.parentHdr,
		nodes:          make(map[string]*Service),
	}
	s.nodeSvcs = make([]*Service, len(parts))
	for i := range s.nodeSvcs {
		s.nodeSvcs[i] = New(&mockNeoFSNet{getNodesForObject: s.getNodesMap, localPubKey: s.nodeLists[0][i].PublicKey()})
		s.setPart(i, s.partObjs[i].Payload())
		s.tc.nodes[string(s.nodeLists[0][i].Marshal())] = s.nodeSvcs[i]
	}
	return s
}

// setPart makes node #i serve pld as payload of part #i (nil = part is missing).
func (s *ecAuditScene) setPart(i int, pld []byte) {
	m := map[getECPartKey]getECPartValue{}
	if pld != nil {
		m[getECPartKey{cnr: s.cnr, parent: s.parentHdr.GetID(), pi: iec.PartInfo{Index: i}}] = getECPartValue{
			hdr: *s.partObjs[i].CutPayload(), pld: pld, rdr: io.NopCloser(bytes.NewReader(pld)),
		}
	}
	s.nodeSvcs[i].localObjects = &mockLocalObjects{getECPart: m}
}

func (s *ecAuditScene) service() *Service {
	svc := New(&mockNeoFSNet{getNodesForObject: s.getNodesMap})
	svc.keyStore = s.tc
	svc.conns = s.tc
	return svc
}

func flipByte(b []byte, i int) []byte {
	c := bytes.Clone(b)
	c[i] ^= 0xFF
	return c
}

func TestECAuditC2_NoIntegrityOnRead(t *testing.T) {
	ctx := context.Background()
	rule := iec.Rule{DataPartNum: 3, ParityPartNum: 2}
	const pldLen = 4096

	t.Run("full GET, all parts present, one data part corrupted", func(t *testing.T) {
		s := newECAuditScene(t, rule, pldLen)
		s.setPart(1, flipByte(s.partObjs[1].Payload(), 7))

		var w mockObjectWriter
		var prm Prm
		prm.WithAddress(s.parentAddr)
		prm.SetCommonParameters(newCommonParameters(false))
		prm.SetObjectWriter(&w)

		require.NoError(t, s.service().Get(ctx, prm))
		require.Len(t, w.buf.Bytes(), pldLen)
		require.NotEqual(t, s.parentPayload, w.buf.Bytes())
		t.Log("GET returned OK with payload differing from the stored object")
	})

	t.Run("full GET, one data part missing, another corrupted", func(t *testing.T) {
		s := newECAuditScene(t, rule, pldLen)
		s.setPart(0, flipByte(s.partObjs[0].Payload(), 7))
		s.setPart(2, nil)

		var w mockObjectWriter
		var prm Prm
		prm.WithAddress(s.parentAddr)
		prm.SetCommonParameters(newCommonParameters(false))
		prm.SetObjectWriter(&w)

		require.NoError(t, s.service().Get(ctx, prm))
		require.Len(t, w.buf.Bytes(), pldLen)
		require.NotEqual(t, s.parentPayload, w.buf.Bytes())
		diff := 0
		for i := range w.buf.Bytes() {
			if w.buf.Bytes()[i] != s.parentPayload[i] {
				diff++
			}
		}
		t.Logf("GET returned OK; 1 corrupted byte in part 0 + reconstruction of part 2 => %d differing bytes", diff)
	})

	t.Run("ranged GET over a corrupted data part", func(t *testing.T) {
		s := newECAuditScene(t, rule, pldLen)
		partLen := len(s.partObjs[0].Payload())
		s.setPart(1, flipByte(s.partObjs[1].Payload(), 7))

		var w mockObjectWriter
		var prm Prm
		prm.WithAddress(s.parentAddr)
		prm.SetCommonParameters(newCommonParameters(false))
		prm.SetObjectWriter(&w)
		prm.MarkPayloadOnly()
		rng := object.NewRange()
		rng.SetOffset(uint64(partLen))
		rng.SetLength(64)
		prm.SetRange(rng)

		require.NoError(t, s.service().Get(ctx, prm))
		require.Len(t, w.buf.Bytes(), 64)
		require.NotEqual(t, s.parentPayload[partLen:partLen+64], w.buf.Bytes())
		t.Log("ranged GET returned OK with payload differing from the stored object")
	})
}

Context

Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.

Regression

Not known.

Your Environment

  • Version used: 0.58.2 (1f0d51b5d)
  • Server setup and configuration: none, unit tests only
  • Operating System and version (uname -a): Darwin 25.6.0 arm64, Go 1.27.3
Lingua principale
Go
Stelle
37
Fork
56
Merge medio
1g 21h
PR unite (30g)
46

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di nspcc-dev/neofs-node

Tutte le issue di nspcc-dev/neofs-node

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.