EC Audit: C2: EC GET verifies neither the parts it reads nor the payload it assembles

Đang mở
#4,326 2 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
55/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go

Hướng nghiên cứu

Start with the referenced paths in pkg/services/object/get/util.go and pkg/services/object/get/ec.go, then read pkg/core/object/ec.go to see where part hashes are stored. Run the reproduction described in the issue with go test ./pkg/services/object/get/ -run TestECAuditC2 -v. Done means GET detects corrupt parts and assembled payloads and returns correct data or an error, including the ranged case.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug hal:proposed neofs-storage

Expected Behavior

A node that assembles an EC object uses only parts whose payload matches the hash recorded for them in the parent's __NEOFS__EC_PART_HASHES (or the part's own payload checksum). A part that fails is treated as missing and rebuilt from parity or read from another node. The client gets the stored bytes or an error.

Current Behavior

One part with a wrong payload gives the client wrong bytes with status OK.

  • Parts are requested from other nodes with checksum verification switched off (get/util.go#L330-L344, the TODO there and #3566).
  • getECPart reads as many bytes as the part header declares and checks nothing else about them (get/ec.go#L577-L638).
  • iec.Decode and iec.ConcatDataParts do not hash anything.
  • __NEOFS__EC_PART_HASHES is read only when a part is stored (core/object/ec.go#L135-L147). Neither GET nor the policer uses it.
  • The assembled payload is not compared with the parent's payload checksum.

Results of the test below (rule 3/2, 4096-byte object, one flipped byte in one stored part):

Request Parts Result
full GET all present, part 1 corrupted OK, payload differs from the object
full GET part 2 missing, part 0 corrupted OK, payload differs in 2 bytes (the flipped one and the same position in the rebuilt part)
ranged GET over part 1 part 1 corrupted OK, range differs from the object

A client using the SDK notices the first two cases after the download: it checks the payload against the header checksum for a full GET. It does not for ranged or payload-only requests (verifyChecksums = !skipChecksumVerification && !hasRange && !payloadOnly in client/object_get.go), so those have no check anywhere. In all cases the node could have served the right data from the remaining parts and does not try.

#3566 asks to bring the checksum check back for the node-to-node part request. That covers whole-part reads. Ranges of parts and the assembled result need their own check.

Possible Solution

Check each whole part against its entry in __NEOFS__EC_PART_HASHES before use; on mismatch count it as unavailable and go on to parity. Drop SkipChecksumVerification for part requests as #3566 proposes. A range of a part cannot be checked against a whole-part hash without reading the whole part, so ranged GET needs a decision: read whole parts, or keep the gap and document it.

Steps to Reproduce (for bugs)

  1. Save the test below as pkg/services/object/get/ecaudit_test.go.
  2. go test ./pkg/services/object/get/ -run TestECAuditC2 -v

All three subtests pass, that is, GET returns no error and a payload that differs from the stored one.

ecaudit_test.go
package getsvc

import (
	"bytes"
	"context"
	"io"
	"testing"

	iec "github.com/nspcc-dev/neofs-node/internal/ec"
	"github.com/nspcc-dev/neofs-node/internal/testutil"
	cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
	cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
	neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
	"github.com/nspcc-dev/neofs-sdk-go/netmap"
	"github.com/nspcc-dev/neofs-sdk-go/object"
	oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
	usertest "github.com/nspcc-dev/neofs-sdk-go/user/test"
	"github.com/stretchr/testify/require"
)

type ecAuditScene struct {
	cnr           cid.ID
	rule          iec.Rule
	parentHdr     object.Object
	parentAddr    oid.Address
	parentPayload []byte
	partObjs      []object.Object
	nodeSvcs      []*Service
	nodeLists     [][]netmap.NodeInfo
	getNodesMap   map[oid.Address]getNodesForObjectValue
	tc            *testECServiceConn
}

func newECAuditScene(t *testing.T, rule iec.Rule, pldLen int) *ecAuditScene {
	s := &ecAuditScene{cnr: cidtest.ID(), rule: rule, parentPayload: testutil.RandByteSlice(pldLen)}
	s.parentHdr.SetContainerID(s.cnr)
	s.parentHdr.SetPayloadSize(uint64(pldLen))
	require.NoError(t, s.parentHdr.CalculateAndSetID())
	s.parentAddr = oid.NewAddress(s.cnr, s.parentHdr.GetID())

	parts, _, err := iec.Encode(rule, s.parentPayload)
	require.NoError(t, err)
	s.partObjs = make([]object.Object, len(parts))
	for i := range parts {
		s.partObjs[i], err = iec.FormObjectForECPart(usertest.User(), s.parentHdr, parts[i], iec.PartInfo{Index: i})
		require.NoError(t, err)
	}

	s.nodeLists, _ = testNodeMatrix(t, []int{len(parts)})
	s.getNodesMap = map[oid.Address]getNodesForObjectValue{
		s.parentAddr: {nodeSets: s.nodeLists, ecRules: []iec.Rule{rule}},
	}
	s.tc = &testECServiceConn{
		mockKeyStorage: mockKeyStorage{privKey: neofscryptotest.ECDSAPrivateKey()},
		parentHdr:      s.parentHdr,
		nodes:          make(map[string]*Service),
	}
	s.nodeSvcs = make([]*Service, len(parts))
	for i := range s.nodeSvcs {
		s.nodeSvcs[i] = New(&mockNeoFSNet{getNodesForObject: s.getNodesMap, localPubKey: s.nodeLists[0][i].PublicKey()})
		s.setPart(i, s.partObjs[i].Payload())
		s.tc.nodes[string(s.nodeLists[0][i].Marshal())] = s.nodeSvcs[i]
	}
	return s
}

// setPart makes node #i serve pld as payload of part #i (nil = part is missing).
func (s *ecAuditScene) setPart(i int, pld []byte) {
	m := map[getECPartKey]getECPartValue{}
	if pld != nil {
		m[getECPartKey{cnr: s.cnr, parent: s.parentHdr.GetID(), pi: iec.PartInfo{Index: i}}] = getECPartValue{
			hdr: *s.partObjs[i].CutPayload(), pld: pld, rdr: io.NopCloser(bytes.NewReader(pld)),
		}
	}
	s.nodeSvcs[i].localObjects = &mockLocalObjects{getECPart: m}
}

func (s *ecAuditScene) service() *Service {
	svc := New(&mockNeoFSNet{getNodesForObject: s.getNodesMap})
	svc.keyStore = s.tc
	svc.conns = s.tc
	return svc
}

func flipByte(b []byte, i int) []byte {
	c := bytes.Clone(b)
	c[i] ^= 0xFF
	return c
}

func TestECAuditC2_NoIntegrityOnRead(t *testing.T) {
	ctx := context.Background()
	rule := iec.Rule{DataPartNum: 3, ParityPartNum: 2}
	const pldLen = 4096

	t.Run("full GET, all parts present, one data part corrupted", func(t *testing.T) {
		s := newECAuditScene(t, rule, pldLen)
		s.setPart(1, flipByte(s.partObjs[1].Payload(), 7))

		var w mockObjectWriter
		var prm Prm
		prm.WithAddress(s.parentAddr)
		prm.SetCommonParameters(newCommonParameters(false))
		prm.SetObjectWriter(&w)

		require.NoError(t, s.service().Get(ctx, prm))
		require.Len(t, w.buf.Bytes(), pldLen)
		require.NotEqual(t, s.parentPayload, w.buf.Bytes())
		t.Log("GET returned OK with payload differing from the stored object")
	})

	t.Run("full GET, one data part missing, another corrupted", func(t *testing.T) {
		s := newECAuditScene(t, rule, pldLen)
		s.setPart(0, flipByte(s.partObjs[0].Payload(), 7))
		s.setPart(2, nil)

		var w mockObjectWriter
		var prm Prm
		prm.WithAddress(s.parentAddr)
		prm.SetCommonParameters(newCommonParameters(false))
		prm.SetObjectWriter(&w)

		require.NoError(t, s.service().Get(ctx, prm))
		require.Len(t, w.buf.Bytes(), pldLen)
		require.NotEqual(t, s.parentPayload, w.buf.Bytes())
		diff := 0
		for i := range w.buf.Bytes() {
			if w.buf.Bytes()[i] != s.parentPayload[i] {
				diff++
			}
		}
		t.Logf("GET returned OK; 1 corrupted byte in part 0 + reconstruction of part 2 => %d differing bytes", diff)
	})

	t.Run("ranged GET over a corrupted data part", func(t *testing.T) {
		s := newECAuditScene(t, rule, pldLen)
		partLen := len(s.partObjs[0].Payload())
		s.setPart(1, flipByte(s.partObjs[1].Payload(), 7))

		var w mockObjectWriter
		var prm Prm
		prm.WithAddress(s.parentAddr)
		prm.SetCommonParameters(newCommonParameters(false))
		prm.SetObjectWriter(&w)
		prm.MarkPayloadOnly()
		rng := object.NewRange()
		rng.SetOffset(uint64(partLen))
		rng.SetLength(64)
		prm.SetRange(rng)

		require.NoError(t, s.service().Get(ctx, prm))
		require.Len(t, w.buf.Bytes(), 64)
		require.NotEqual(t, s.parentPayload[partLen:partLen+64], w.buf.Bytes())
		t.Log("ranged GET returned OK with payload differing from the stored object")
	})
}

Context

Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.

Regression

Not known.

Your Environment

  • Version used: 0.58.2 (1f0d51b5d)
  • Server setup and configuration: none, unit tests only
  • Operating System and version (uname -a): Darwin 25.6.0 arm64, Go 1.27.3
Ngôn ngữ chính
Go
Star
37
Fork
56
Merge trung bình
1 ngày 21 giờ
Pull request đã merge (30 ngày)
46

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của nspcc-dev/neofs-node

Tất cả issue của nspcc-dev/neofs-node

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.