EC Audit: C2: EC GET verifies neither the parts it reads nor the payload it assembles
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 55/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- go
- Ambito
- databases, distributed-systems
Direzione di ricerca
Start with the referenced paths in pkg/services/object/get/util.go and pkg/services/object/get/ec.go, then read pkg/core/object/ec.go to see where part hashes are stored. Run the reproduction described in the issue with go test ./pkg/services/object/get/ -run TestECAuditC2 -v. Done means GET detects corrupt parts and assembled payloads and returns correct data or an error, including the ranged case.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Expected Behavior
A node that assembles an EC object uses only parts whose payload matches the hash recorded for them in the parent's __NEOFS__EC_PART_HASHES (or the part's own payload checksum). A part that fails is treated as missing and rebuilt from parity or read from another node. The client gets the stored bytes or an error.
Current Behavior
One part with a wrong payload gives the client wrong bytes with status OK.
- Parts are requested from other nodes with checksum verification switched off (get/util.go#L330-L344, the TODO there and #3566).
getECPartreads as many bytes as the part header declares and checks nothing else about them (get/ec.go#L577-L638).iec.Decodeandiec.ConcatDataPartsdo not hash anything.__NEOFS__EC_PART_HASHESis read only when a part is stored (core/object/ec.go#L135-L147). Neither GET nor the policer uses it.- The assembled payload is not compared with the parent's payload checksum.
Results of the test below (rule 3/2, 4096-byte object, one flipped byte in one stored part):
| Request | Parts | Result |
|---|---|---|
| full GET | all present, part 1 corrupted | OK, payload differs from the object |
| full GET | part 2 missing, part 0 corrupted | OK, payload differs in 2 bytes (the flipped one and the same position in the rebuilt part) |
| ranged GET over part 1 | part 1 corrupted | OK, range differs from the object |
A client using the SDK notices the first two cases after the download: it checks the payload against the header checksum for a full GET. It does not for ranged or payload-only requests (verifyChecksums = !skipChecksumVerification && !hasRange && !payloadOnly in client/object_get.go), so those have no check anywhere. In all cases the node could have served the right data from the remaining parts and does not try.
#3566 asks to bring the checksum check back for the node-to-node part request. That covers whole-part reads. Ranges of parts and the assembled result need their own check.
Possible Solution
Check each whole part against its entry in __NEOFS__EC_PART_HASHES before use; on mismatch count it as unavailable and go on to parity. Drop SkipChecksumVerification for part requests as #3566 proposes. A range of a part cannot be checked against a whole-part hash without reading the whole part, so ranged GET needs a decision: read whole parts, or keep the gap and document it.
Steps to Reproduce (for bugs)
- Save the test below as
pkg/services/object/get/ecaudit_test.go. go test ./pkg/services/object/get/ -run TestECAuditC2 -v
All three subtests pass, that is, GET returns no error and a payload that differs from the stored one.
ecaudit_test.go
package getsvc
import (
"bytes"
"context"
"io"
"testing"
iec "github.com/nspcc-dev/neofs-node/internal/ec"
"github.com/nspcc-dev/neofs-node/internal/testutil"
cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
usertest "github.com/nspcc-dev/neofs-sdk-go/user/test"
"github.com/stretchr/testify/require"
)
type ecAuditScene struct {
cnr cid.ID
rule iec.Rule
parentHdr object.Object
parentAddr oid.Address
parentPayload []byte
partObjs []object.Object
nodeSvcs []*Service
nodeLists [][]netmap.NodeInfo
getNodesMap map[oid.Address]getNodesForObjectValue
tc *testECServiceConn
}
func newECAuditScene(t *testing.T, rule iec.Rule, pldLen int) *ecAuditScene {
s := &ecAuditScene{cnr: cidtest.ID(), rule: rule, parentPayload: testutil.RandByteSlice(pldLen)}
s.parentHdr.SetContainerID(s.cnr)
s.parentHdr.SetPayloadSize(uint64(pldLen))
require.NoError(t, s.parentHdr.CalculateAndSetID())
s.parentAddr = oid.NewAddress(s.cnr, s.parentHdr.GetID())
parts, _, err := iec.Encode(rule, s.parentPayload)
require.NoError(t, err)
s.partObjs = make([]object.Object, len(parts))
for i := range parts {
s.partObjs[i], err = iec.FormObjectForECPart(usertest.User(), s.parentHdr, parts[i], iec.PartInfo{Index: i})
require.NoError(t, err)
}
s.nodeLists, _ = testNodeMatrix(t, []int{len(parts)})
s.getNodesMap = map[oid.Address]getNodesForObjectValue{
s.parentAddr: {nodeSets: s.nodeLists, ecRules: []iec.Rule{rule}},
}
s.tc = &testECServiceConn{
mockKeyStorage: mockKeyStorage{privKey: neofscryptotest.ECDSAPrivateKey()},
parentHdr: s.parentHdr,
nodes: make(map[string]*Service),
}
s.nodeSvcs = make([]*Service, len(parts))
for i := range s.nodeSvcs {
s.nodeSvcs[i] = New(&mockNeoFSNet{getNodesForObject: s.getNodesMap, localPubKey: s.nodeLists[0][i].PublicKey()})
s.setPart(i, s.partObjs[i].Payload())
s.tc.nodes[string(s.nodeLists[0][i].Marshal())] = s.nodeSvcs[i]
}
return s
}
// setPart makes node #i serve pld as payload of part #i (nil = part is missing).
func (s *ecAuditScene) setPart(i int, pld []byte) {
m := map[getECPartKey]getECPartValue{}
if pld != nil {
m[getECPartKey{cnr: s.cnr, parent: s.parentHdr.GetID(), pi: iec.PartInfo{Index: i}}] = getECPartValue{
hdr: *s.partObjs[i].CutPayload(), pld: pld, rdr: io.NopCloser(bytes.NewReader(pld)),
}
}
s.nodeSvcs[i].localObjects = &mockLocalObjects{getECPart: m}
}
func (s *ecAuditScene) service() *Service {
svc := New(&mockNeoFSNet{getNodesForObject: s.getNodesMap})
svc.keyStore = s.tc
svc.conns = s.tc
return svc
}
func flipByte(b []byte, i int) []byte {
c := bytes.Clone(b)
c[i] ^= 0xFF
return c
}
func TestECAuditC2_NoIntegrityOnRead(t *testing.T) {
ctx := context.Background()
rule := iec.Rule{DataPartNum: 3, ParityPartNum: 2}
const pldLen = 4096
t.Run("full GET, all parts present, one data part corrupted", func(t *testing.T) {
s := newECAuditScene(t, rule, pldLen)
s.setPart(1, flipByte(s.partObjs[1].Payload(), 7))
var w mockObjectWriter
var prm Prm
prm.WithAddress(s.parentAddr)
prm.SetCommonParameters(newCommonParameters(false))
prm.SetObjectWriter(&w)
require.NoError(t, s.service().Get(ctx, prm))
require.Len(t, w.buf.Bytes(), pldLen)
require.NotEqual(t, s.parentPayload, w.buf.Bytes())
t.Log("GET returned OK with payload differing from the stored object")
})
t.Run("full GET, one data part missing, another corrupted", func(t *testing.T) {
s := newECAuditScene(t, rule, pldLen)
s.setPart(0, flipByte(s.partObjs[0].Payload(), 7))
s.setPart(2, nil)
var w mockObjectWriter
var prm Prm
prm.WithAddress(s.parentAddr)
prm.SetCommonParameters(newCommonParameters(false))
prm.SetObjectWriter(&w)
require.NoError(t, s.service().Get(ctx, prm))
require.Len(t, w.buf.Bytes(), pldLen)
require.NotEqual(t, s.parentPayload, w.buf.Bytes())
diff := 0
for i := range w.buf.Bytes() {
if w.buf.Bytes()[i] != s.parentPayload[i] {
diff++
}
}
t.Logf("GET returned OK; 1 corrupted byte in part 0 + reconstruction of part 2 => %d differing bytes", diff)
})
t.Run("ranged GET over a corrupted data part", func(t *testing.T) {
s := newECAuditScene(t, rule, pldLen)
partLen := len(s.partObjs[0].Payload())
s.setPart(1, flipByte(s.partObjs[1].Payload(), 7))
var w mockObjectWriter
var prm Prm
prm.WithAddress(s.parentAddr)
prm.SetCommonParameters(newCommonParameters(false))
prm.SetObjectWriter(&w)
prm.MarkPayloadOnly()
rng := object.NewRange()
rng.SetOffset(uint64(partLen))
rng.SetLength(64)
prm.SetRange(rng)
require.NoError(t, s.service().Get(ctx, prm))
require.Len(t, w.buf.Bytes(), 64)
require.NotEqual(t, s.parentPayload[partLen:partLen+64], w.buf.Bytes())
t.Log("ranged GET returned OK with payload differing from the stored object")
})
}
Context
Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.
Regression
Not known.
Your Environment
- Version used: 0.58.2 (
1f0d51b5d) - Server setup and configuration: none, unit tests only
- Operating System and version (
uname -a): Darwin 25.6.0 arm64, Go 1.27.3
- Lingua principale
- Go
- Stelle
- 37
- Fork
- 56
- Merge medio
- 1g 21h
- PR unite (30g)
- 46
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di nspcc-dev/neofs-node
-
bug hal:proposed neofs-storage
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
nspcc-dev/neofs-node#4325 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Check alpha scope in IRApertahal:proposed I4 neofs-storage S3 security U4
Difficoltà 2/5 1-3 ore Idoneità per principianti 69/100
nspcc-dev/neofs-node#4280 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
nspcc-dev/neofs-node#4339 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement I3 neofs-storage performance S3 U3
Difficoltà 5/5 Più di una settimana Idoneità per principianti 20/100
nspcc-dev/neofs-node#4337 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
bug neofs-storage
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
nspcc-dev/neofs-node#4336 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di nspcc-dev/neofs-node
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
trufflesecurity/trufflehog#5573 ·
I maintainer di solito rispondono entro 1 giorno
-
Type/Improvement
Difficoltà 2/5 1-3 ore Idoneità per principianti 67/100
OpenNSW/nsw-srilanka#1259 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
katzenpost/katzenpost#1287 ·
I maintainer di solito rispondono entro 2 giorni
-
C-bug O-community X-blathers-untriaged
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
cockroachdb/cockroach#176245 · 1 commento ·