EC Audit: H1: ranged GET of an EC object fails with "object not found" when data part 0 is unavailable
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 62/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- go
- Ambito
- backend, distributed-systems
Direzione di ricerca
Start in pkg/services/object/get/ec.go, where copyECObjectRangeByRule reads part 0 before range recovery; compare that flow with getECObjectHeader and copyECObjectRangeByParts. Add the supplied regression test as pkg/services/object/get/ecaudit_test.go and run go test ./pkg/services/object/get/ -run TestECAuditH1 -v. Done when ranged GETs succeed with part 0 missing, including ranges that need its reconstruction.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Expected Behavior
A ranged GET of an EC object succeeds whenever enough parts are available to rebuild the requested range, as a full GET does.
Current Behavior
copyECObjectRangeByRule learns the parent header and payload length by opening part 0 of the rule, and gives up on the rule if that fails (get/ec.go#L1017-L1020). The code that rebuilds a part from parity (get/ec.go#L1145-L1152) comes after that and is not reached. When every rule fails this way the request ends with object not found (get/ec.go#L807).
So if part 0 cannot be read from any node of the rule, every ranged GET of the object fails, including ranges that do not touch part 0. Losing any other data part is handled.
Test below, rule 3/2, 3000-byte object:
| Lost part | Request | Result |
|---|---|---|
| 1 | range inside part 1 | OK, rebuilt from parity |
| 0 | full GET | OK |
| 0 | range inside part 0 | object not found (2049) |
| 0 | range inside part 1 | object not found (2049) |
status: code = 2049 message = object not found: failed processing of all 1 EC rules, first error: resolve parent payload length: all nodes failed
The object is intact and recoverable the whole time. The state lasts until the policer recreates part 0. With several EC rules the request fails only if part 0 is unavailable in each of them.
Possible Solution
When part 0 is unavailable, take the parent header from any other part of the rule (getECObjectHeader already works from any part) and call copyECObjectRangeByParts without a first-part stream. Its recovery branch then rebuilds part 0 if the range needs it.
Steps to Reproduce (for bugs)
- Save the test below as
pkg/services/object/get/ecaudit_test.go. go test ./pkg/services/object/get/ -run TestECAuditH1 -v
It passes, that is, it asserts the two object not found results.
ecaudit_test.go
package getsvc
import (
"bytes"
"context"
"io"
"testing"
iec "github.com/nspcc-dev/neofs-node/internal/ec"
"github.com/nspcc-dev/neofs-node/internal/testutil"
apistatus "github.com/nspcc-dev/neofs-sdk-go/client/status"
cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
usertest "github.com/nspcc-dev/neofs-sdk-go/user/test"
"github.com/stretchr/testify/require"
)
type ecAuditScene struct {
cnr cid.ID
rule iec.Rule
parentHdr object.Object
parentAddr oid.Address
parentPayload []byte
partObjs []object.Object
nodeSvcs []*Service
nodeLists [][]netmap.NodeInfo
getNodesMap map[oid.Address]getNodesForObjectValue
tc *testECServiceConn
}
func newECAuditScene(t *testing.T, rule iec.Rule, pldLen int) *ecAuditScene {
s := &ecAuditScene{cnr: cidtest.ID(), rule: rule, parentPayload: testutil.RandByteSlice(pldLen)}
s.parentHdr.SetContainerID(s.cnr)
s.parentHdr.SetPayloadSize(uint64(pldLen))
require.NoError(t, s.parentHdr.CalculateAndSetID())
s.parentAddr = oid.NewAddress(s.cnr, s.parentHdr.GetID())
parts, _, err := iec.Encode(rule, s.parentPayload)
require.NoError(t, err)
s.partObjs = make([]object.Object, len(parts))
for i := range parts {
s.partObjs[i], err = iec.FormObjectForECPart(usertest.User(), s.parentHdr, parts[i], iec.PartInfo{Index: i})
require.NoError(t, err)
}
s.nodeLists, _ = testNodeMatrix(t, []int{len(parts)})
s.getNodesMap = map[oid.Address]getNodesForObjectValue{
s.parentAddr: {nodeSets: s.nodeLists, ecRules: []iec.Rule{rule}},
}
s.tc = &testECServiceConn{
mockKeyStorage: mockKeyStorage{privKey: neofscryptotest.ECDSAPrivateKey()},
parentHdr: s.parentHdr,
nodes: make(map[string]*Service),
}
s.nodeSvcs = make([]*Service, len(parts))
for i := range s.nodeSvcs {
s.nodeSvcs[i] = New(&mockNeoFSNet{getNodesForObject: s.getNodesMap, localPubKey: s.nodeLists[0][i].PublicKey()})
s.setPart(i, s.partObjs[i].Payload())
s.tc.nodes[string(s.nodeLists[0][i].Marshal())] = s.nodeSvcs[i]
}
return s
}
// setPart makes node #i serve pld as payload of part #i (nil = part is missing).
func (s *ecAuditScene) setPart(i int, pld []byte) {
m := map[getECPartKey]getECPartValue{}
if pld != nil {
m[getECPartKey{cnr: s.cnr, parent: s.parentHdr.GetID(), pi: iec.PartInfo{Index: i}}] = getECPartValue{
hdr: *s.partObjs[i].CutPayload(), pld: pld, rdr: io.NopCloser(bytes.NewReader(pld)),
}
}
s.nodeSvcs[i].localObjects = &mockLocalObjects{getECPart: m}
}
func (s *ecAuditScene) service() *Service {
svc := New(&mockNeoFSNet{getNodesForObject: s.getNodesMap})
svc.keyStore = s.tc
svc.conns = s.tc
return svc
}
func TestECAuditH1_RangeNeedsPartZero(t *testing.T) {
ctx := context.Background()
s := newECAuditScene(t, iec.Rule{DataPartNum: 3, ParityPartNum: 2}, 3000)
partLen := uint64(len(s.partObjs[0].Payload()))
get := func(off, ln uint64) ([]byte, error) {
var w mockObjectWriter
var prm Prm
prm.WithAddress(s.parentAddr)
prm.SetCommonParameters(newCommonParameters(false))
prm.SetObjectWriter(&w)
if ln != 0 {
prm.MarkPayloadOnly()
rng := object.NewRange()
rng.SetOffset(off)
rng.SetLength(ln)
prm.SetRange(rng)
}
err := s.service().Get(ctx, prm)
return w.buf.Bytes(), err
}
reset := func(lost int) {
for i := range s.partObjs {
if i == lost {
s.setPart(i, nil)
} else {
s.setPart(i, s.partObjs[i].Payload())
}
}
}
// control: another data part is lost, the range inside it is rebuilt from parity
reset(1)
got, err := get(partLen+10, 20)
require.NoError(t, err)
require.Equal(t, s.parentPayload[partLen+10:partLen+30], got)
t.Log("part 1 lost, ranged GET inside part 1: OK")
// only part 0 is lost: 4 of 5 parts are there, 3 are enough
reset(0)
got, err = get(0, 0)
require.NoError(t, err)
require.Equal(t, s.parentPayload, got)
t.Log("part 0 lost, full GET: OK")
reset(0)
_, err = get(10, 20)
require.ErrorIs(t, err, apistatus.ErrObjectNotFound)
t.Logf("part 0 lost, ranged GET inside part 0: %v", err)
reset(0)
_, err = get(partLen+10, 20) // does not touch part 0
require.ErrorIs(t, err, apistatus.ErrObjectNotFound)
t.Logf("part 0 lost, ranged GET inside part 1: %v", err)
}
Context
Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.
Regression
Not known.
Your Environment
- Version used: 0.58.2 (
1f0d51b5d) - Server setup and configuration: none, unit tests only
- Operating System and version (
uname -a): Darwin 25.6.0 arm64, Go 1.27.3
- Lingua principale
- Go
- Stelle
- 37
- Fork
- 56
- Merge medio
- 1g 21h
- PR unite (30g)
- 46
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di nspcc-dev/neofs-node
-
bug hal:proposed neofs-storage
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
nspcc-dev/neofs-node#4325 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Check alpha scope in IRApertahal:proposed I4 neofs-storage S3 security U4
Difficoltà 2/5 1-3 ore Idoneità per principianti 69/100
nspcc-dev/neofs-node#4280 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
nspcc-dev/neofs-node#4339 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement I3 neofs-storage performance S3 U3
Difficoltà 5/5 Più di una settimana Idoneità per principianti 20/100
nspcc-dev/neofs-node#4337 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
bug neofs-storage
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
nspcc-dev/neofs-node#4336 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di nspcc-dev/neofs-node
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
trufflesecurity/trufflehog#5573 ·
I maintainer di solito rispondono entro 1 giorno
-
Type/Improvement
Difficoltà 2/5 1-3 ore Idoneità per principianti 67/100
OpenNSW/nsw-srilanka#1259 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
katzenpost/katzenpost#1287 ·
I maintainer di solito rispondono entro 2 giorni
-
[Bug] AI 对话切换 Claude Code 必定 400「不支持的执行器」:assistantEngineClaude 与 executorModelCatalogs 键名不一致Forse già presa Una pull request collegata a questa issue è aperta o già unita. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
Tencent/LoopForge#986 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno