EC Audit: H6: partly stored EC PUT is answered with "internal server error" and no object ID
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 68/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- go
- Ambito
- backend, distributed-systems
Direzione di ricerca
Start in pkg/services/object/put/distributed.go, especially handleECRule, Close, and the REP branch that uses newCompletionError. Compare how EC and REP failures are classified when some parts or replicas are stored, then add or update a focused test in pkg/services/object/put/ for the EC case. Done means a partial EC PUT returns INCOMPLETE with the object ID, consistent with the expected behavior in the issue.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Expected Behavior
When a PUT into an EC container stores some parts but not all, the client gets status INCOMPLETE and the object ID, as it does for a REP container. It can then tell "nothing was stored" from "stored but under-replicated", and can read or delete the object.
Current Behavior
The EC path returns errIncompletePut, which has neither Is nor Unwrap (put/distributed.go#L108-L121), and handleECRule passes it on as is (put/distributed.go#L468-L474). Close returns the object ID only for errors that match apistatus.ErrIncomplete (put/distributed.go#L305-L311), so this one comes back with a zero ID and is converted to status 1024. The REP path wraps its failure with newCompletionError (put/distributed.go#L557-L562) and gets INCOMPLETE.
Test below, three nodes, one of them refuses to store, PUT with node-side encoding:
| Policy | Stored | Status | Object ID in the reply |
|---|---|---|---|
EC 2/1 |
2 of 3 parts | 1024 internal server error | none |
REP 3 |
2 of 3 replicas | 1 incomplete | yes |
status=1024 msg="incomplete object PUT by placement: only 2 EC parts were put successfully for 2/1 EC rule, latest error: save on SN #2: …"
In the EC case two parts of a 2/1 rule are stored, so the object is complete enough to be read. The client is told the PUT failed and has no ID. A retry stores a second object; the parts of the first stay.
Possible Solution
Wrap the EC failure with newCompletionError(err, <any part stored>) as the REP branch does, so that Close returns the ID with INCOMPLETE. The existing test for this error checks only the message.
Steps to Reproduce (for bugs)
- Save the test below as
pkg/services/object/put/ecaudit_test.go. go test ./pkg/services/object/put/ -run TestECAuditH6 -v
It passes, that is, it asserts status 1024 and a zero ID for EC, and INCOMPLETE with an ID for REP.
ecaudit_test.go
package putsvc
import (
"context"
"errors"
"fmt"
"testing"
iec "github.com/nspcc-dev/neofs-node/internal/ec"
"github.com/nspcc-dev/neofs-node/internal/testutil"
"github.com/nspcc-dev/neofs-node/pkg/services/object/common"
objutil "github.com/nspcc-dev/neofs-node/pkg/services/object/util"
svcutil "github.com/nspcc-dev/neofs-node/pkg/services/util"
apistatus "github.com/nspcc-dev/neofs-sdk-go/client/status"
"github.com/nspcc-dev/neofs-sdk-go/container"
cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
sessionv2 "github.com/nspcc-dev/neofs-sdk-go/session/v2"
"github.com/nspcc-dev/neofs-sdk-go/user"
usertest "github.com/nspcc-dev/neofs-sdk-go/user/test"
"github.com/stretchr/testify/require"
)
func ecAuditECContainer(rules []iec.Rule, repRules []uint32, ip *netmap.InitialPlacementPolicy) container.Container {
var policy netmap.PlacementPolicy
ecRules := make([]netmap.ECRule, len(rules))
for i := range rules {
ecRules[i] = netmap.NewECRule(uint32(rules[i].DataPartNum), uint32(rules[i].ParityPartNum))
}
policy.SetECRules(ecRules)
if len(repRules) > 0 {
rds := make([]netmap.ReplicaDescriptor, len(repRules))
for i := range repRules {
rds[i].SetNumberOfObjects(repRules[i])
}
policy.SetReplicas(rds)
}
if ip != nil {
policy.SetInitial(*ip)
}
var cnr container.Container
cnr.SetPlacementPolicy(policy)
return cnr
}
// EC-only cluster, all rules share one node list of given size.
func ecAuditECCluster(t *testing.T, rules []iec.Rule, nodes uint, ip *netmap.InitialPlacementPolicy) *testCluster {
cluster := newTestClusterForRepPolicyWithContainer(t, nodes, 0, 0, ecAuditECContainer(rules, nil, ip))
for i := range cluster.nodeNetworks {
cn := &cluster.nodeNetworks[i].cnrNodes
cn.repCounts = nil
for range len(rules) - 1 {
cn.unsorted = append(cn.unsorted, cn.unsorted[0])
cn.sorted = append(cn.sorted, cn.sorted[0])
}
cn.ecRules = rules
}
return cluster
}
func ecAuditPut(svc *Service, obj object.Object, st2 *sessionv2.Token, ttl uint32) (oid.ID, error) {
stream, err := svc.Put(context.Background())
if err != nil {
return oid.ID{}, err
}
commonPrm := objutil.CommonPrmFromRequest(ttl, nil, common.RequestTokens{Session: st2})
if err = stream.Init(new(PutInitPrm).WithObject(obj.CutPayload()).WithCommonPrm(commonPrm)); err != nil {
return oid.ID{}, fmt.Errorf("init: %w", err)
}
if err = stream.SendChunk(new(PutChunkPrm).WithChunk(obj.Payload())); err != nil {
return oid.ID{}, fmt.Errorf("chunk: %w", err)
}
return stream.Close()
}
func ecAuditTokenForNode(t *testing.T, cluster *testCluster, nodeIdx int, owner usertest.UserSigner) *sessionv2.Token {
tok := newSessionTokenV2(t, cidtest.ID(), owner, nil, []sessionv2.Verb{sessionv2.VerbObjectPut})
pk := cluster.nodeSessions[nodeIdx].signer.ECDSAPrivateKey.PublicKey
require.NoError(t, tok.SetSubjects([]sessionv2.Target{sessionv2.NewTargetUser(user.NewFromECDSAPublicKey(pk))}))
require.NoError(t, tok.Sign(owner))
return tok
}
func ecAuditSrcObject(owner usertest.UserSigner, ln int) object.Object {
var obj object.Object
obj.SetContainerID(cidtest.ID())
obj.SetOwner(owner.UserID())
obj.SetAttributes(object.NewAttribute("attr1", "val1"))
obj.SetPayload(testutil.RandByteSlice(ln))
return obj
}
func TestECAuditH6_PartialECPut(t *testing.T) {
owner := usertest.User()
t.Run("EC 2/1, one of three nodes fails", func(t *testing.T) {
rules := []iec.Rule{{DataPartNum: 2, ParityPartNum: 1}}
cluster := ecAuditECCluster(t, rules, 3, nil)
cluster.nodeLocalStorages[2].err = errors.New("cannot store object")
id, err := ecAuditPut(cluster.nodeServices[0].svc, ecAuditSrcObject(owner, 100), ecAuditTokenForNode(t, cluster, 0, owner), 2)
require.Error(t, err)
require.NotErrorIs(t, err, apistatus.ErrIncomplete)
require.True(t, id.IsZero())
// same wrapping and conversion as putStream.close + Server.sendPutResponse
st := svcutil.ToStatus(fmt.Errorf("close put stream: %w", err))
require.EqualValues(t, 1024, st.GetCode())
var stored int
for i := range cluster.nodeLocalStorages {
stored += len(cluster.nodeLocalStorages[i].objs)
}
require.Equal(t, 2, stored) // == DataPartNum, i.e. object is fully recoverable
t.Logf("H6/EC: id=%s status=%d msg=%q, parts stored cluster-wide=%d", id, st.GetCode(), st.GetMessage(), stored)
})
t.Run("REP 3, one of three nodes fails (contrast)", func(t *testing.T) {
cluster := newTestClusterForRepPolicy(t, 3, 0, 0)
cluster.nodeLocalStorages[2].err = errors.New("cannot store object")
id, err := ecAuditPut(cluster.nodeServices[0].svc, ecAuditSrcObject(owner, 100), ecAuditTokenForNode(t, cluster, 0, owner), 2)
require.ErrorIs(t, err, apistatus.ErrIncomplete)
require.False(t, id.IsZero())
st := svcutil.ToStatus(fmt.Errorf("close put stream: %w", err))
t.Logf("H6/REP: id=%s status=%d", id, st.GetCode())
})
}
Context
Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.
Regression
Not known.
Your Environment
- Version used: 0.58.2 (
1f0d51b5d) - Server setup and configuration: none, unit tests only
- Operating System and version (
uname -a): Darwin 25.6.0 arm64, Go 1.27.3
- Lingua principale
- Go
- Stelle
- 37
- Fork
- 56
- Merge medio
- 1g 18h
- PR unite (30g)
- 42
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di nspcc-dev/neofs-node
-
bug hal:proposed neofs-storage
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
nspcc-dev/neofs-node#4325 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
nspcc-dev/neofs-node#4311 ·
I maintainer di solito rispondono entro 1 giorno
-
Check alpha scope in IRApertahal:proposed I4 neofs-storage S3 security U4
Difficoltà 2/5 1-3 ore Idoneità per principianti 69/100
nspcc-dev/neofs-node#4280 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
bug neofs-storage
Difficoltà 3/5 1-2 giorni Idoneità per principianti 68/100
nspcc-dev/neofs-node#4333 ·
I maintainer di solito rispondono entro 1 giorno
-
bug neofs-storage
Difficoltà 3/5 1-2 giorni Idoneità per principianti 65/100
nspcc-dev/neofs-node#4332 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di nspcc-dev/neofs-node
Issue simili
-
Pin and verify Security Insights schema in lintinsights targetForse già presa @imshubham22apr-gif l’ha presa 5 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
gemaraproj/gemara#618 ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
a2aproject/a2a-go#597 ·
I maintainer di solito rispondono entro 1 giorno
-
scopes: GetAllowedTopics, GetProtectedTopics, and GetScopedTopics include empty topic names and do not trim whitespace charactersForse già presa @patelshivani2283-lab l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
area/networking kind/bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 69/100
siderolabs/talos#14972 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno