EC Audit: H6: partly stored EC PUT is answered with "internal server error" and no object ID
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 68/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- go
- Área
- backend, distributed-systems
Línea de trabajo
Start in pkg/services/object/put/distributed.go, especially handleECRule, Close, and the REP branch that uses newCompletionError. Compare how EC and REP failures are classified when some parts or replicas are stored, then add or update a focused test in pkg/services/object/put/ for the EC case. Done means a partial EC PUT returns INCOMPLETE with the object ID, consistent with the expected behavior in the issue.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Expected Behavior
When a PUT into an EC container stores some parts but not all, the client gets status INCOMPLETE and the object ID, as it does for a REP container. It can then tell "nothing was stored" from "stored but under-replicated", and can read or delete the object.
Current Behavior
The EC path returns errIncompletePut, which has neither Is nor Unwrap (put/distributed.go#L108-L121), and handleECRule passes it on as is (put/distributed.go#L468-L474). Close returns the object ID only for errors that match apistatus.ErrIncomplete (put/distributed.go#L305-L311), so this one comes back with a zero ID and is converted to status 1024. The REP path wraps its failure with newCompletionError (put/distributed.go#L557-L562) and gets INCOMPLETE.
Test below, three nodes, one of them refuses to store, PUT with node-side encoding:
| Policy | Stored | Status | Object ID in the reply |
|---|---|---|---|
EC 2/1 |
2 of 3 parts | 1024 internal server error | none |
REP 3 |
2 of 3 replicas | 1 incomplete | yes |
status=1024 msg="incomplete object PUT by placement: only 2 EC parts were put successfully for 2/1 EC rule, latest error: save on SN #2: …"
In the EC case two parts of a 2/1 rule are stored, so the object is complete enough to be read. The client is told the PUT failed and has no ID. A retry stores a second object; the parts of the first stay.
Possible Solution
Wrap the EC failure with newCompletionError(err, <any part stored>) as the REP branch does, so that Close returns the ID with INCOMPLETE. The existing test for this error checks only the message.
Steps to Reproduce (for bugs)
- Save the test below as
pkg/services/object/put/ecaudit_test.go. go test ./pkg/services/object/put/ -run TestECAuditH6 -v
It passes, that is, it asserts status 1024 and a zero ID for EC, and INCOMPLETE with an ID for REP.
ecaudit_test.go
package putsvc
import (
"context"
"errors"
"fmt"
"testing"
iec "github.com/nspcc-dev/neofs-node/internal/ec"
"github.com/nspcc-dev/neofs-node/internal/testutil"
"github.com/nspcc-dev/neofs-node/pkg/services/object/common"
objutil "github.com/nspcc-dev/neofs-node/pkg/services/object/util"
svcutil "github.com/nspcc-dev/neofs-node/pkg/services/util"
apistatus "github.com/nspcc-dev/neofs-sdk-go/client/status"
"github.com/nspcc-dev/neofs-sdk-go/container"
cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
sessionv2 "github.com/nspcc-dev/neofs-sdk-go/session/v2"
"github.com/nspcc-dev/neofs-sdk-go/user"
usertest "github.com/nspcc-dev/neofs-sdk-go/user/test"
"github.com/stretchr/testify/require"
)
func ecAuditECContainer(rules []iec.Rule, repRules []uint32, ip *netmap.InitialPlacementPolicy) container.Container {
var policy netmap.PlacementPolicy
ecRules := make([]netmap.ECRule, len(rules))
for i := range rules {
ecRules[i] = netmap.NewECRule(uint32(rules[i].DataPartNum), uint32(rules[i].ParityPartNum))
}
policy.SetECRules(ecRules)
if len(repRules) > 0 {
rds := make([]netmap.ReplicaDescriptor, len(repRules))
for i := range repRules {
rds[i].SetNumberOfObjects(repRules[i])
}
policy.SetReplicas(rds)
}
if ip != nil {
policy.SetInitial(*ip)
}
var cnr container.Container
cnr.SetPlacementPolicy(policy)
return cnr
}
// EC-only cluster, all rules share one node list of given size.
func ecAuditECCluster(t *testing.T, rules []iec.Rule, nodes uint, ip *netmap.InitialPlacementPolicy) *testCluster {
cluster := newTestClusterForRepPolicyWithContainer(t, nodes, 0, 0, ecAuditECContainer(rules, nil, ip))
for i := range cluster.nodeNetworks {
cn := &cluster.nodeNetworks[i].cnrNodes
cn.repCounts = nil
for range len(rules) - 1 {
cn.unsorted = append(cn.unsorted, cn.unsorted[0])
cn.sorted = append(cn.sorted, cn.sorted[0])
}
cn.ecRules = rules
}
return cluster
}
func ecAuditPut(svc *Service, obj object.Object, st2 *sessionv2.Token, ttl uint32) (oid.ID, error) {
stream, err := svc.Put(context.Background())
if err != nil {
return oid.ID{}, err
}
commonPrm := objutil.CommonPrmFromRequest(ttl, nil, common.RequestTokens{Session: st2})
if err = stream.Init(new(PutInitPrm).WithObject(obj.CutPayload()).WithCommonPrm(commonPrm)); err != nil {
return oid.ID{}, fmt.Errorf("init: %w", err)
}
if err = stream.SendChunk(new(PutChunkPrm).WithChunk(obj.Payload())); err != nil {
return oid.ID{}, fmt.Errorf("chunk: %w", err)
}
return stream.Close()
}
func ecAuditTokenForNode(t *testing.T, cluster *testCluster, nodeIdx int, owner usertest.UserSigner) *sessionv2.Token {
tok := newSessionTokenV2(t, cidtest.ID(), owner, nil, []sessionv2.Verb{sessionv2.VerbObjectPut})
pk := cluster.nodeSessions[nodeIdx].signer.ECDSAPrivateKey.PublicKey
require.NoError(t, tok.SetSubjects([]sessionv2.Target{sessionv2.NewTargetUser(user.NewFromECDSAPublicKey(pk))}))
require.NoError(t, tok.Sign(owner))
return tok
}
func ecAuditSrcObject(owner usertest.UserSigner, ln int) object.Object {
var obj object.Object
obj.SetContainerID(cidtest.ID())
obj.SetOwner(owner.UserID())
obj.SetAttributes(object.NewAttribute("attr1", "val1"))
obj.SetPayload(testutil.RandByteSlice(ln))
return obj
}
func TestECAuditH6_PartialECPut(t *testing.T) {
owner := usertest.User()
t.Run("EC 2/1, one of three nodes fails", func(t *testing.T) {
rules := []iec.Rule{{DataPartNum: 2, ParityPartNum: 1}}
cluster := ecAuditECCluster(t, rules, 3, nil)
cluster.nodeLocalStorages[2].err = errors.New("cannot store object")
id, err := ecAuditPut(cluster.nodeServices[0].svc, ecAuditSrcObject(owner, 100), ecAuditTokenForNode(t, cluster, 0, owner), 2)
require.Error(t, err)
require.NotErrorIs(t, err, apistatus.ErrIncomplete)
require.True(t, id.IsZero())
// same wrapping and conversion as putStream.close + Server.sendPutResponse
st := svcutil.ToStatus(fmt.Errorf("close put stream: %w", err))
require.EqualValues(t, 1024, st.GetCode())
var stored int
for i := range cluster.nodeLocalStorages {
stored += len(cluster.nodeLocalStorages[i].objs)
}
require.Equal(t, 2, stored) // == DataPartNum, i.e. object is fully recoverable
t.Logf("H6/EC: id=%s status=%d msg=%q, parts stored cluster-wide=%d", id, st.GetCode(), st.GetMessage(), stored)
})
t.Run("REP 3, one of three nodes fails (contrast)", func(t *testing.T) {
cluster := newTestClusterForRepPolicy(t, 3, 0, 0)
cluster.nodeLocalStorages[2].err = errors.New("cannot store object")
id, err := ecAuditPut(cluster.nodeServices[0].svc, ecAuditSrcObject(owner, 100), ecAuditTokenForNode(t, cluster, 0, owner), 2)
require.ErrorIs(t, err, apistatus.ErrIncomplete)
require.False(t, id.IsZero())
st := svcutil.ToStatus(fmt.Errorf("close put stream: %w", err))
t.Logf("H6/REP: id=%s status=%d", id, st.GetCode())
})
}
Context
Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.
Regression
Not known.
Your Environment
- Version used: 0.58.2 (
1f0d51b5d) - Server setup and configuration: none, unit tests only
- Operating System and version (
uname -a): Darwin 25.6.0 arm64, Go 1.27.3
- Lenguaje dominante
- Go
- Estrellas
- 37
- Forks
- 56
- Merge medio
- 1 d 18 h
- PR fusionados (30 d)
- 42
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de nspcc-dev/neofs-node
-
bug hal:proposed neofs-storage
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
nspcc-dev/neofs-node#4325 · 3 comentarios ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
nspcc-dev/neofs-node#4311 ·
Los mantenedores suelen responder en 1 día
-
Check alpha scope in IRAbiertohal:proposed I4 neofs-storage S3 security U4
Dificultad 2/5 1-3 horas Aptitud para principiantes 69/100
nspcc-dev/neofs-node#4280 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
bug neofs-storage
Dificultad 3/5 1-2 días Aptitud para principiantes 68/100
nspcc-dev/neofs-node#4333 ·
Los mantenedores suelen responder en 1 día
-
bug neofs-storage
Dificultad 3/5 1-2 días Aptitud para principiantes 65/100
nspcc-dev/neofs-node#4332 ·
Los mantenedores suelen responder en 1 día
Todos los issues de nspcc-dev/neofs-node
Issues similares
-
Pin and verify Security Insights schema in lintinsights targetPosiblemente ocupada @imshubham22apr-gif la tomó hace 5 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
gemaraproj/gemara#618 ·
Los mantenedores suelen responder en 2 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 64/100
a2aproject/a2a-go#597 ·
Los mantenedores suelen responder en 1 día
-
scopes: GetAllowedTopics, GetProtectedTopics, and GetScopedTopics include empty topic names and do not trim whitespace charactersPosiblemente ocupada @patelshivani2283-lab la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
area/networking kind/bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 69/100
siderolabs/talos#14972 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día