EC Audit: H6: partly stored EC PUT is answered with "internal server error" and no object ID
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 新手友好度
- 68/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- go
调研方向
Start in pkg/services/object/put/distributed.go, especially handleECRule, Close, and the REP branch that uses newCompletionError. Compare how EC and REP failures are classified when some parts or replicas are stored, then add or update a focused test in pkg/services/object/put/ for the EC case. Done means a partial EC PUT returns INCOMPLETE with the object ID, consistent with the expected behavior in the issue.
由索引模型根据 Issue 内容生成。
描述
Expected Behavior
When a PUT into an EC container stores some parts but not all, the client gets status INCOMPLETE and the object ID, as it does for a REP container. It can then tell "nothing was stored" from "stored but under-replicated", and can read or delete the object.
Current Behavior
The EC path returns errIncompletePut, which has neither Is nor Unwrap (put/distributed.go#L108-L121), and handleECRule passes it on as is (put/distributed.go#L468-L474). Close returns the object ID only for errors that match apistatus.ErrIncomplete (put/distributed.go#L305-L311), so this one comes back with a zero ID and is converted to status 1024. The REP path wraps its failure with newCompletionError (put/distributed.go#L557-L562) and gets INCOMPLETE.
Test below, three nodes, one of them refuses to store, PUT with node-side encoding:
| Policy | Stored | Status | Object ID in the reply |
|---|---|---|---|
EC 2/1 |
2 of 3 parts | 1024 internal server error | none |
REP 3 |
2 of 3 replicas | 1 incomplete | yes |
status=1024 msg="incomplete object PUT by placement: only 2 EC parts were put successfully for 2/1 EC rule, latest error: save on SN #2: …"
In the EC case two parts of a 2/1 rule are stored, so the object is complete enough to be read. The client is told the PUT failed and has no ID. A retry stores a second object; the parts of the first stay.
Possible Solution
Wrap the EC failure with newCompletionError(err, <any part stored>) as the REP branch does, so that Close returns the ID with INCOMPLETE. The existing test for this error checks only the message.
Steps to Reproduce (for bugs)
- Save the test below as
pkg/services/object/put/ecaudit_test.go. go test ./pkg/services/object/put/ -run TestECAuditH6 -v
It passes, that is, it asserts status 1024 and a zero ID for EC, and INCOMPLETE with an ID for REP.
ecaudit_test.go
package putsvc
import (
"context"
"errors"
"fmt"
"testing"
iec "github.com/nspcc-dev/neofs-node/internal/ec"
"github.com/nspcc-dev/neofs-node/internal/testutil"
"github.com/nspcc-dev/neofs-node/pkg/services/object/common"
objutil "github.com/nspcc-dev/neofs-node/pkg/services/object/util"
svcutil "github.com/nspcc-dev/neofs-node/pkg/services/util"
apistatus "github.com/nspcc-dev/neofs-sdk-go/client/status"
"github.com/nspcc-dev/neofs-sdk-go/container"
cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
sessionv2 "github.com/nspcc-dev/neofs-sdk-go/session/v2"
"github.com/nspcc-dev/neofs-sdk-go/user"
usertest "github.com/nspcc-dev/neofs-sdk-go/user/test"
"github.com/stretchr/testify/require"
)
func ecAuditECContainer(rules []iec.Rule, repRules []uint32, ip *netmap.InitialPlacementPolicy) container.Container {
var policy netmap.PlacementPolicy
ecRules := make([]netmap.ECRule, len(rules))
for i := range rules {
ecRules[i] = netmap.NewECRule(uint32(rules[i].DataPartNum), uint32(rules[i].ParityPartNum))
}
policy.SetECRules(ecRules)
if len(repRules) > 0 {
rds := make([]netmap.ReplicaDescriptor, len(repRules))
for i := range repRules {
rds[i].SetNumberOfObjects(repRules[i])
}
policy.SetReplicas(rds)
}
if ip != nil {
policy.SetInitial(*ip)
}
var cnr container.Container
cnr.SetPlacementPolicy(policy)
return cnr
}
// EC-only cluster, all rules share one node list of given size.
func ecAuditECCluster(t *testing.T, rules []iec.Rule, nodes uint, ip *netmap.InitialPlacementPolicy) *testCluster {
cluster := newTestClusterForRepPolicyWithContainer(t, nodes, 0, 0, ecAuditECContainer(rules, nil, ip))
for i := range cluster.nodeNetworks {
cn := &cluster.nodeNetworks[i].cnrNodes
cn.repCounts = nil
for range len(rules) - 1 {
cn.unsorted = append(cn.unsorted, cn.unsorted[0])
cn.sorted = append(cn.sorted, cn.sorted[0])
}
cn.ecRules = rules
}
return cluster
}
func ecAuditPut(svc *Service, obj object.Object, st2 *sessionv2.Token, ttl uint32) (oid.ID, error) {
stream, err := svc.Put(context.Background())
if err != nil {
return oid.ID{}, err
}
commonPrm := objutil.CommonPrmFromRequest(ttl, nil, common.RequestTokens{Session: st2})
if err = stream.Init(new(PutInitPrm).WithObject(obj.CutPayload()).WithCommonPrm(commonPrm)); err != nil {
return oid.ID{}, fmt.Errorf("init: %w", err)
}
if err = stream.SendChunk(new(PutChunkPrm).WithChunk(obj.Payload())); err != nil {
return oid.ID{}, fmt.Errorf("chunk: %w", err)
}
return stream.Close()
}
func ecAuditTokenForNode(t *testing.T, cluster *testCluster, nodeIdx int, owner usertest.UserSigner) *sessionv2.Token {
tok := newSessionTokenV2(t, cidtest.ID(), owner, nil, []sessionv2.Verb{sessionv2.VerbObjectPut})
pk := cluster.nodeSessions[nodeIdx].signer.ECDSAPrivateKey.PublicKey
require.NoError(t, tok.SetSubjects([]sessionv2.Target{sessionv2.NewTargetUser(user.NewFromECDSAPublicKey(pk))}))
require.NoError(t, tok.Sign(owner))
return tok
}
func ecAuditSrcObject(owner usertest.UserSigner, ln int) object.Object {
var obj object.Object
obj.SetContainerID(cidtest.ID())
obj.SetOwner(owner.UserID())
obj.SetAttributes(object.NewAttribute("attr1", "val1"))
obj.SetPayload(testutil.RandByteSlice(ln))
return obj
}
func TestECAuditH6_PartialECPut(t *testing.T) {
owner := usertest.User()
t.Run("EC 2/1, one of three nodes fails", func(t *testing.T) {
rules := []iec.Rule{{DataPartNum: 2, ParityPartNum: 1}}
cluster := ecAuditECCluster(t, rules, 3, nil)
cluster.nodeLocalStorages[2].err = errors.New("cannot store object")
id, err := ecAuditPut(cluster.nodeServices[0].svc, ecAuditSrcObject(owner, 100), ecAuditTokenForNode(t, cluster, 0, owner), 2)
require.Error(t, err)
require.NotErrorIs(t, err, apistatus.ErrIncomplete)
require.True(t, id.IsZero())
// same wrapping and conversion as putStream.close + Server.sendPutResponse
st := svcutil.ToStatus(fmt.Errorf("close put stream: %w", err))
require.EqualValues(t, 1024, st.GetCode())
var stored int
for i := range cluster.nodeLocalStorages {
stored += len(cluster.nodeLocalStorages[i].objs)
}
require.Equal(t, 2, stored) // == DataPartNum, i.e. object is fully recoverable
t.Logf("H6/EC: id=%s status=%d msg=%q, parts stored cluster-wide=%d", id, st.GetCode(), st.GetMessage(), stored)
})
t.Run("REP 3, one of three nodes fails (contrast)", func(t *testing.T) {
cluster := newTestClusterForRepPolicy(t, 3, 0, 0)
cluster.nodeLocalStorages[2].err = errors.New("cannot store object")
id, err := ecAuditPut(cluster.nodeServices[0].svc, ecAuditSrcObject(owner, 100), ecAuditTokenForNode(t, cluster, 0, owner), 2)
require.ErrorIs(t, err, apistatus.ErrIncomplete)
require.False(t, id.IsZero())
st := svcutil.ToStatus(fmt.Errorf("close put stream: %w", err))
t.Logf("H6/REP: id=%s status=%d", id, st.GetCode())
})
}
Context
Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.
Regression
Not known.
Your Environment
- Version used: 0.58.2 (
1f0d51b5d) - Server setup and configuration: none, unit tests only
- Operating System and version (
uname -a): Darwin 25.6.0 arm64, Go 1.27.3
- 主要语言
- Go
- 星标
- 37
- 派生
- 56
- 平均合并
- 1 天 18 小时
- 30 天内合并 PR
- 42
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
nspcc-dev/neofs-node 的其他 Issue
-
bug hal:proposed neofs-storage
难度 2/5 1-3 小时 新手友好度 78/100
nspcc-dev/neofs-node#4325 · 3 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
nspcc-dev/neofs-node#4311 ·
维护者通常 1 天内回复
-
hal:proposed I4 neofs-storage S3 security U4
难度 2/5 1-3 小时 新手友好度 69/100
nspcc-dev/neofs-node#4280 · 2 条评论 ·
维护者通常 1 天内回复
-
bug neofs-storage
难度 3/5 1-2 天 新手友好度 68/100
nspcc-dev/neofs-node#4333 ·
维护者通常 1 天内回复
-
bug neofs-storage
难度 3/5 1-2 天 新手友好度 65/100
nspcc-dev/neofs-node#4332 ·
维护者通常 1 天内回复
查看 nspcc-dev/neofs-node 的全部 Issue
相似的 Issue
-
Pin and verify Security Insights schema in lintinsights target可能已有人在做 @imshubham22apr-gif 于 5 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
gemaraproj/gemara#618 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 64/100
a2aproject/a2a-go#597 ·
维护者通常 1 天内回复
-
scopes: GetAllowedTopics, GetProtectedTopics, and GetScopedTopics include empty topic names and do not trim whitespace characters可能已有人在做 @patelshivani2283-lab 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
area/networking kind/bug
难度 2/5 1-3 小时 新手友好度 69/100
siderolabs/talos#14972 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 72/100
维护者通常 1 天内回复