EC Audit: H4: policer rebuilds wrong EC parts when it restores a rule from another rule
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 55/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- go
調査の方向性
Start in pkg/services/policer/ec.go at fillParityParts and sliceECParts, then trace the recursive restore path around the lines linked in the issue. Run the supplied TestECAuditH4 probe with go test ./pkg/services/policer/ -run TestECAuditH4 -v; done means restored parts match the originals for both payload sizes and the recursive path handles the source rule's padding correctly.
索引モデルが issue の本文から書いたものです。
説明
Expected Behavior
When a rule has lost more parts than its parity covers and the container has another EC rule, the policer rebuilds the lost parts from the other rule. The rebuilt parts are byte-identical to the original ones.
Current Behavior
The policer fetches all parts of the other rule into one buffer, data and parity, and then cuts that buffer into parts of the rule being restored (policer/ec.go#L344-L352, fillParityParts → sliceECParts at #L535-L562). The data area of a rule is d * ceil(len/d) bytes: the payload followed by zero padding. This length differs between rules. When the restored rule's data area is longer than the source rule's, the end of its last data part is taken from the source rule's first parity part instead of zeros. Parity is then computed over that.
Test below, rules 3/2 (restored) and 2/2 (source), parts 2, 3, 4 of the first rule lost:
| Payload | Data area 3/2 | Data area 2/2 | Recreated parts |
|---|---|---|---|
| 1002 bytes | 1002 | 1002 | identical to the originals |
| 1000 bytes | 1002 | 1000 | all three wrong, 2 of 334 bytes differ in each |
The recreated objects have other OIDs than the originals and do not match the parent's __NEOFS__EC_PART_HASHES. They fail the EC part check of the node they are sent to (checkECParent), so the policer repeats the same attempt every pass and the rule stays degraded. That last step is from reading the code; the test replaces the replicator with a mock.
The same re-slicing is done a second time on the recursive path, when the source rule is itself short of parts and is restored from a third rule (policer/ec.go#L511-L524). There obj = obj[:needLen] also extends the slice over whatever the buffer held before.
Possible Solution
Take only the payload (parentHeader.PayloadSize() bytes) from the source rule and encode it with the target rule, as PUT does, instead of re-cutting the source rule's parts.
Steps to Reproduce (for bugs)
- Save the test below as
pkg/services/policer/ecaudit_test.go. go test ./pkg/services/policer/ -run TestECAuditH4 -v
payload 1002: part 2: recreated, identical to the original (same for 3, 4)
payload 1000: part 2: recreated with WRONG payload (2 of 334 bytes differ), OID … != original … (same for 3, 4)
ecaudit_test.go
package policer
import (
"bytes"
"context"
"strconv"
"sync"
"testing"
"testing/synctest"
"time"
iec "github.com/nspcc-dev/neofs-node/internal/ec"
"github.com/nspcc-dev/neofs-node/internal/testutil"
objectcore "github.com/nspcc-dev/neofs-node/pkg/core/object"
"github.com/nspcc-dev/neofs-node/pkg/services/replicator"
apistatus "github.com/nspcc-dev/neofs-sdk-go/client/status"
neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
"github.com/stretchr/testify/require"
"go.uber.org/zap/zaptest"
)
// runs one policer pass over the local part #0 of rule #0. Parts of rule #0
// listed in lost are missing on their nodes, all other parts of all rules are
// healthy. Returns recreated objects by part index and the original chain.
func ecAuditRepair(t *testing.T, ecRules []iec.Rule, parentSize uint64, lost []int) (map[int]object.Object, [][]object.Object) {
var nodesPlacement [][]netmap.NodeInfo
for _, r := range ecRules {
nodesPlacement = append(nodesPlacement, testutil.Nodes(int(r.DataPartNum+r.ParityPartNum)))
}
// testutil.Nodes is deterministic, make nodes of different rules distinct
all := testutil.Nodes(64)
off := 0
for i := range nodesPlacement {
nodesPlacement[i] = all[off : off+len(nodesPlacement[i])]
off += len(nodesPlacement[i])
}
parent, ecChain := generateECChain(t, parentSize, ecRules)
parentAddr := parent.Address()
parentOID := parentAddr.Object()
localPart := ecChain[0][0]
localObj := objectcore.AddressWithAttributes{
Address: oid.NewAddress(parentAddr.Container(), localPart.GetID()),
Type: object.TypeRegular,
Attributes: []string{"0", "0", string(parentOID[:]),
strconv.FormatUint(localPart.PayloadSize(), 10)},
}
localNode := newTestLocalNode()
localNode.objList = []objectcore.AddressWithAttributes{localObj}
localNode.obj = localPart
localNode.setECPartResult(t, localPart)
mockNet := newMockNetwork()
mockNet.setObjectNodesECResult(parentAddr.Container(), parentOID, nodesPlacement, ecRules)
mockNet.pubKey = nodesPlacement[0][0].PublicKey()
conns := newMockAPIConnections()
isLost := func(i int) bool {
for _, l := range lost {
if l == i {
return true
}
}
return false
}
for ruleIdx := range ecRules {
for i, obj := range ecChain[ruleIdx] {
n := nodesPlacement[ruleIdx][i]
if ruleIdx == 0 && isLost(i) {
conns.setHeadResult(n, parentAddr, object.Object{}, error(apistatus.ErrObjectNotFound))
conns.setRangeResult(n, parentAddr, uint8(ruleIdx), uint8(i), nil, apistatus.ErrObjectNotFound)
continue
}
conns.setHeadResult(n, parentAddr, obj, nil)
conns.setRangeResult(n, parentAddr, uint8(ruleIdx), uint8(i), obj.Payload(), nil)
}
}
r := newTestReplicator(t)
r.success = true
var m sync.Mutex
restored := make(map[int]object.Object)
r.onTask = func(task replicator.Task, _ []netmap.NodeInfo) {
obj := task.Object()
pi, err := iec.GetPartInfo(*obj)
if err != nil || pi.RuleIndex != 0 {
return
}
m.Lock()
restored[pi.Index] = *obj
m.Unlock()
}
p := New(neofscryptotest.Signer(),
WithReplicationCooldown(50*time.Millisecond),
WithNetwork(mockNet),
WithLogger(zaptest.NewLogger(t)),
WithMetrics(&mockMetrics{}),
)
p.localStorage = localNode
p.apiConns = conns
p.replicator = r
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
go p.Run(ctx)
time.Sleep(time.Second)
synctest.Wait()
})
m.Lock()
defer m.Unlock()
return restored, ecChain
}
func ecAuditReport(t *testing.T, restored map[int]object.Object, orig []object.Object, lost []int) (wrong int) {
for _, i := range lost {
got, ok := restored[i]
if !ok {
t.Logf("part %d: NOT recreated", i)
continue
}
if bytes.Equal(got.Payload(), orig[i].Payload()) && got.GetID() == orig[i].GetID() {
t.Logf("part %d: recreated, identical to the original", i)
continue
}
wrong++
diff := 0
for j := range got.Payload() {
if got.Payload()[j] != orig[i].Payload()[j] {
diff++
}
}
t.Logf("part %d: recreated with WRONG payload (%d of %d bytes differ), OID %s != original %s",
i, diff, len(got.Payload()), got.GetID(), orig[i].GetID())
}
return wrong
}
// H4: rule #0 lost more than its parity, restored from rule #1.
func TestECAuditH4_CrossRuleRestore(t *testing.T) {
rules := []iec.Rule{{DataPartNum: 3, ParityPartNum: 2}, {DataPartNum: 2, ParityPartNum: 2}}
lost := []int{2, 3, 4}
t.Run("payload 1002 (3*334 == 2*501)", func(t *testing.T) {
restored, chain := ecAuditRepair(t, rules, 1002, lost)
require.Zero(t, ecAuditReport(t, restored, chain[0], lost))
})
t.Run("payload 1000 (3*334 > 2*500)", func(t *testing.T) {
restored, chain := ecAuditRepair(t, rules, 1000, lost)
require.NotZero(t, ecAuditReport(t, restored, chain[0], lost))
})
}
Context
Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.
Regression
Not known.
Your Environment
- Version used: 0.58.2 (
1f0d51b5d) - Server setup and configuration: none, unit tests only
- Operating System and version (
uname -a): Darwin 25.6.0 arm64, Go 1.27.3
- 主要言語
- Go
- スター
- 37
- フォーク
- 56
- 平均マージ
- 1日 21時間
- マージ済み PR(30日)
- 46
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
nspcc-dev/neofs-node のほかの issue
-
bug hal:proposed neofs-storage
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
nspcc-dev/neofs-node#4325 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
-
hal:proposed I4 neofs-storage S3 security U4
難易度 2/5 1〜3時間 初心者へのやさしさ 69/100
nspcc-dev/neofs-node#4280 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 45/100
nspcc-dev/neofs-node#4339 ·
メンテナーはふだん 1 日以内に返信
-
enhancement I3 neofs-storage performance S3 U3
難易度 5/5 1週間以上 初心者へのやさしさ 20/100
nspcc-dev/neofs-node#4337 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
bug neofs-storage
難易度 4/5 3〜5日 初心者へのやさしさ 45/100
nspcc-dev/neofs-node#4336 ·
メンテナーはふだん 1 日以内に返信
nspcc-dev/neofs-node の issue をすべて見る
似ている issue
-
難易度 1/5 1時間未満 初心者へのやさしさ 86/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 64/100
projectdiscovery/cdncheck#554 ·
メンテナーはふだん 1 日以内に返信
-
triage needed
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 78/100
OpenSlides/openslides-go#342 ·
メンテナーはふだん 4 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100