EC Audit: H4: policer rebuilds wrong EC parts when it restores a rule from another rule
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 55/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- go
- Área
- distributed-systems
Línea de trabajo
Start in pkg/services/policer/ec.go at fillParityParts and sliceECParts, then trace the recursive restore path around the lines linked in the issue. Run the supplied TestECAuditH4 probe with go test ./pkg/services/policer/ -run TestECAuditH4 -v; done means restored parts match the originals for both payload sizes and the recursive path handles the source rule's padding correctly.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Expected Behavior
When a rule has lost more parts than its parity covers and the container has another EC rule, the policer rebuilds the lost parts from the other rule. The rebuilt parts are byte-identical to the original ones.
Current Behavior
The policer fetches all parts of the other rule into one buffer, data and parity, and then cuts that buffer into parts of the rule being restored (policer/ec.go#L344-L352, fillParityParts → sliceECParts at #L535-L562). The data area of a rule is d * ceil(len/d) bytes: the payload followed by zero padding. This length differs between rules. When the restored rule's data area is longer than the source rule's, the end of its last data part is taken from the source rule's first parity part instead of zeros. Parity is then computed over that.
Test below, rules 3/2 (restored) and 2/2 (source), parts 2, 3, 4 of the first rule lost:
| Payload | Data area 3/2 | Data area 2/2 | Recreated parts |
|---|---|---|---|
| 1002 bytes | 1002 | 1002 | identical to the originals |
| 1000 bytes | 1002 | 1000 | all three wrong, 2 of 334 bytes differ in each |
The recreated objects have other OIDs than the originals and do not match the parent's __NEOFS__EC_PART_HASHES. They fail the EC part check of the node they are sent to (checkECParent), so the policer repeats the same attempt every pass and the rule stays degraded. That last step is from reading the code; the test replaces the replicator with a mock.
The same re-slicing is done a second time on the recursive path, when the source rule is itself short of parts and is restored from a third rule (policer/ec.go#L511-L524). There obj = obj[:needLen] also extends the slice over whatever the buffer held before.
Possible Solution
Take only the payload (parentHeader.PayloadSize() bytes) from the source rule and encode it with the target rule, as PUT does, instead of re-cutting the source rule's parts.
Steps to Reproduce (for bugs)
- Save the test below as
pkg/services/policer/ecaudit_test.go. go test ./pkg/services/policer/ -run TestECAuditH4 -v
payload 1002: part 2: recreated, identical to the original (same for 3, 4)
payload 1000: part 2: recreated with WRONG payload (2 of 334 bytes differ), OID … != original … (same for 3, 4)
ecaudit_test.go
package policer
import (
"bytes"
"context"
"strconv"
"sync"
"testing"
"testing/synctest"
"time"
iec "github.com/nspcc-dev/neofs-node/internal/ec"
"github.com/nspcc-dev/neofs-node/internal/testutil"
objectcore "github.com/nspcc-dev/neofs-node/pkg/core/object"
"github.com/nspcc-dev/neofs-node/pkg/services/replicator"
apistatus "github.com/nspcc-dev/neofs-sdk-go/client/status"
neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
"github.com/stretchr/testify/require"
"go.uber.org/zap/zaptest"
)
// runs one policer pass over the local part #0 of rule #0. Parts of rule #0
// listed in lost are missing on their nodes, all other parts of all rules are
// healthy. Returns recreated objects by part index and the original chain.
func ecAuditRepair(t *testing.T, ecRules []iec.Rule, parentSize uint64, lost []int) (map[int]object.Object, [][]object.Object) {
var nodesPlacement [][]netmap.NodeInfo
for _, r := range ecRules {
nodesPlacement = append(nodesPlacement, testutil.Nodes(int(r.DataPartNum+r.ParityPartNum)))
}
// testutil.Nodes is deterministic, make nodes of different rules distinct
all := testutil.Nodes(64)
off := 0
for i := range nodesPlacement {
nodesPlacement[i] = all[off : off+len(nodesPlacement[i])]
off += len(nodesPlacement[i])
}
parent, ecChain := generateECChain(t, parentSize, ecRules)
parentAddr := parent.Address()
parentOID := parentAddr.Object()
localPart := ecChain[0][0]
localObj := objectcore.AddressWithAttributes{
Address: oid.NewAddress(parentAddr.Container(), localPart.GetID()),
Type: object.TypeRegular,
Attributes: []string{"0", "0", string(parentOID[:]),
strconv.FormatUint(localPart.PayloadSize(), 10)},
}
localNode := newTestLocalNode()
localNode.objList = []objectcore.AddressWithAttributes{localObj}
localNode.obj = localPart
localNode.setECPartResult(t, localPart)
mockNet := newMockNetwork()
mockNet.setObjectNodesECResult(parentAddr.Container(), parentOID, nodesPlacement, ecRules)
mockNet.pubKey = nodesPlacement[0][0].PublicKey()
conns := newMockAPIConnections()
isLost := func(i int) bool {
for _, l := range lost {
if l == i {
return true
}
}
return false
}
for ruleIdx := range ecRules {
for i, obj := range ecChain[ruleIdx] {
n := nodesPlacement[ruleIdx][i]
if ruleIdx == 0 && isLost(i) {
conns.setHeadResult(n, parentAddr, object.Object{}, error(apistatus.ErrObjectNotFound))
conns.setRangeResult(n, parentAddr, uint8(ruleIdx), uint8(i), nil, apistatus.ErrObjectNotFound)
continue
}
conns.setHeadResult(n, parentAddr, obj, nil)
conns.setRangeResult(n, parentAddr, uint8(ruleIdx), uint8(i), obj.Payload(), nil)
}
}
r := newTestReplicator(t)
r.success = true
var m sync.Mutex
restored := make(map[int]object.Object)
r.onTask = func(task replicator.Task, _ []netmap.NodeInfo) {
obj := task.Object()
pi, err := iec.GetPartInfo(*obj)
if err != nil || pi.RuleIndex != 0 {
return
}
m.Lock()
restored[pi.Index] = *obj
m.Unlock()
}
p := New(neofscryptotest.Signer(),
WithReplicationCooldown(50*time.Millisecond),
WithNetwork(mockNet),
WithLogger(zaptest.NewLogger(t)),
WithMetrics(&mockMetrics{}),
)
p.localStorage = localNode
p.apiConns = conns
p.replicator = r
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
go p.Run(ctx)
time.Sleep(time.Second)
synctest.Wait()
})
m.Lock()
defer m.Unlock()
return restored, ecChain
}
func ecAuditReport(t *testing.T, restored map[int]object.Object, orig []object.Object, lost []int) (wrong int) {
for _, i := range lost {
got, ok := restored[i]
if !ok {
t.Logf("part %d: NOT recreated", i)
continue
}
if bytes.Equal(got.Payload(), orig[i].Payload()) && got.GetID() == orig[i].GetID() {
t.Logf("part %d: recreated, identical to the original", i)
continue
}
wrong++
diff := 0
for j := range got.Payload() {
if got.Payload()[j] != orig[i].Payload()[j] {
diff++
}
}
t.Logf("part %d: recreated with WRONG payload (%d of %d bytes differ), OID %s != original %s",
i, diff, len(got.Payload()), got.GetID(), orig[i].GetID())
}
return wrong
}
// H4: rule #0 lost more than its parity, restored from rule #1.
func TestECAuditH4_CrossRuleRestore(t *testing.T) {
rules := []iec.Rule{{DataPartNum: 3, ParityPartNum: 2}, {DataPartNum: 2, ParityPartNum: 2}}
lost := []int{2, 3, 4}
t.Run("payload 1002 (3*334 == 2*501)", func(t *testing.T) {
restored, chain := ecAuditRepair(t, rules, 1002, lost)
require.Zero(t, ecAuditReport(t, restored, chain[0], lost))
})
t.Run("payload 1000 (3*334 > 2*500)", func(t *testing.T) {
restored, chain := ecAuditRepair(t, rules, 1000, lost)
require.NotZero(t, ecAuditReport(t, restored, chain[0], lost))
})
}
Context
Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.
Regression
Not known.
Your Environment
- Version used: 0.58.2 (
1f0d51b5d) - Server setup and configuration: none, unit tests only
- Operating System and version (
uname -a): Darwin 25.6.0 arm64, Go 1.27.3
- Lenguaje dominante
- Go
- Estrellas
- 37
- Forks
- 56
- Merge medio
- 1 d 21 h
- PR fusionados (30 d)
- 46
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de nspcc-dev/neofs-node
-
bug hal:proposed neofs-storage
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
nspcc-dev/neofs-node#4325 · 3 comentarios ·
Los mantenedores suelen responder en 1 día
-
Check alpha scope in IRAbiertohal:proposed I4 neofs-storage S3 security U4
Dificultad 2/5 1-3 horas Aptitud para principiantes 69/100
nspcc-dev/neofs-node#4280 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
Dificultad 4/5 3-5 días Aptitud para principiantes 45/100
nspcc-dev/neofs-node#4339 ·
Los mantenedores suelen responder en 1 día
-
enhancement I3 neofs-storage performance S3 U3
Dificultad 5/5 Más de una semana Aptitud para principiantes 20/100
nspcc-dev/neofs-node#4337 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
bug neofs-storage
Dificultad 4/5 3-5 días Aptitud para principiantes 45/100
nspcc-dev/neofs-node#4336 ·
Los mantenedores suelen responder en 1 día
Todos los issues de nspcc-dev/neofs-node
Issues similares
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 86/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 64/100
projectdiscovery/cdncheck#554 ·
Los mantenedores suelen responder en 1 día
-
[Bug]: Miniflux integration sends internal OAuth-protected URL instead of original article URLAbiertotriage needed
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100
Los mantenedores suelen responder en 1 día
-
Missing Readme InstructionsAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 78/100
OpenSlides/openslides-go#342 ·
Los mantenedores suelen responder en 4 días
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100