EC Audit: H4: policer rebuilds wrong EC parts when it restores a rule from another rule

Abierto
#4,328 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go

Línea de trabajo

Start in pkg/services/policer/ec.go at fillParityParts and sliceECParts, then trace the recursive restore path around the lines linked in the issue. Run the supplied TestECAuditH4 probe with go test ./pkg/services/policer/ -run TestECAuditH4 -v; done means restored parts match the originals for both payload sizes and the recursive path handles the source rule's padding correctly.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug hal:proposed neofs-storage

Expected Behavior

When a rule has lost more parts than its parity covers and the container has another EC rule, the policer rebuilds the lost parts from the other rule. The rebuilt parts are byte-identical to the original ones.

Current Behavior

The policer fetches all parts of the other rule into one buffer, data and parity, and then cuts that buffer into parts of the rule being restored (policer/ec.go#L344-L352, fillParityParts → sliceECParts at #L535-L562). The data area of a rule is d * ceil(len/d) bytes: the payload followed by zero padding. This length differs between rules. When the restored rule's data area is longer than the source rule's, the end of its last data part is taken from the source rule's first parity part instead of zeros. Parity is then computed over that.

Test below, rules 3/2 (restored) and 2/2 (source), parts 2, 3, 4 of the first rule lost:

Payload Data area 3/2 Data area 2/2 Recreated parts
1002 bytes 1002 1002 identical to the originals
1000 bytes 1002 1000 all three wrong, 2 of 334 bytes differ in each

The recreated objects have other OIDs than the originals and do not match the parent's __NEOFS__EC_PART_HASHES. They fail the EC part check of the node they are sent to (checkECParent), so the policer repeats the same attempt every pass and the rule stays degraded. That last step is from reading the code; the test replaces the replicator with a mock.

The same re-slicing is done a second time on the recursive path, when the source rule is itself short of parts and is restored from a third rule (policer/ec.go#L511-L524). There obj = obj[:needLen] also extends the slice over whatever the buffer held before.

Possible Solution

Take only the payload (parentHeader.PayloadSize() bytes) from the source rule and encode it with the target rule, as PUT does, instead of re-cutting the source rule's parts.

Steps to Reproduce (for bugs)

  1. Save the test below as pkg/services/policer/ecaudit_test.go.
  2. go test ./pkg/services/policer/ -run TestECAuditH4 -v
payload 1002: part 2: recreated, identical to the original (same for 3, 4)
payload 1000: part 2: recreated with WRONG payload (2 of 334 bytes differ), OID … != original … (same for 3, 4)
ecaudit_test.go
package policer

import (
	"bytes"
	"context"
	"strconv"
	"sync"
	"testing"
	"testing/synctest"
	"time"

	iec "github.com/nspcc-dev/neofs-node/internal/ec"
	"github.com/nspcc-dev/neofs-node/internal/testutil"
	objectcore "github.com/nspcc-dev/neofs-node/pkg/core/object"
	"github.com/nspcc-dev/neofs-node/pkg/services/replicator"
	apistatus "github.com/nspcc-dev/neofs-sdk-go/client/status"
	neofscryptotest "github.com/nspcc-dev/neofs-sdk-go/crypto/test"
	"github.com/nspcc-dev/neofs-sdk-go/netmap"
	"github.com/nspcc-dev/neofs-sdk-go/object"
	oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
	"github.com/stretchr/testify/require"
	"go.uber.org/zap/zaptest"
)

// runs one policer pass over the local part #0 of rule #0. Parts of rule #0
// listed in lost are missing on their nodes, all other parts of all rules are
// healthy. Returns recreated objects by part index and the original chain.
func ecAuditRepair(t *testing.T, ecRules []iec.Rule, parentSize uint64, lost []int) (map[int]object.Object, [][]object.Object) {
	var nodesPlacement [][]netmap.NodeInfo
	for _, r := range ecRules {
		nodesPlacement = append(nodesPlacement, testutil.Nodes(int(r.DataPartNum+r.ParityPartNum)))
	}
	// testutil.Nodes is deterministic, make nodes of different rules distinct
	all := testutil.Nodes(64)
	off := 0
	for i := range nodesPlacement {
		nodesPlacement[i] = all[off : off+len(nodesPlacement[i])]
		off += len(nodesPlacement[i])
	}

	parent, ecChain := generateECChain(t, parentSize, ecRules)
	parentAddr := parent.Address()
	parentOID := parentAddr.Object()

	localPart := ecChain[0][0]
	localObj := objectcore.AddressWithAttributes{
		Address: oid.NewAddress(parentAddr.Container(), localPart.GetID()),
		Type:    object.TypeRegular,
		Attributes: []string{"0", "0", string(parentOID[:]),
			strconv.FormatUint(localPart.PayloadSize(), 10)},
	}
	localNode := newTestLocalNode()
	localNode.objList = []objectcore.AddressWithAttributes{localObj}
	localNode.obj = localPart
	localNode.setECPartResult(t, localPart)

	mockNet := newMockNetwork()
	mockNet.setObjectNodesECResult(parentAddr.Container(), parentOID, nodesPlacement, ecRules)
	mockNet.pubKey = nodesPlacement[0][0].PublicKey()
	conns := newMockAPIConnections()

	isLost := func(i int) bool {
		for _, l := range lost {
			if l == i {
				return true
			}
		}
		return false
	}
	for ruleIdx := range ecRules {
		for i, obj := range ecChain[ruleIdx] {
			n := nodesPlacement[ruleIdx][i]
			if ruleIdx == 0 && isLost(i) {
				conns.setHeadResult(n, parentAddr, object.Object{}, error(apistatus.ErrObjectNotFound))
				conns.setRangeResult(n, parentAddr, uint8(ruleIdx), uint8(i), nil, apistatus.ErrObjectNotFound)
				continue
			}
			conns.setHeadResult(n, parentAddr, obj, nil)
			conns.setRangeResult(n, parentAddr, uint8(ruleIdx), uint8(i), obj.Payload(), nil)
		}
	}

	r := newTestReplicator(t)
	r.success = true
	var m sync.Mutex
	restored := make(map[int]object.Object)
	r.onTask = func(task replicator.Task, _ []netmap.NodeInfo) {
		obj := task.Object()
		pi, err := iec.GetPartInfo(*obj)
		if err != nil || pi.RuleIndex != 0 {
			return
		}
		m.Lock()
		restored[pi.Index] = *obj
		m.Unlock()
	}

	p := New(neofscryptotest.Signer(),
		WithReplicationCooldown(50*time.Millisecond),
		WithNetwork(mockNet),
		WithLogger(zaptest.NewLogger(t)),
		WithMetrics(&mockMetrics{}),
	)
	p.localStorage = localNode
	p.apiConns = conns
	p.replicator = r

	synctest.Test(t, func(t *testing.T) {
		ctx, cancel := context.WithCancel(context.Background())
		t.Cleanup(cancel)
		go p.Run(ctx)
		time.Sleep(time.Second)
		synctest.Wait()
	})

	m.Lock()
	defer m.Unlock()
	return restored, ecChain
}

func ecAuditReport(t *testing.T, restored map[int]object.Object, orig []object.Object, lost []int) (wrong int) {
	for _, i := range lost {
		got, ok := restored[i]
		if !ok {
			t.Logf("part %d: NOT recreated", i)
			continue
		}
		if bytes.Equal(got.Payload(), orig[i].Payload()) && got.GetID() == orig[i].GetID() {
			t.Logf("part %d: recreated, identical to the original", i)
			continue
		}
		wrong++
		diff := 0
		for j := range got.Payload() {
			if got.Payload()[j] != orig[i].Payload()[j] {
				diff++
			}
		}
		t.Logf("part %d: recreated with WRONG payload (%d of %d bytes differ), OID %s != original %s",
			i, diff, len(got.Payload()), got.GetID(), orig[i].GetID())
	}
	return wrong
}

// H4: rule #0 lost more than its parity, restored from rule #1.
func TestECAuditH4_CrossRuleRestore(t *testing.T) {
	rules := []iec.Rule{{DataPartNum: 3, ParityPartNum: 2}, {DataPartNum: 2, ParityPartNum: 2}}
	lost := []int{2, 3, 4}

	t.Run("payload 1002 (3*334 == 2*501)", func(t *testing.T) {
		restored, chain := ecAuditRepair(t, rules, 1002, lost)
		require.Zero(t, ecAuditReport(t, restored, chain[0], lost))
	})
	t.Run("payload 1000 (3*334 > 2*500)", func(t *testing.T) {
		restored, chain := ecAuditRepair(t, rules, 1000, lost)
		require.NotZero(t, ecAuditReport(t, restored, chain[0], lost))
	})
}

Context

Found by an audit of the erasure coding code. The audit was static: code reading plus unit-level probe tests in the packages concerned. No cluster was run, so there is no end-to-end reproduction.

Regression

Not known.

Your Environment

  • Version used: 0.58.2 (1f0d51b5d)
  • Server setup and configuration: none, unit tests only
  • Operating System and version (uname -a): Darwin 25.6.0 arm64, Go 1.27.3
Lenguaje dominante
Go
Estrellas
37
Forks
56
Merge medio
1 d 21 h
PR fusionados (30 d)
46

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de nspcc-dev/neofs-node

Todos los issues de nspcc-dev/neofs-node

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.