commitment v3: a DB unwind does not restore a deleted record; other findings from the #24304 post-merge review

Open
#26,438 74 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@awskii is already working on this.

Since Nov 17, 2026.

Assessment

This issue has not been assessed yet.

Description

Findings from a review of #24304 as merged (fd679ca053). Point 1 is a defect for a node that runs with --experimental.commitment-v3: after a reorg it can reject a valid block. Points 2 to 6 are small. Legacy commitment is not affected by any of them.

1. A DB unwind does not restore a deleted v3 record

A reorg can bring back a stale commitment record. The next block that changes that storage is then rejected with invalid state root hash, or the fold fails with commitment v3: invalid node record.

Cause

DomainRoTx.unwind does not restore an empty previous value in the commitment domain (db/state/domain.go, from #26172):

if d.Name == kv.CommitmentDomain && len(diff.Value) == 0 {
	continue
}

This is right for the legacy trie, where an empty previous value means that the branch was not in the trie. In v3 an empty value is the stored deletion of a node record, and reads depend on it.

The sequence:

  • Key K has a record in an older step, in the DB or in files.
  • A kept block deletes K at step S. The DB now holds an empty value for K at step S.
  • A later block of step S writes K again. It replaces the same row, and its diff holds the empty previous value.
  • That block is unwound. The row is deleted and the empty value is not written back, so the old record of K is the latest again.

The storage root record is read by its key, without the account leaf, so the next fold of that storage starts from the stale record. If nothing changes that storage again, the stale record stays as an orphan, and FoldCommitmentV3 reports it once it is in files.

#24304 got this with its merge of main on 2026-11-16. Each of the two changes is correct without the other.

How to reproduce

A unit test for db/state, package state. On main (29f1e11f97) it fails with Should be empty, but was [1]:

// A v3 record deletion is stored as an empty value. When a later block of the same step
// writes the record again and is unwound, the deletion must come back.
func TestDomain_UnwindRestoresDeletedCommitmentV3Record(t *testing.T) {
	cfg := statecfg.Schema.CommitmentDomain
	statecfg.EnableCommitmentV3Records(&cfg)
	db, d := testDbAndDomainOfStep(t, cfg, 16, log.New())
	tx, err := db.BeginRw(t.Context())
	require.NoError(t, err)
	defer tx.Rollback()
	view := d.beginForTests()
	defer view.Close()
	writer := view.NewWriter(db)
	defer writer.Close()

	key, old := []byte("key"), []byte{1}
	require.NoError(t, writer.PutWithPrev(key, old, 2, nil))
	require.NoError(t, writer.DeleteWithPrev(key, 18, old))
	writer.SetDiff(&kv.DomainDiff{})
	require.NoError(t, writer.PutWithPrev(key, []byte{2}, 20, []byte{}))
	diffs := writer.Diff().GetDiffSet()
	require.NoError(t, writer.Flush(t.Context(), tx))
	require.NoError(t, view.unwind(t.Context(), tx, 20, diffs, nil))
	value, _, _, err := view.GetLatest(key, tx)
	require.NoError(t, err)
	require.Empty(t, value)
}

End to end through the exec module, with step size 64: a contract has slots 1 and 2. Block 25 clears both and block 26 sets slot 1. The fork choice moves to a side block 26' with a plain transfer, then side block 27' sets slot 2. With ERIGON_COMMITMENT_V3=true, 27' is rejected with invalid state root hash. With the flag off it is valid.

The end-to-end test, package execmoduletester
// Canonical: 24 transfers | clear slots 1 and 2 | set slot 1.
// Side:      24 transfers | clear slots 1 and 2 | transfer | set slot 2.
// With step size 64 the genesis storage is in step 0, and the clearing block and
// the block after it are in step 1.
func TestReorgAfterStorageEmptiedAndRefilled(t *testing.T) {
	const stepSize, clearIdx = 64, 24
	key, err := crypto.HexToECDSA("b71c71a67e1177ad4e901695e1b4b9ee17ae16c6668d313eac2f96dbcda3f291")
	require.NoError(t, err)
	sender := crypto.PubkeyToAddress(key.PublicKey)
	store := common.HexToAddress("0xaa")
	slot := func(i byte) (h common.Hash) { h[31] = i; return h }
	m := New(t, WithStepSize(stepSize), WithKey(key), WithGenesisSpec(&types.Genesis{
		Config: chain.AllProtocolChanges,
		Alloc: types.GenesisAlloc{
			sender: {Balance: new(big.Int).Exp(big.NewInt(10), big.NewInt(18), nil)},
			// sstore(calldata[0:32], calldata[32:64])
			store: {Code: []byte{0x60, 0x20, 0x35, 0x60, 0x00, 0x35, 0x55, 0x00}, Storage: map[common.Hash]common.Hash{slot(1): slot(1), slot(2): slot(1)}},
		},
	}))
	signer := *types.LatestSignerForChainID(m.ChainConfig.ChainID)
	gasPrice := uint256.NewInt(m.Genesis.BaseFee().Uint64() * 2)
	sign := func(txn types.Transaction) types.Transaction {
		signed, err := types.SignTx(txn, signer, key)
		require.NoError(t, err)
		return signed
	}
	sstore := func(nonce uint64, k, v byte) types.Transaction {
		ks, vs := slot(k), slot(v)
		return sign(types.NewTransaction(nonce, store, uint256.NewInt(0), 2_000_000, gasPrice, append(ks[:], vs[:]...)))
	}
	gen := func(side bool) func(int, *blockgen.BlockGen) {
		return func(i int, b *blockgen.BlockGen) {
			nonce := b.TxNonce(sender)
			switch {
			case i == clearIdx:
				b.AddTx(sstore(nonce, 1, 0))
				b.AddTx(sstore(nonce+1, 2, 0))
			case i == clearIdx+1 && !side:
				b.AddTx(sstore(nonce, 1, 7))
			case i == clearIdx+2:
				b.AddTx(sstore(nonce, 2, 9))
			default:
				b.AddTx(sign(types.NewTransaction(nonce, common.Address{1}, uint256.NewInt(1), 21_000, gasPrice, nil)))
			}
		}
	}
	canonical, err := m.GenerateChain(clearIdx+2, gen(false))
	require.NoError(t, err)
	side, err := m.GenerateChain(clearIdx+3, gen(true))
	require.NoError(t, err)

	ctx := t.Context()
	require.NoError(t, m.InsertValidateAndUfc1By1(ctx, canonical.Blocks))
	m.ExecModule.WaitIdle(ctx)
	for _, block := range side.Blocks[clearIdx+1:] {
		status, err := m.InsertBlocks(ctx, []*types.Block{block})
		require.NoError(t, err)
		require.Equal(t, execmodule.ExecutionStatusSuccess, status)
		fcu, err := m.UpdateForkChoice(ctx, block.Header())
		require.NoError(t, err)
		require.Equalf(t, execmodule.ExecutionStatusSuccess, fcu.Status, "side block %d: %s", block.NumberU64(), fcu.ValidationError)
		m.ExecModule.WaitIdle(ctx)
	}
}

A random reorg test through the exec module (5 reorgs of depth 1 to 4 per seed, Prague and Amsterdam) fails for 8 of 40 seeds with v3 and for no seed with the flag off.

Fix

Adding && !d.CommitmentV3Records to the condition makes all of these pass, the random test included. TestDomain_UnwindDoesNotRestoreAbsentBranch then needs the legacy config: it takes statecfg.Schema.CommitmentDomain, which is the v3 config in the v3 CI leg, and its case is a legacy one.

Smaller points

  1. db/state: a refused convert --v3 prints the advice to add --no-history "if commitment history was pruned". When commitment history is complete but accounts, storage or code history does not start at step 0, rebuild --yes fails with [rebuild_commitment_history] root mismatch at block 0, after rm-state has already removed the commitment files. rebuild --yes --no-history then works. The advice could name any pruned history.
  2. cmd/integration/commands: TestCommitmentRebuildWithoutHistoryAfterPrunedHistoryRefusal also passes without --no-history, because its DB does not have the commitment-history flag set. So it does not show that the flag is needed.
  3. db/state: verifyCommitmentV3History now folds once per history file, as of the first commit. This sample reads entries that the earlier two samples per file did not, and misses others: with one byte changed in the account root record at a later txNum of a file, convert now accepts the file, and the earlier samples rejected it.
  4. cmd/utils: no test fails when SetEthConfig stops calling setCommitmentV3, so the node flag has no test.
  5. execution/commitment/backtester always builds a legacy trie. On a v3 datadir it stops in SeekCommitment with the hint to set ERIGON_COMMITMENT_V3=true, also when it is set. integration commitment branch reads legacy keys, so on a v3 datadir it finds nothing and prints <empty>.
Dominant language
Go
Stars
3.6k
Forks
1.6k
Avg merge
1d 17h
Merged PRs (30d)
820

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from erigontech/erigon

All issues in erigontech/erigon

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.