engine: forkchoiceUpdated with payload attributes right after newPayload sometimes fails with "head header not found"

Open
#26,426 2 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@yperbasis is already working on this.

Since Nov 18, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
backend, databases

Research direction

Start by tracing the Engine API path for engine_forkchoiceUpdatedV3 into fork choice handling, then inspect how the validated block becomes visible through parallel state flushing and overlay reads. Reproduce with the included repro.py and compare default flags with --sync.parallel-state-flushing=false. Done means the first call after newPayload returns VALID with a payload ID reliably under the default settings.

Written by the indexing model from the issue text.

Description

What happens

When a forkchoiceUpdated with payload attributes is the first call that makes a block just accepted by newPayload the head, Erigon sometimes answers it with -32000 head header not found for hash <head> instead of VALID with a payloadId. The fork choice itself is applied (the head moves). Only the payload build is not started. Sending the same call again a few milliseconds later works.

This is the call Prysm makes when its validator proposes the next slot: it imports the block, sends newPayload, and then right away forkchoiceUpdated with the attributes for its own slot. Lighthouse sends a plain forkchoiceUpdated first and the attributes in a later call, so it does not hit this.

Seen on a devnet

Kurtosis devnet (ethereum-package, minimal preset, 6 s slots) with Erigon built from main at fd679ca053 paired with Prysm (the node that hit it also had the change from #26412, which does not touch this path; the single node runs below are plain main). Engine API traffic between Prysm and Erigon, recorded with rpc-snooper (bodies trimmed):

04:58:25  engine_newPayloadV4         block 96 0xc64c7ede...       -> VALID, latestValidHash 0xc64c7ede...
04:58:25  engine_forkchoiceUpdatedV3  head 0xc64c7ede..., attrs  -> error -32000 "head header not found for hash c64c7ede..."
04:58:31  engine_forkchoiceUpdatedV3  head 0xc64c7ede..., attrs  -> VALID, payloadId 0x0000000000000010
04:58:31  engine_getPayloadV5         0x0000000000000010          -> block 97

Erigon log for the failing call:

[INFO] [11-17|04:58:25.231] head validated                           hash=0xc64c7ede... number=96
[WARN] [11-17|04:58:25.231] [rpc] served                             method=engine_forkchoiceUpdatedV3 reqid=275 err="head header not found for hash c64c7ede..."
[INFO] [11-17|04:58:25.233] Timings: Forkchoice                      flush+commit=1ms

Prysm logged ERROR blockchain: received an undefined execution engine error error=head header not found for hash c64c7ede.... At its slot 6 s later it had no payload id, so it asked again 175 ms into the slot and called getPayload straight after. The block went out on time, but its payload was built at the slot instead of in the 6 s before it. In that run this happened once in 111 such calls across the two Erigon nodes paired with Prysm.

Reproduce on a single node

Engine API only, no consensus client. Each round builds block N on the node, sends newPayload(N) and then forkchoiceUpdated(head=N) with attributes, like Prysm. Script and genesis below.

erigon init --datadir ./data genesis.json
erigon --datadir ./data --externalcl --networkid 7929 --authrpc.jwtsecret ./jwt.hex --authrpc.port 8551 \
  --http --http.port 8545 --nodiscover --maxpeers 0 --nat none --no-downloader
python3 repro.py http://127.0.0.1:8551 ./jwt.hex 1000

Results on main (fd679ca053), 2000 rounds each:

setup failed calls
default flags 4 (2 of 1000 twice, once with the node pinned to one CPU)
--sync.parallel-state-flushing=false 0
default flags, plain forkchoiceUpdated first (--lighthouse-order) 0

Every failure is head header not found for hash <block N>, and the retry right after it succeeds.

Why

With parallel state flushing on (the default), when the new head is the block the last newPayload validated, the fork choice update sends VALID back before it merges that block into the database and publishes it for readers. The engine server then reads the head header to start the payload build. The header is not committed yet, and the overlay fallback in that read only sees it once the update has published its state. The debug log of a failing round shows the order:

11:58:23.049 HandleForkChoice done status=VALID
11:58:23.051 [dispatchNotifications] publishing SD
11:58:23.053 [rpc] served engine_forkchoiceUpdatedV3 err="head header not found for hash 8f977957..."
11:58:23.055 [dispatcher] PublishOverlay: publishing sdNil=false
11:58:23.061 Timings: Forkchoice flush+commit=5ms

In a passing round the build starts after the commit (BlockBuilder added at 22.706, after flush+commit at 22.703).

Expected

VALID with a payloadId, since the head is valid and the fork choice was applied. If the head header can't be read yet, the build could take it from the block the update just validated, or wait for the update to publish its state, instead of failing the call.

repro.py
#!/usr/bin/env python3
"""Each round: build block N on the node, newPayload(N), then forkchoiceUpdated(head=N) with payload attributes,
the call Prysm makes when its validator proposes the next slot. Prints how many of those calls failed.

usage: repro.py <auth rpc url> <jwt secret file> [rounds] [--lighthouse-order]
"""
import base64, hashlib, hmac, json, sys, time, urllib.request

URL, SECRET = sys.argv[1], bytes.fromhex(open(sys.argv[2]).read().strip().removeprefix("0x"))
ROUNDS = int(sys.argv[3]) if len(sys.argv) > 3 and sys.argv[3].isdigit() else 1000
PLAIN_FCU_FIRST = "--lighthouse-order" in sys.argv
ZERO = "0x" + "00" * 32


def token():
    enc = lambda b: base64.urlsafe_b64encode(b).rstrip(b"=")
    msg = enc(b'{"alg":"HS256","typ":"JWT"}') + b"." + enc(json.dumps({"iat": int(time.time())}).encode())
    return (msg + b"." + enc(hmac.new(SECRET, msg, hashlib.sha256).digest())).decode()


def call(method, *params):
    req = urllib.request.Request(URL, json.dumps({"jsonrpc": "2.0", "id": 1, "method": method, "params": list(params)}).encode(),
                                 {"Content-Type": "application/json", "Authorization": "Bearer " + token()})
    return json.load(urllib.request.urlopen(req, timeout=30))


def fcu(head, ts=None):
    attrs = None if ts is None else {"timestamp": hex(ts), "prevRandao": ZERO, "suggestedFeeRecipient": "0x" + "00" * 20,
                                     "withdrawals": [], "parentBeaconBlockRoot": ZERO}
    return call("engine_forkchoiceUpdatedV3", {"headBlockHash": head, "safeBlockHash": ZERO, "finalizedBlockHash": ZERO}, attrs)


head = call("eth_getBlockByNumber", "latest", False)["result"]
r = fcu(head["hash"], int(head["timestamp"], 16) + 1)
failed = 0
for i in range(ROUNDS):
    time.sleep(0.3)
    payload = call("engine_getPayloadV3", r["result"]["payloadId"])["result"]["executionPayload"]
    status = call("engine_newPayloadV3", payload, [], ZERO)["result"]["status"]
    assert status == "VALID", status
    next_ts = int(payload["timestamp"], 16) + 1
    if PLAIN_FCU_FIRST:
        fcu(payload["blockHash"])
    r = fcu(payload["blockHash"], next_ts)
    if "error" in r:
        failed += 1
        print(f"block {int(payload['blockNumber'], 16)}: newPayload VALID, then forkchoiceUpdated with attributes -> {r['error']}")
        r = fcu(payload["blockHash"], next_ts)  # the same call again succeeds
        assert r["result"]["payloadStatus"]["status"] == "VALID", r
print(f"{failed} of {ROUNDS} forkchoiceUpdated calls with payload attributes failed")
genesis.json
{
  "config": {
    "chainId": 7929,
    "homesteadBlock": 0, "eip150Block": 0, "eip155Block": 0, "eip158Block": 0, "byzantiumBlock": 0,
    "constantinopleBlock": 0, "petersburgBlock": 0, "istanbulBlock": 0, "berlinBlock": 0, "londonBlock": 0,
    "mergeNetsplitBlock": 0, "terminalTotalDifficulty": 0, "terminalTotalDifficultyPassed": true,
    "shanghaiTime": 0, "cancunTime": 0,
    "blobSchedule": {"cancun": {"target": 3, "max": 6, "baseFeeUpdateFraction": 3338477}}
  },
  "nonce": "0x0", "timestamp": "0x0", "extraData": "0x", "gasLimit": "0x2faf080", "difficulty": "0x0",
  "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "coinbase": "0x0000000000000000000000000000000000000000", "baseFeePerGas": "0x3b9aca00",
  "excessBlobGas": "0x0", "blobGasUsed": "0x0", "alloc": {}
}
Dominant language
Go
Stars
3.6k
Forks
1.6k
Avg merge
1d 17h
Merged PRs (30d)
820

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from erigontech/erigon

All issues in erigontech/erigon

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.