engine: forkchoiceUpdated with payload attributes right after newPayload sometimes fails with "head header not found"
Maintainers usually reply within 1 day
@yperbasis is already working on this.
Since Nov 18, 2026.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
Research direction
Start by tracing the Engine API path for engine_forkchoiceUpdatedV3 into fork choice handling, then inspect how the validated block becomes visible through parallel state flushing and overlay reads. Reproduce with the included repro.py and compare default flags with --sync.parallel-state-flushing=false. Done means the first call after newPayload returns VALID with a payload ID reliably under the default settings.
Written by the indexing model from the issue text.
Description
What happens
When a forkchoiceUpdated with payload attributes is the first call that makes a block just accepted by newPayload the head, Erigon sometimes answers it with -32000 head header not found for hash <head> instead of VALID with a payloadId. The fork choice itself is applied (the head moves). Only the payload build is not started. Sending the same call again a few milliseconds later works.
This is the call Prysm makes when its validator proposes the next slot: it imports the block, sends newPayload, and then right away forkchoiceUpdated with the attributes for its own slot. Lighthouse sends a plain forkchoiceUpdated first and the attributes in a later call, so it does not hit this.
Seen on a devnet
Kurtosis devnet (ethereum-package, minimal preset, 6 s slots) with Erigon built from main at fd679ca053 paired with Prysm (the node that hit it also had the change from #26412, which does not touch this path; the single node runs below are plain main). Engine API traffic between Prysm and Erigon, recorded with rpc-snooper (bodies trimmed):
04:58:25 engine_newPayloadV4 block 96 0xc64c7ede... -> VALID, latestValidHash 0xc64c7ede...
04:58:25 engine_forkchoiceUpdatedV3 head 0xc64c7ede..., attrs -> error -32000 "head header not found for hash c64c7ede..."
04:58:31 engine_forkchoiceUpdatedV3 head 0xc64c7ede..., attrs -> VALID, payloadId 0x0000000000000010
04:58:31 engine_getPayloadV5 0x0000000000000010 -> block 97
Erigon log for the failing call:
[INFO] [11-17|04:58:25.231] head validated hash=0xc64c7ede... number=96
[WARN] [11-17|04:58:25.231] [rpc] served method=engine_forkchoiceUpdatedV3 reqid=275 err="head header not found for hash c64c7ede..."
[INFO] [11-17|04:58:25.233] Timings: Forkchoice flush+commit=1ms
Prysm logged ERROR blockchain: received an undefined execution engine error error=head header not found for hash c64c7ede.... At its slot 6 s later it had no payload id, so it asked again 175 ms into the slot and called getPayload straight after. The block went out on time, but its payload was built at the slot instead of in the 6 s before it. In that run this happened once in 111 such calls across the two Erigon nodes paired with Prysm.
Reproduce on a single node
Engine API only, no consensus client. Each round builds block N on the node, sends newPayload(N) and then forkchoiceUpdated(head=N) with attributes, like Prysm. Script and genesis below.
erigon init --datadir ./data genesis.json
erigon --datadir ./data --externalcl --networkid 7929 --authrpc.jwtsecret ./jwt.hex --authrpc.port 8551 \
--http --http.port 8545 --nodiscover --maxpeers 0 --nat none --no-downloader
python3 repro.py http://127.0.0.1:8551 ./jwt.hex 1000
Results on main (fd679ca053), 2000 rounds each:
| setup | failed calls |
|---|---|
| default flags | 4 (2 of 1000 twice, once with the node pinned to one CPU) |
--sync.parallel-state-flushing=false |
0 |
default flags, plain forkchoiceUpdated first (--lighthouse-order) |
0 |
Every failure is head header not found for hash <block N>, and the retry right after it succeeds.
Why
With parallel state flushing on (the default), when the new head is the block the last newPayload validated, the fork choice update sends VALID back before it merges that block into the database and publishes it for readers. The engine server then reads the head header to start the payload build. The header is not committed yet, and the overlay fallback in that read only sees it once the update has published its state. The debug log of a failing round shows the order:
11:58:23.049 HandleForkChoice done status=VALID
11:58:23.051 [dispatchNotifications] publishing SD
11:58:23.053 [rpc] served engine_forkchoiceUpdatedV3 err="head header not found for hash 8f977957..."
11:58:23.055 [dispatcher] PublishOverlay: publishing sdNil=false
11:58:23.061 Timings: Forkchoice flush+commit=5ms
In a passing round the build starts after the commit (BlockBuilder added at 22.706, after flush+commit at 22.703).
Expected
VALID with a payloadId, since the head is valid and the fork choice was applied. If the head header can't be read yet, the build could take it from the block the update just validated, or wait for the update to publish its state, instead of failing the call.
repro.py
#!/usr/bin/env python3
"""Each round: build block N on the node, newPayload(N), then forkchoiceUpdated(head=N) with payload attributes,
the call Prysm makes when its validator proposes the next slot. Prints how many of those calls failed.
usage: repro.py <auth rpc url> <jwt secret file> [rounds] [--lighthouse-order]
"""
import base64, hashlib, hmac, json, sys, time, urllib.request
URL, SECRET = sys.argv[1], bytes.fromhex(open(sys.argv[2]).read().strip().removeprefix("0x"))
ROUNDS = int(sys.argv[3]) if len(sys.argv) > 3 and sys.argv[3].isdigit() else 1000
PLAIN_FCU_FIRST = "--lighthouse-order" in sys.argv
ZERO = "0x" + "00" * 32
def token():
enc = lambda b: base64.urlsafe_b64encode(b).rstrip(b"=")
msg = enc(b'{"alg":"HS256","typ":"JWT"}') + b"." + enc(json.dumps({"iat": int(time.time())}).encode())
return (msg + b"." + enc(hmac.new(SECRET, msg, hashlib.sha256).digest())).decode()
def call(method, *params):
req = urllib.request.Request(URL, json.dumps({"jsonrpc": "2.0", "id": 1, "method": method, "params": list(params)}).encode(),
{"Content-Type": "application/json", "Authorization": "Bearer " + token()})
return json.load(urllib.request.urlopen(req, timeout=30))
def fcu(head, ts=None):
attrs = None if ts is None else {"timestamp": hex(ts), "prevRandao": ZERO, "suggestedFeeRecipient": "0x" + "00" * 20,
"withdrawals": [], "parentBeaconBlockRoot": ZERO}
return call("engine_forkchoiceUpdatedV3", {"headBlockHash": head, "safeBlockHash": ZERO, "finalizedBlockHash": ZERO}, attrs)
head = call("eth_getBlockByNumber", "latest", False)["result"]
r = fcu(head["hash"], int(head["timestamp"], 16) + 1)
failed = 0
for i in range(ROUNDS):
time.sleep(0.3)
payload = call("engine_getPayloadV3", r["result"]["payloadId"])["result"]["executionPayload"]
status = call("engine_newPayloadV3", payload, [], ZERO)["result"]["status"]
assert status == "VALID", status
next_ts = int(payload["timestamp"], 16) + 1
if PLAIN_FCU_FIRST:
fcu(payload["blockHash"])
r = fcu(payload["blockHash"], next_ts)
if "error" in r:
failed += 1
print(f"block {int(payload['blockNumber'], 16)}: newPayload VALID, then forkchoiceUpdated with attributes -> {r['error']}")
r = fcu(payload["blockHash"], next_ts) # the same call again succeeds
assert r["result"]["payloadStatus"]["status"] == "VALID", r
print(f"{failed} of {ROUNDS} forkchoiceUpdated calls with payload attributes failed")
genesis.json
{
"config": {
"chainId": 7929,
"homesteadBlock": 0, "eip150Block": 0, "eip155Block": 0, "eip158Block": 0, "byzantiumBlock": 0,
"constantinopleBlock": 0, "petersburgBlock": 0, "istanbulBlock": 0, "berlinBlock": 0, "londonBlock": 0,
"mergeNetsplitBlock": 0, "terminalTotalDifficulty": 0, "terminalTotalDifficultyPassed": true,
"shanghaiTime": 0, "cancunTime": 0,
"blobSchedule": {"cancun": {"target": 3, "max": 6, "baseFeeUpdateFraction": 3338477}}
},
"nonce": "0x0", "timestamp": "0x0", "extraData": "0x", "gasLimit": "0x2faf080", "difficulty": "0x0",
"mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"coinbase": "0x0000000000000000000000000000000000000000", "baseFeePerGas": "0x3b9aca00",
"excessBlobGas": "0x0", "blobGasUsed": "0x0", "alloc": {}
}
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.6k
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 820
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from erigontech/erigon
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
erigontech/erigon#26523 ·
Maintainers usually reply within 1 day
-
flaky test
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
erigontech/erigon#26230 ·
Maintainers usually reply within 1 day
-
txnprovider/txpool: pending nonce counts past gaps in the poolPossibly taken @yperbasis claimed this 7 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
erigontech/erigon#26014 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
tracing: expose custom error selector and raw payload in callTracer revertReasonPossibly taken @lupin012 claimed this 9 days ago. OpenRPC
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
erigontech/erigon#25928 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
txnprovider/txpool: remove redundant blob/commitment length checkPossibly taken @yperbasis claimed this 14 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
erigontech/erigon#25432 ·
Maintainers usually reply within 1 day
All issues in erigontech/erigon
Similar issues
-
enhancement exporter/awskinesis needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
open-telemetry/opentelemetry-collector-contrib#53334 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 83/100
SocialGouv/iterion#3376 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
seaweedfs/seaweedfs#12740 · 2 comments ·
Maintainers usually reply within 1 day
-
area:indexer bug needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
seanb4t/codegraph-go#385 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day