NullPointerException in NettyClientHandler.onHeadersRead when a HEADERS frame races concurrent stream teardown
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- grpc, java
- Domain
- backend-api-design, networking
Research direction
Start in NettyClientHandler.java with onHeadersRead, then compare onDataRead and onRstStreamRead with the server-side guard pattern from #10384. Verify behavior when clientStream() returns null after stream teardown; done means cleared TransportState is handled without a null dereference across the affected callbacks.
Written by the indexing model from the issue text.
Description
What version of gRPC-Java are you using?
1.69.0 (confirmed the same unguarded code is still present at HEAD of the latest release, 1.84.0)
What did you see instead?
A NullPointerException inside NettyClientHandler.onHeadersRead, wrapped by google-cloud-pubsub's StreamingSubscriberConnection:
com.google.api.gax.rpc.UnknownException: io.grpc.StatusRuntimeException: UNKNOWN
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:119)
...
Caused by: io.grpc.StatusRuntimeException: UNKNOWN
at io.grpc.Status.asRuntimeException(Status.java:532)
... 16 common frames omitted
Caused by: java.lang.NullPointerException: Cannot invoke "io.grpc.netty.NettyClientStream$TransportState.tag()" because "stream" is null
at io.grpc.netty.NettyClientHandler.onHeadersRead(NettyClientHandler.java:400)
at io.grpc.netty.NettyClientHandler$FrameListener.onHeadersRead(NettyClientHandler.java:981)
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
...
Observed on a com.google.cloud:google-cloud-pubsub:1.134.2 Subscriber (StreamingPull), during a burst where ~16 subscriber connections were being established concurrently within a ~10s window (a batch of pods restarting and each re-subscribing to many Pub/Sub subscriptions at once).
What did you expect to see?
No NPE — a stream whose TransportState has already been cleared should be handled gracefully, the same way it's handled elsewhere in this same class.
Root cause (traced from source)
NettyClientHandler.java:399—NettyClientStream.TransportState stream = clientStream(requireHttp2Stream(streamId));NettyClientHandler.java:400—PerfMark.event("NettyClientHandler.onHeadersRead", stream.tag());— dereferencesstreamwith no null check.NettyClientHandler.java:930-931—clientStream():
This returnsprivate NettyClientStream.TransportState clientStream(Http2Stream stream) { return stream == null ? null : (NettyClientStream.TransportState) stream.getProperty(streamKey); }nullwhenever theHttp2Streamobject is still live in the connection (sorequireHttp2Streamdoesn't throw) but itsTransportStateproperty was already cleared — e.g. a stream-cancel/RST race. That's exactly the condition hit when a HEADERS frame for that stream arrives concurrently with its own teardown.
The same unguarded clientStream(requireHttp2Stream(streamId)) → immediate dereference pattern also exists at:
NettyClientHandler.java:436-437(onDataRead)NettyClientHandler.java:448-450(onRstStreamRead)
Suggested fix
This looks like the same class of bug as #10364, which was fixed server-side in #10384 by adding a null guard right after the serverStream(requireHttp2Stream(streamId)) call:
NettyServerStream.TransportState stream = serverStream(requireHttp2Stream(streamId));
if (stream == null) {
return;
}
The same guard pattern would apply to NettyClientHandler's onHeadersRead (and likely onDataRead/onRstStreamRead, which have the identical shape).
Impact observed
Low — this was WARN-level in our application logs (not fatal), and the underlying gax retry logic (UNKNOWN status is retryable) recovered the stream automatically. Filing for visibility since the null-dereference itself is a real, reachable bug with a known-good fix pattern already established for the server-side sibling.
- Dominant language
- Java
- Stars
- 12.1k
- Forks
- 4k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 29
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from grpc/grpc-java
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
grpc/grpc-java#13063 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
grpc/grpc-java#13052 · 4 comments ·
Maintainers usually reply within 1 day
-
docs enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
grpc/grpc-java#10824 · 8 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
grpc/grpc-java#13075 · 1 comment ·
Maintainers usually reply within 1 day
-
question
Difficulty 5/5 Over a week Newbie friendliness 25/100
grpc/grpc-java#13066 · 5 comments ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
aoqia194/leaf-loader#19 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/streampark#4521 ·
-
Update license yearOpen0 - Backlog 1 - Ready documentation good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
cbor
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
FasterXML/jackson-dataformats-binary#844 ·
Maintainers usually reply within 1 day