"Cancelled" streams causing gRPC status code 13 in Envoy proxy

Open
#239 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
grpc, java
Domain
api, backend

Research direction

Start with the DiscoveryRequestStreamObserver.onCancelled and cancel entry points mentioned in the issue, using java-control-plane v0.1.31 and Envoy 1.18.6 while comparing both sides' logs. Trace when cancellation and stream closure occur and determine whether status code 13 is expected or indicates an error; done means the cause and a reproducible correction or explanation are established.

Written by the indexing model from the issue text.

Description

I see an info level [n] stream cancelled in my xDS server logs (where the [n] is the stream id) immediately followed by a warning level StreamEndpoints gRPC config stream closed: 13 in my Envoy logs. I'm running v0.1.31 of java-control-plane and 1.18.6 of Envoy. Envoy appears to be getting the data successfully but logs are getting filled on both sides of the connection. It "feels" like something is abruptly ending the gRPC connection.

My reading of the code suggests the [n] stream cancelled message is a red herring. It appears this will be outputted whenever a stream is cancelled (the DiscoveryRequestStreamObserver.onCancelled method) and that the DiscoveryRequestStreamObserver.cancel method is called whenever a stream closed, whether with an error or not. The cancel method appears to only cancel watchers though.

I'm trying to collect more information on this issue. I'll try getting a debugger on the xDS server tomorrow.

Dominant language
Java
Stars
312
Forks
150
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from envoyproxy/java-control-plane

All issues in envoyproxy/java-control-plane

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.