xds: application receives CANCELLED (grpc-status=1, grpc-message="CANCELLED") on in-flight streams during EDS locality-weight updates (1.82.2)

Start with core/src/main/java/io/grpc/internal/Http2ClientStreamTransportState.java, especially statusFromTrailers, and the XdsNameResolver stack entry shown in the report. Reproduce an EDS locality-weight-only update with in-flight streams and inspect the HTTP/2 trailers. Done means determining whether grpc-status CANCELLED with this description is expected and identifying the relevant grpc-java behavior or test coverage to change.

Environment

• grpc-java 1.82.2, grpc-xds 1.82.2 (Traffic Director as xDS control plane)
• grpc-opentelemetry 1.82.2
• Client library: google-cloud-bigtable 2.81.0 / gax-grpc 2.83.0

Symptom

When Traffic Director sends an EDS ClusterLoadAssignment update that changes only locality weights (localities not added/removed), a subset of in-flight streams close with:

io.grpc.StatusRuntimeException: CANCELLED: CANCELLED

The description string is literally the 9 characters CANCELLED — a null description would render as CANCELLED alone with no colon-space-word. Application layers above grpc-java (google-cloud-bigtable / gax retry) do not retry CANCELLED, so this surfaces to the app.

Stacktrace

io.grpc.StatusRuntimeException: CANCELLED: CANCELLED
    at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:150) ~[gax-grpc-2.83.0.jar:2.83.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.cloud.bigtable.data.v2.stub.MetadataExtractorInterceptor$1$1.onClose(MetadataExtractorInterceptor.java:82) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.cloud.bigtable.gaxx.grpc.BigtableChannelPool$ReleasingClientCall$1.onClose(BigtableChannelPool.java:782) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:653) ~[gax-grpc-2.83.0.jar:2.83.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.opentelemetry.OpenTelemetryMetricsModule$MetricsClientInterceptor$1$1.onClose(OpenTelemetryMetricsModule.java:760) ~[grpc-opentelemetry-1.82.2.jar:1.82.2]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.cloud.bigtable.data.v2.stub.CookiesInterceptor$UpdateCookieListener.onClose(CookiesInterceptor.java:92) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.api.gax.grpc.GrpcLoggingInterceptor$1$1.onClose(GrpcLoggingInterceptor.java:100) ~[gax-grpc-2.83.0.jar:2.83.0]
    at io.grpc.xds.XdsNameResolver$ConfigSelector$1ClusterSelectionInterceptor$1$1.onClose(XdsNameResolver.java:527) ~[grpc-xds-1.82.2.jar:1.82.2]
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:566) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:734) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:715) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.82.2.jar:1.82.2]
    ... 3 more
Wrapped by: com.google.api.gax.rpc.CancelledException: io.grpc.StatusRuntimeException: CANCELLED: CANCELLED
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:108) ~[gax-2.83.0.jar:2.83.0]
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:88) ~[gax-grpc-2.83.0.jar:2.83.0]
    at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:84) ~[gax-grpc-2.83.0.jar:2.83.0]
    at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86) ~[gax-2.83.0.jar:2.83.0]
    at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:150) ~[gax-grpc-2.83.0.jar:2.83.0]
    ... (identical grpc interceptor chain as above, ending at XdsNameResolver$ClusterSelectionInterceptor:527 → ClientCallImpl.closeObserver:566)
Wrapped by: com.google.cloud.bigtable.data.v2.models.MutateRowsException: Some mutations failed to apply
    at com.google.cloud.bigtable.data.v2.models.MutateRowsException.create(MutateRowsException.java:53) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at com.google.cloud.bigtable.data.v2.stub.mutaterows.MutateRowsAttemptCallable.handleAttemptError(MutateRowsAttemptCallable.java:259) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at com.google.cloud.bigtable.data.v2.stub.mutaterows.MutateRowsAttemptCallable$3.apply(MutateRowsAttemptCallable.java:133) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction.apply(ApiFutures.java:396) ~[api-common-2.66.0.jar:2.66.0]
    at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:238) ~[guava-33.6.0-jre.jar:?]
    ... [application frames elided] ...

Investigation

I traced the producer of this status inside grpc-java v1.82.2:

core/src/main/java/io/grpc/internal/Http2ClientStreamTransportState.java:196 — statusFromTrailers:

private Status statusFromTrailers(Metadata trailers) {
  Status status = trailers.get(InternalStatus.CODE_KEY);
  if (status != null) {
    return status.withDescription(trailers.get(InternalStatus.MESSAGE_KEY));
  }
  ...

This is the only path in v1.82.2 that yields Status.CANCELLED with description exactly "CANCELLED", which means the server is sending HTTP/2 trailers with grpc-status: 1 and grpc-message: CANCELLED.

Questions

1. I though grpc java does not deliver CANCELLED error. is this intended?
2. the customer started seeing once they adopted directpath.