OTLP gRPC exporter reads RetryInfo only from the google.rpc.retryinfo-bin trailer, not from the status details

Open
#5,942 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
grpc, python

Research direction

Start in exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py around the RetryInfo trailer handling at lines 432–440. Read the gRPC status details handling and the test in exporter/opentelemetry-exporter-otlp-proto-grpc/tests/test_otlp_exporter_mixin.py, which currently mocks only google.rpc.retryinfo-bin. Done means RetryInfo in grpc-status-details-bin sets the retry delay, with the existing trailer behavior covered as a fallback.

Written by the indexing model from the issue text.

Description

Describe your environment

opentelemetry-exporter-otlp-proto-grpc 1.45.2 (the code is the same on main).

What happened?

The OTLP gRPC exporter reads the server's RetryInfo only from a trailer named google.rpc.retryinfo-bin:

https://github.com/open-telemetry/opentelemetry-python/blob/ad755945ad69008219e661c33ff1d72ff1bbcb51/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py#L432-L440

The OTLP specification tells a server to send RetryInfo as a detail of the gRPC status (Failures, OTLP/gRPC Throttling). Its sample server code uses status.New(...).WithDetails(&errdetails.RetryInfo{...}). gRPC sends those details in the grpc-status-details-bin trailer, as an encoded google.rpc.Status. A server that follows the specification, such as one written with grpc-go or the OpenTelemetry Collector's receiver, sends no google.rpc.retryinfo-bin trailer.

So the Python exporter never sees the delay that such a server asks for. It falls back to its own backoff of 2**n * uniform(0.8, 1.2) seconds. The OTel Go SDK, the Collector's exporter and the .NET exporter read the delay from the status details.

Steps to Reproduce

  1. Run a gRPC OTLP server whose Export returns UNAVAILABLE (or RESOURCE_EXHAUSTED) built with status.New(code, msg).WithDetails(&errdetails.RetryInfo{RetryDelay: durationpb.New(3 * time.Second)}).
  2. Export a span with OTLPSpanExporter to it.
  3. Log the time of each attempt on the server.

Expected Result

The exporter waits the 3 s that RetryInfo asks for before it retries.

Actual Result

The exporter retries after about 1 s, then 2 s, then 4 s: its own backoff. The RetryInfo in grpc-status-details-bin is ignored.

Additional context

A fix could decode grpc-status-details-bin as google.rpc.status_pb2.Status (or use grpc_status.rpc_status.from_call), take the RetryInfo from its details, and keep the google.rpc.retryinfo-bin lookup as a fallback. The test at exporter/opentelemetry-exporter-otlp-proto-grpc/tests/test_otlp_exporter_mixin.py mocks only the google.rpc.retryinfo-bin trailer.

Dominant language
Python
Stars
2.7k
Forks
1.1k
Avg merge
4d 13m
Merged PRs (30d)
38

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from open-telemetry/opentelemetry-python

All issues in open-telemetry/opentelemetry-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.