bigtable: client-side metrics shutdown export fails with "written more frequently than the maximum sampling period" when the process exits within ~30s after a periodic export

Open
  • Is something wrong?
#18,576 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

  • #18630 by @abhishekacharya200 — closed without merging

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
67/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
google-cloud, python
Domain
databases, observability

Research direction

Start with gcp_exporter.py and trace how BigtableMetricsExporter exports metrics during periodic operation and shutdown; also check GoogleCloudMetricsHandler.close() as a possible shutdown entry point. Reproduce with the provided script, then verify that shutdown shortly after a periodic export no longer logs a failure while normal exports still work.

Written by the indexing model from the issue text.

Description

Determine this is the right repository
  • I determined this is the correct repository in which to report this bug.
Summary of the issue

Context

Since google-cloud-bigtable 2.46.0, the classic Table.read_rows goes through the data client, so client-side metrics are recorded and exported to Cloud Monitoring by BigtableMetricsExporter (via an OpenTelemetry MeterProvider + PeriodicExportingMetricReader with a hard-coded 60 s interval).

Expected Behavior:

A process that uses the client and then exits normally should not log an export failure.

Actual Behavior:

On process exit, MeterProvider.shutdown() (registered with atexit) triggers a final export. If the previous periodic export happened less than ~30 s earlier, Cloud Monitoring rejects the final write:

Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written:
timeSeries[0-5] (example metric.type="bigtable.googleapis.com/internal/client/operation_latencies", ...
resource.type="bigtable_client_raw", ...): write for resource failed: One or more points were written
more frequently than the maximum sampling period configured for the metric.
{Metric: bigtable.googleapis.com/internal/client/attempt_latencies,
 Timestamps: {Youngest Existing: '...:19.856', New: '...:25.411'}}

The public descriptors bigtable.googleapis.com/client/attempt_latencies and .../operation_latencies report samplePeriod: 60s. Because the periodic export runs every 60 s, any process that lives longer than ~60 s after its first Bigtable call has roughly a 50 % chance of hitting this on exit (whenever exit falls within ~30 s after a periodic export).

In serverless environments (Cloud Run services scaling down, Cloud Run Jobs finishing) this happens constantly. The warning is emitted with exc_info=True; in our Cloud Run setup the traceback ends up on stderr and is recorded as ERROR, so it creates recurring error-log noise even though nothing is functionally wrong.

Prior art in another Google client library

The same issue was reported for Go Spanner built-in metrics: https://github.com/googleapis/google-cloud-go/issues/12017 ("when we close the client there is scenario when metrics are trying to get exported within 1 minute interval window"). It was fixed in https://github.com/googleapis/google-cloud-go/pull/12266 ("do not export metrics during shutdown if prev export was less than 30 seconds ago"), which tracks lastExportedAt in the exporter and skips the shutdown export if the last export happened within half of the default sample period. Related OTel issue: https://github.com/open-telemetry/opentelemetry-go/issues/6677.

A similar guard in BigtableMetricsExporter (or in GoogleCloudMetricsHandler.close()) would fix this for Python.

API client name and version

google-cloud-bigtable 2.46.0 (reproduced) and 2.49.0 (same gcp_exporter.py / shutdown behavior; observed in production on Cloud Run). opentelemetry-sdk as pulled in by the package.

Reproduction steps: code

file: main.py

import sys
import time
import traceback

import google.auth
from google.cloud import bigtable
from google.cloud.bigtable.data._metrics.handlers import gcp_exporter as ge

PROJECT = "my-project"      # any project with a Bigtable instance
INSTANCE = "my-instance"
TABLE = "my-table"

T0 = time.time()

# Instrument export() only to show when it is called and with what result.
_orig = ge.BigtableMetricsExporter.export
def _export(self, metrics_data, *args, **kwargs):
    result = _orig(self, metrics_data, *args, **kwargs)
    print(f"export at t={time.time() - T0:5.1f}s -> {result}", flush=True)
    return result
ge.BigtableMetricsExporter.export = _export

# cloud-platform scope so that the exporter is not rejected with 403 (separate issue)
creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
client = bigtable.Client(project=PROJECT, credentials=creds)
table = client.instance(INSTANCE).table(TABLE)
for _ in table.read_rows(limit=3):
    pass

time.sleep(float(sys.argv[1]))  # how long the process keeps running before it exits
print(f"exit at t={time.time() - T0:5.1f}s", flush=True)
Reproduction steps: supporting files

None. Any table with a few rows.

Reproduction steps: actual results
$ python main.py 2      # exits before the first periodic export
exit at t=  2.4s
export at t=  2.4s -> MetricExportResult.SUCCESS

$ python main.py 75     # exits ~15 s after the periodic export
export at t= 62.4s -> MetricExportResult.SUCCESS     (periodic)
exit at t= 77.8s
Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written ...
  more frequently than the maximum sampling period configured for the metric ...
export at t= 77.8s -> MetricExportResult.FAILURE     (atexit shutdown)

$ python main.py 95     # exits ~35 s after the periodic export
export at t= 61.4s -> MetricExportResult.SUCCESS
exit at t= 97.0s
export at t= 97.0s -> MetricExportResult.SUCCESS
gap between periodic export and shutdown export result
~3 s, 5.5 s, 15.4 s 400 FailedPrecondition
35.6 s, 50.7 s success

Only these two exports happen, from the same exporter instance (no duplicate shutdown path). Calling the classic Client.close() does not help, since it does not close the internal data client, and closing the data client still performs the same final export.

Reproduction steps: expected results

The shutdown export is skipped (or delayed) when the previous export was too recent for the metric's sample period, as in the Go fix, so no error is logged on normal process exit.

OS & version + platform

Cloud Run (Python 3.12, gunicorn) and Cloud Run Jobs in asia-northeast1; reproduced locally on Linux (WSL2), Python 3.12.

Python environment

Python 3.12

Python dependencies

google-cloud-bigtable 2.46.0 / 2.49.0

Additional context

Workaround we use for now: setting OTEL_SDK_DISABLED=true before creating the client, which disables the client-side metrics entirely. It would be nice to have either the fix above or an official option to disable client-side metrics in the Python client.

Dominant language
Python
Stars
5.4k
Forks
1.8k
Avg merge
2d 7h
Merged PRs (30d)
173

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from googleapis/google-cloud-python

All issues in googleapis/google-cloud-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.