bigtable: client-side metrics shutdown export fails with "written more frequently than the maximum sampling period" when the process exits within ~30s after a periodic export

Ouverte
  • Quelque chose ne va pas ?
#18,576 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

  • #18630 par @abhishekacharya200 — fermée sans fusion

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
67/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
google-cloud, python
Domaine
databases, observability

Piste de recherche

Start with gcp_exporter.py and trace how BigtableMetricsExporter exports metrics during periodic operation and shutdown; also check GoogleCloudMetricsHandler.close() as a possible shutdown entry point. Reproduce with the provided script, then verify that shutdown shortly after a periodic export no longer logs a failure while normal exports still work.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Determine this is the right repository
  • I determined this is the correct repository in which to report this bug.
Summary of the issue

Context

Since google-cloud-bigtable 2.46.0, the classic Table.read_rows goes through the data client, so client-side metrics are recorded and exported to Cloud Monitoring by BigtableMetricsExporter (via an OpenTelemetry MeterProvider + PeriodicExportingMetricReader with a hard-coded 60 s interval).

Expected Behavior:

A process that uses the client and then exits normally should not log an export failure.

Actual Behavior:

On process exit, MeterProvider.shutdown() (registered with atexit) triggers a final export. If the previous periodic export happened less than ~30 s earlier, Cloud Monitoring rejects the final write:

Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written:
timeSeries[0-5] (example metric.type="bigtable.googleapis.com/internal/client/operation_latencies", ...
resource.type="bigtable_client_raw", ...): write for resource failed: One or more points were written
more frequently than the maximum sampling period configured for the metric.
{Metric: bigtable.googleapis.com/internal/client/attempt_latencies,
 Timestamps: {Youngest Existing: '...:19.856', New: '...:25.411'}}

The public descriptors bigtable.googleapis.com/client/attempt_latencies and .../operation_latencies report samplePeriod: 60s. Because the periodic export runs every 60 s, any process that lives longer than ~60 s after its first Bigtable call has roughly a 50 % chance of hitting this on exit (whenever exit falls within ~30 s after a periodic export).

In serverless environments (Cloud Run services scaling down, Cloud Run Jobs finishing) this happens constantly. The warning is emitted with exc_info=True; in our Cloud Run setup the traceback ends up on stderr and is recorded as ERROR, so it creates recurring error-log noise even though nothing is functionally wrong.

Prior art in another Google client library

The same issue was reported for Go Spanner built-in metrics: https://github.com/googleapis/google-cloud-go/issues/12017 ("when we close the client there is scenario when metrics are trying to get exported within 1 minute interval window"). It was fixed in https://github.com/googleapis/google-cloud-go/pull/12266 ("do not export metrics during shutdown if prev export was less than 30 seconds ago"), which tracks lastExportedAt in the exporter and skips the shutdown export if the last export happened within half of the default sample period. Related OTel issue: https://github.com/open-telemetry/opentelemetry-go/issues/6677.

A similar guard in BigtableMetricsExporter (or in GoogleCloudMetricsHandler.close()) would fix this for Python.

API client name and version

google-cloud-bigtable 2.46.0 (reproduced) and 2.49.0 (same gcp_exporter.py / shutdown behavior; observed in production on Cloud Run). opentelemetry-sdk as pulled in by the package.

Reproduction steps: code

file: main.py

import sys
import time
import traceback

import google.auth
from google.cloud import bigtable
from google.cloud.bigtable.data._metrics.handlers import gcp_exporter as ge

PROJECT = "my-project"      # any project with a Bigtable instance
INSTANCE = "my-instance"
TABLE = "my-table"

T0 = time.time()

# Instrument export() only to show when it is called and with what result.
_orig = ge.BigtableMetricsExporter.export
def _export(self, metrics_data, *args, **kwargs):
    result = _orig(self, metrics_data, *args, **kwargs)
    print(f"export at t={time.time() - T0:5.1f}s -> {result}", flush=True)
    return result
ge.BigtableMetricsExporter.export = _export

# cloud-platform scope so that the exporter is not rejected with 403 (separate issue)
creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
client = bigtable.Client(project=PROJECT, credentials=creds)
table = client.instance(INSTANCE).table(TABLE)
for _ in table.read_rows(limit=3):
    pass

time.sleep(float(sys.argv[1]))  # how long the process keeps running before it exits
print(f"exit at t={time.time() - T0:5.1f}s", flush=True)
Reproduction steps: supporting files

None. Any table with a few rows.

Reproduction steps: actual results
$ python main.py 2      # exits before the first periodic export
exit at t=  2.4s
export at t=  2.4s -> MetricExportResult.SUCCESS

$ python main.py 75     # exits ~15 s after the periodic export
export at t= 62.4s -> MetricExportResult.SUCCESS     (periodic)
exit at t= 77.8s
Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written ...
  more frequently than the maximum sampling period configured for the metric ...
export at t= 77.8s -> MetricExportResult.FAILURE     (atexit shutdown)

$ python main.py 95     # exits ~35 s after the periodic export
export at t= 61.4s -> MetricExportResult.SUCCESS
exit at t= 97.0s
export at t= 97.0s -> MetricExportResult.SUCCESS
gap between periodic export and shutdown export result
~3 s, 5.5 s, 15.4 s 400 FailedPrecondition
35.6 s, 50.7 s success

Only these two exports happen, from the same exporter instance (no duplicate shutdown path). Calling the classic Client.close() does not help, since it does not close the internal data client, and closing the data client still performs the same final export.

Reproduction steps: expected results

The shutdown export is skipped (or delayed) when the previous export was too recent for the metric's sample period, as in the Go fix, so no error is logged on normal process exit.

OS & version + platform

Cloud Run (Python 3.12, gunicorn) and Cloud Run Jobs in asia-northeast1; reproduced locally on Linux (WSL2), Python 3.12.

Python environment

Python 3.12

Python dependencies

google-cloud-bigtable 2.46.0 / 2.49.0

Additional context

Workaround we use for now: setting OTEL_SDK_DISABLED=true before creating the client, which disables the client-side metrics entirely. It would be nice to have either the fix above or an official option to disable client-side metrics in the Python client.

Langage dominant
Python
Étoiles
5.4k
Forks
1.8k
Merge moyen
2 j 7 h
PR mergées (30 j)
173

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de googleapis/google-cloud-python

Toutes les issues de googleapis/google-cloud-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.