bigtable: client-side metrics shutdown export fails with "written more frequently than the maximum sampling period" when the process exits within ~30s after a periodic export

Offen
  • Stimmt etwas nicht?
#18,576 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

  • #18630 von @abhishekacharya200 — ohne Merge geschlossen

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
67/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
google-cloud, python
Bereich
databases, observability

Rechercherichtung

Start with gcp_exporter.py and trace how BigtableMetricsExporter exports metrics during periodic operation and shutdown; also check GoogleCloudMetricsHandler.close() as a possible shutdown entry point. Reproduce with the provided script, then verify that shutdown shortly after a periodic export no longer logs a failure while normal exports still work.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Determine this is the right repository
  • I determined this is the correct repository in which to report this bug.
Summary of the issue

Context

Since google-cloud-bigtable 2.46.0, the classic Table.read_rows goes through the data client, so client-side metrics are recorded and exported to Cloud Monitoring by BigtableMetricsExporter (via an OpenTelemetry MeterProvider + PeriodicExportingMetricReader with a hard-coded 60 s interval).

Expected Behavior:

A process that uses the client and then exits normally should not log an export failure.

Actual Behavior:

On process exit, MeterProvider.shutdown() (registered with atexit) triggers a final export. If the previous periodic export happened less than ~30 s earlier, Cloud Monitoring rejects the final write:

Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written:
timeSeries[0-5] (example metric.type="bigtable.googleapis.com/internal/client/operation_latencies", ...
resource.type="bigtable_client_raw", ...): write for resource failed: One or more points were written
more frequently than the maximum sampling period configured for the metric.
{Metric: bigtable.googleapis.com/internal/client/attempt_latencies,
 Timestamps: {Youngest Existing: '...:19.856', New: '...:25.411'}}

The public descriptors bigtable.googleapis.com/client/attempt_latencies and .../operation_latencies report samplePeriod: 60s. Because the periodic export runs every 60 s, any process that lives longer than ~60 s after its first Bigtable call has roughly a 50 % chance of hitting this on exit (whenever exit falls within ~30 s after a periodic export).

In serverless environments (Cloud Run services scaling down, Cloud Run Jobs finishing) this happens constantly. The warning is emitted with exc_info=True; in our Cloud Run setup the traceback ends up on stderr and is recorded as ERROR, so it creates recurring error-log noise even though nothing is functionally wrong.

Prior art in another Google client library

The same issue was reported for Go Spanner built-in metrics: https://github.com/googleapis/google-cloud-go/issues/12017 ("when we close the client there is scenario when metrics are trying to get exported within 1 minute interval window"). It was fixed in https://github.com/googleapis/google-cloud-go/pull/12266 ("do not export metrics during shutdown if prev export was less than 30 seconds ago"), which tracks lastExportedAt in the exporter and skips the shutdown export if the last export happened within half of the default sample period. Related OTel issue: https://github.com/open-telemetry/opentelemetry-go/issues/6677.

A similar guard in BigtableMetricsExporter (or in GoogleCloudMetricsHandler.close()) would fix this for Python.

API client name and version

google-cloud-bigtable 2.46.0 (reproduced) and 2.49.0 (same gcp_exporter.py / shutdown behavior; observed in production on Cloud Run). opentelemetry-sdk as pulled in by the package.

Reproduction steps: code

file: main.py

import sys
import time
import traceback

import google.auth
from google.cloud import bigtable
from google.cloud.bigtable.data._metrics.handlers import gcp_exporter as ge

PROJECT = "my-project"      # any project with a Bigtable instance
INSTANCE = "my-instance"
TABLE = "my-table"

T0 = time.time()

# Instrument export() only to show when it is called and with what result.
_orig = ge.BigtableMetricsExporter.export
def _export(self, metrics_data, *args, **kwargs):
    result = _orig(self, metrics_data, *args, **kwargs)
    print(f"export at t={time.time() - T0:5.1f}s -> {result}", flush=True)
    return result
ge.BigtableMetricsExporter.export = _export

# cloud-platform scope so that the exporter is not rejected with 403 (separate issue)
creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
client = bigtable.Client(project=PROJECT, credentials=creds)
table = client.instance(INSTANCE).table(TABLE)
for _ in table.read_rows(limit=3):
    pass

time.sleep(float(sys.argv[1]))  # how long the process keeps running before it exits
print(f"exit at t={time.time() - T0:5.1f}s", flush=True)
Reproduction steps: supporting files

None. Any table with a few rows.

Reproduction steps: actual results
$ python main.py 2      # exits before the first periodic export
exit at t=  2.4s
export at t=  2.4s -> MetricExportResult.SUCCESS

$ python main.py 75     # exits ~15 s after the periodic export
export at t= 62.4s -> MetricExportResult.SUCCESS     (periodic)
exit at t= 77.8s
Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written ...
  more frequently than the maximum sampling period configured for the metric ...
export at t= 77.8s -> MetricExportResult.FAILURE     (atexit shutdown)

$ python main.py 95     # exits ~35 s after the periodic export
export at t= 61.4s -> MetricExportResult.SUCCESS
exit at t= 97.0s
export at t= 97.0s -> MetricExportResult.SUCCESS
gap between periodic export and shutdown export result
~3 s, 5.5 s, 15.4 s 400 FailedPrecondition
35.6 s, 50.7 s success

Only these two exports happen, from the same exporter instance (no duplicate shutdown path). Calling the classic Client.close() does not help, since it does not close the internal data client, and closing the data client still performs the same final export.

Reproduction steps: expected results

The shutdown export is skipped (or delayed) when the previous export was too recent for the metric's sample period, as in the Go fix, so no error is logged on normal process exit.

OS & version + platform

Cloud Run (Python 3.12, gunicorn) and Cloud Run Jobs in asia-northeast1; reproduced locally on Linux (WSL2), Python 3.12.

Python environment

Python 3.12

Python dependencies

google-cloud-bigtable 2.46.0 / 2.49.0

Additional context

Workaround we use for now: setting OTEL_SDK_DISABLED=true before creating the client, which disables the client-side metrics entirely. It would be nice to have either the fix above or an official option to disable client-side metrics in the Python client.

Vorherrschende Sprache
Python
Sterne
5.4k
Forks
1.8k
Ø Merge
2 T. 7 Std.
Gemergte PRs (30 T.)
173

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus googleapis/google-cloud-python

Alle Issues in googleapis/google-cloud-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.