bigtable: client-side metrics shutdown export fails with "written more frequently than the maximum sampling period" when the process exits within ~30s after a periodic export

Aberta
  • Algo errado?
#18,576 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

  • #18630 de @abhishekacharya200 — fechado sem integrar

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
67/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
google-cloud, python
Domínio
databases, observability

Direção de pesquisa

Start with gcp_exporter.py and trace how BigtableMetricsExporter exports metrics during periodic operation and shutdown; also check GoogleCloudMetricsHandler.close() as a possible shutdown entry point. Reproduce with the provided script, then verify that shutdown shortly after a periodic export no longer logs a failure while normal exports still work.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Determine this is the right repository
  • I determined this is the correct repository in which to report this bug.
Summary of the issue

Context

Since google-cloud-bigtable 2.46.0, the classic Table.read_rows goes through the data client, so client-side metrics are recorded and exported to Cloud Monitoring by BigtableMetricsExporter (via an OpenTelemetry MeterProvider + PeriodicExportingMetricReader with a hard-coded 60 s interval).

Expected Behavior:

A process that uses the client and then exits normally should not log an export failure.

Actual Behavior:

On process exit, MeterProvider.shutdown() (registered with atexit) triggers a final export. If the previous periodic export happened less than ~30 s earlier, Cloud Monitoring rejects the final write:

Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written:
timeSeries[0-5] (example metric.type="bigtable.googleapis.com/internal/client/operation_latencies", ...
resource.type="bigtable_client_raw", ...): write for resource failed: One or more points were written
more frequently than the maximum sampling period configured for the metric.
{Metric: bigtable.googleapis.com/internal/client/attempt_latencies,
 Timestamps: {Youngest Existing: '...:19.856', New: '...:25.411'}}

The public descriptors bigtable.googleapis.com/client/attempt_latencies and .../operation_latencies report samplePeriod: 60s. Because the periodic export runs every 60 s, any process that lives longer than ~60 s after its first Bigtable call has roughly a 50 % chance of hitting this on exit (whenever exit falls within ~30 s after a periodic export).

In serverless environments (Cloud Run services scaling down, Cloud Run Jobs finishing) this happens constantly. The warning is emitted with exc_info=True; in our Cloud Run setup the traceback ends up on stderr and is recorded as ERROR, so it creates recurring error-log noise even though nothing is functionally wrong.

Prior art in another Google client library

The same issue was reported for Go Spanner built-in metrics: https://github.com/googleapis/google-cloud-go/issues/12017 ("when we close the client there is scenario when metrics are trying to get exported within 1 minute interval window"). It was fixed in https://github.com/googleapis/google-cloud-go/pull/12266 ("do not export metrics during shutdown if prev export was less than 30 seconds ago"), which tracks lastExportedAt in the exporter and skips the shutdown export if the last export happened within half of the default sample period. Related OTel issue: https://github.com/open-telemetry/opentelemetry-go/issues/6677.

A similar guard in BigtableMetricsExporter (or in GoogleCloudMetricsHandler.close()) would fix this for Python.

API client name and version

google-cloud-bigtable 2.46.0 (reproduced) and 2.49.0 (same gcp_exporter.py / shutdown behavior; observed in production on Cloud Run). opentelemetry-sdk as pulled in by the package.

Reproduction steps: code

file: main.py

import sys
import time
import traceback

import google.auth
from google.cloud import bigtable
from google.cloud.bigtable.data._metrics.handlers import gcp_exporter as ge

PROJECT = "my-project"      # any project with a Bigtable instance
INSTANCE = "my-instance"
TABLE = "my-table"

T0 = time.time()

# Instrument export() only to show when it is called and with what result.
_orig = ge.BigtableMetricsExporter.export
def _export(self, metrics_data, *args, **kwargs):
    result = _orig(self, metrics_data, *args, **kwargs)
    print(f"export at t={time.time() - T0:5.1f}s -> {result}", flush=True)
    return result
ge.BigtableMetricsExporter.export = _export

# cloud-platform scope so that the exporter is not rejected with 403 (separate issue)
creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
client = bigtable.Client(project=PROJECT, credentials=creds)
table = client.instance(INSTANCE).table(TABLE)
for _ in table.read_rows(limit=3):
    pass

time.sleep(float(sys.argv[1]))  # how long the process keeps running before it exits
print(f"exit at t={time.time() - T0:5.1f}s", flush=True)
Reproduction steps: supporting files

None. Any table with a few rows.

Reproduction steps: actual results
$ python main.py 2      # exits before the first periodic export
exit at t=  2.4s
export at t=  2.4s -> MetricExportResult.SUCCESS

$ python main.py 75     # exits ~15 s after the periodic export
export at t= 62.4s -> MetricExportResult.SUCCESS     (periodic)
exit at t= 77.8s
Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written ...
  more frequently than the maximum sampling period configured for the metric ...
export at t= 77.8s -> MetricExportResult.FAILURE     (atexit shutdown)

$ python main.py 95     # exits ~35 s after the periodic export
export at t= 61.4s -> MetricExportResult.SUCCESS
exit at t= 97.0s
export at t= 97.0s -> MetricExportResult.SUCCESS
gap between periodic export and shutdown export result
~3 s, 5.5 s, 15.4 s 400 FailedPrecondition
35.6 s, 50.7 s success

Only these two exports happen, from the same exporter instance (no duplicate shutdown path). Calling the classic Client.close() does not help, since it does not close the internal data client, and closing the data client still performs the same final export.

Reproduction steps: expected results

The shutdown export is skipped (or delayed) when the previous export was too recent for the metric's sample period, as in the Go fix, so no error is logged on normal process exit.

OS & version + platform

Cloud Run (Python 3.12, gunicorn) and Cloud Run Jobs in asia-northeast1; reproduced locally on Linux (WSL2), Python 3.12.

Python environment

Python 3.12

Python dependencies

google-cloud-bigtable 2.46.0 / 2.49.0

Additional context

Workaround we use for now: setting OTEL_SDK_DISABLED=true before creating the client, which disables the client-side metrics entirely. It would be nice to have either the fix above or an official option to disable client-side metrics in the Python client.

Linguagem predominante
Python
Estrelas
5.4k
Forks
1.8k
Merge médio
2d 7h
PRs com merge (30d)
173

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de googleapis/google-cloud-python

Todas as issues de googleapis/google-cloud-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.