bigtable: client-side metrics shutdown export fails with "written more frequently than the maximum sampling period" when the process exits within ~30s after a periodic export

Đang mở
  • Có gì không ổn?
#18,576 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

  • #18630 của @abhishekacharya200 — đã đóng, không merge

Đánh giá

Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức phù hợp với người mới
67/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
google-cloud, python
Lĩnh vực
databases, observability

Hướng nghiên cứu

Start with gcp_exporter.py and trace how BigtableMetricsExporter exports metrics during periodic operation and shutdown; also check GoogleCloudMetricsHandler.close() as a possible shutdown entry point. Reproduce with the provided script, then verify that shutdown shortly after a periodic export no longer logs a failure while normal exports still work.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Determine this is the right repository
  • I determined this is the correct repository in which to report this bug.
Summary of the issue

Context

Since google-cloud-bigtable 2.46.0, the classic Table.read_rows goes through the data client, so client-side metrics are recorded and exported to Cloud Monitoring by BigtableMetricsExporter (via an OpenTelemetry MeterProvider + PeriodicExportingMetricReader with a hard-coded 60 s interval).

Expected Behavior:

A process that uses the client and then exits normally should not log an export failure.

Actual Behavior:

On process exit, MeterProvider.shutdown() (registered with atexit) triggers a final export. If the previous periodic export happened less than ~30 s earlier, Cloud Monitoring rejects the final write:

Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written:
timeSeries[0-5] (example metric.type="bigtable.googleapis.com/internal/client/operation_latencies", ...
resource.type="bigtable_client_raw", ...): write for resource failed: One or more points were written
more frequently than the maximum sampling period configured for the metric.
{Metric: bigtable.googleapis.com/internal/client/attempt_latencies,
 Timestamps: {Youngest Existing: '...:19.856', New: '...:25.411'}}

The public descriptors bigtable.googleapis.com/client/attempt_latencies and .../operation_latencies report samplePeriod: 60s. Because the periodic export runs every 60 s, any process that lives longer than ~60 s after its first Bigtable call has roughly a 50 % chance of hitting this on exit (whenever exit falls within ~30 s after a periodic export).

In serverless environments (Cloud Run services scaling down, Cloud Run Jobs finishing) this happens constantly. The warning is emitted with exc_info=True; in our Cloud Run setup the traceback ends up on stderr and is recorded as ERROR, so it creates recurring error-log noise even though nothing is functionally wrong.

Prior art in another Google client library

The same issue was reported for Go Spanner built-in metrics: https://github.com/googleapis/google-cloud-go/issues/12017 ("when we close the client there is scenario when metrics are trying to get exported within 1 minute interval window"). It was fixed in https://github.com/googleapis/google-cloud-go/pull/12266 ("do not export metrics during shutdown if prev export was less than 30 seconds ago"), which tracks lastExportedAt in the exporter and skips the shutdown export if the last export happened within half of the default sample period. Related OTel issue: https://github.com/open-telemetry/opentelemetry-go/issues/6677.

A similar guard in BigtableMetricsExporter (or in GoogleCloudMetricsHandler.close()) would fix this for Python.

API client name and version

google-cloud-bigtable 2.46.0 (reproduced) and 2.49.0 (same gcp_exporter.py / shutdown behavior; observed in production on Cloud Run). opentelemetry-sdk as pulled in by the package.

Reproduction steps: code

file: main.py

import sys
import time
import traceback

import google.auth
from google.cloud import bigtable
from google.cloud.bigtable.data._metrics.handlers import gcp_exporter as ge

PROJECT = "my-project"      # any project with a Bigtable instance
INSTANCE = "my-instance"
TABLE = "my-table"

T0 = time.time()

# Instrument export() only to show when it is called and with what result.
_orig = ge.BigtableMetricsExporter.export
def _export(self, metrics_data, *args, **kwargs):
    result = _orig(self, metrics_data, *args, **kwargs)
    print(f"export at t={time.time() - T0:5.1f}s -> {result}", flush=True)
    return result
ge.BigtableMetricsExporter.export = _export

# cloud-platform scope so that the exporter is not rejected with 403 (separate issue)
creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
client = bigtable.Client(project=PROJECT, credentials=creds)
table = client.instance(INSTANCE).table(TABLE)
for _ in table.read_rows(limit=3):
    pass

time.sleep(float(sys.argv[1]))  # how long the process keeps running before it exits
print(f"exit at t={time.time() - T0:5.1f}s", flush=True)
Reproduction steps: supporting files

None. Any table with a few rows.

Reproduction steps: actual results
$ python main.py 2      # exits before the first periodic export
exit at t=  2.4s
export at t=  2.4s -> MetricExportResult.SUCCESS

$ python main.py 75     # exits ~15 s after the periodic export
export at t= 62.4s -> MetricExportResult.SUCCESS     (periodic)
exit at t= 77.8s
Failed to export metrics to Cloud Monitoring: 400 One or more TimeSeries could not be written ...
  more frequently than the maximum sampling period configured for the metric ...
export at t= 77.8s -> MetricExportResult.FAILURE     (atexit shutdown)

$ python main.py 95     # exits ~35 s after the periodic export
export at t= 61.4s -> MetricExportResult.SUCCESS
exit at t= 97.0s
export at t= 97.0s -> MetricExportResult.SUCCESS
gap between periodic export and shutdown export result
~3 s, 5.5 s, 15.4 s 400 FailedPrecondition
35.6 s, 50.7 s success

Only these two exports happen, from the same exporter instance (no duplicate shutdown path). Calling the classic Client.close() does not help, since it does not close the internal data client, and closing the data client still performs the same final export.

Reproduction steps: expected results

The shutdown export is skipped (or delayed) when the previous export was too recent for the metric's sample period, as in the Go fix, so no error is logged on normal process exit.

OS & version + platform

Cloud Run (Python 3.12, gunicorn) and Cloud Run Jobs in asia-northeast1; reproduced locally on Linux (WSL2), Python 3.12.

Python environment

Python 3.12

Python dependencies

google-cloud-bigtable 2.46.0 / 2.49.0

Additional context

Workaround we use for now: setting OTEL_SDK_DISABLED=true before creating the client, which disables the client-side metrics entirely. It would be nice to have either the fix above or an official option to disable client-side metrics in the Python client.

Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
2 ngày 7 giờ
Pull request đã merge (30 ngày)
173

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của googleapis/google-cloud-python

Tất cả issue của googleapis/google-cloud-python

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.