prometheus/prometheus

Prometheus Server and Prometheus Agent are not honoring the out_of_order_time_window and storage.agent.retention, respectivelyn.max-time values,

オープン

#11,919 opened on 2023/02/01

GitHub で見るコントリビューターガイド
 (8 件のコメント) (0 件のリアクション) (2 人の担当者)Go (10,408 件のフォーク)batch import
help wantednot-as-easy-as-it-looks

Repository metrics

Stars
 (64,042 個のスター)
PR merge metrics
 (平均マージ 11d 5h) (30d で 118 merged PRs)

説明

What did you do?

  1. Configured a Prometheus Server in the cloud with the .storage.tsdb.out_of_order_time_window: 24h config;
  2. Started a Prometheus Agent with --storage.agent.retention.min-time=6h and --storage.agent.retention.max-time=24h args in an edge host;

Then the edge host became offline (without Internet connection) for almost 9 hours.an

What did you expect to see?

I expected after the edge host came online, that the metrics stored in the WAL during the 9 offline hours, will be pushed to the Prometheus Server in the cloud.

What did you see instead? Under which circumstances?

Instead, on the Prometheus Server I saw a 9 hours gap.

image

Neither the metrics of 2 hours before the host came online, were pushed to the Prometheus Server! The host became online around 1:10PM, so, at least I had to see the metrics since 11:10AM.

System information

Ubuntu 22.04 amd64 (both cloud and edge host)

Prometheus version

Prometheus Server: 2.41.0
Prometheus Agent: 2.41.0

Prometheus configuration file

Prometheus Server:
global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m
storage:
  tsdb:
    out_of_order_time_window: 1d
Prometheus Agent:
global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m
  external_labels:
    edge_node: prom-agent
scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  static_configs:
  - targets:
    - localhost:9090
- job_name: kubernetes-apiservers
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: default;kubernetes;https
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
- job_name: kubernetes-nodes
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: kubernetes.default.svc:443
    action: replace
  - source_labels: [__meta_kubernetes_node_name]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/$1/proxy/metrics
    action: replace
  kubernetes_sd_configs:
  - role: node
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
- job_name: kubernetes-nodes-cadvisor
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: kubernetes.default.svc:443
    action: replace
  - source_labels: [__meta_kubernetes_node_name]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
    action: replace
  kubernetes_sd_configs:
  - role: node
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
- job_name: kubernetes-service-endpoints
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
    separator: ;
    regex: "true"
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    separator: ;
    regex: (https?)
    target_label: __scheme__
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    separator: ;
    regex: (.+?)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
    replacement: __param_$1
    action: labelmap
  - separator: ;
    regex: __meta_kubernetes_service_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_node_name]
    separator: ;
    regex: (.*)
    target_label: node
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
- job_name: kubernetes-service-endpoints-slow
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 5m
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    separator: ;
    regex: (https?)
    target_label: __scheme__
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    separator: ;
    regex: (.+?)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
    replacement: __param_$1
    action: labelmap
  - separator: ;
    regex: __meta_kubernetes_service_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_node_name]
    separator: ;
    regex: (.*)
    target_label: node
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
- job_name: prometheus-pushgateway
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    separator: ;
    regex: pushgateway
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: service
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
- job_name: kubernetes-services
  honor_labels: true
  honor_timestamps: true
  params:
    module:
    - http_2xx
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /probe
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: blackbox
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  - separator: ;
    regex: __meta_kubernetes_service_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - role: service
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
- job_name: kubernetes-pods
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
    separator: ;
    regex: "true"
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    separator: ;
    regex: (https?)
    target_label: __scheme__
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    separator: ;
    regex: (.+?)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
    replacement: __param_$1
    action: labelmap
  - separator: ;
    regex: __meta_kubernetes_pod_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: Pending|Succeeded|Failed|Completed
    replacement: $1
    action: drop
  kubernetes_sd_configs:
  - role: pod
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
- job_name: kubernetes-pods-slow
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 5m
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    separator: ;
    regex: (https?)
    target_label: __scheme__
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    separator: ;
    regex: (.+?)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
    replacement: __param_$1
    action: labelmap
  - separator: ;
    regex: __meta_kubernetes_pod_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: Pending|Succeeded|Failed|Completed
    replacement: $1
    action: drop
  kubernetes_sd_configs:
  - role: pod
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
remote_write:
- url: https://<REDDACTED>/promtest/api/v1/write
  remote_timeout: 30s
  name: cloud
  follow_redirects: true
  enable_http2: true
  queue_config:
    capacity: 2500
    max_shards: 200
    min_shards: 1
    max_samples_per_send: 500
    batch_send_deadline: 5s
    min_backoff: 30ms
    max_backoff: 5s
  metadata_config:
    send: true
    send_interval: 1m
    max_samples_per_send: 500

Logs

Prometheus Server:
...
ts=2023-02-01T01:36:53.190Z caller=manager.go:953 level=info component="rule manager" msg="Starting rule manager..."
ts=2023-02-01T13:10:14.399Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1675214886413 maxt=1675216800000 ulid=01GR6HJPA3PKN8MFCQSFBF6AAT duration=3.900613751s
ts=2023-02-01T13:10:14.764Z caller=head.go:1213 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=364.088899ms
ts=2023-02-01T13:10:15.443Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1675217598288 maxt=1675224000000 ulid=01GR6HJTFCQ1ZTDEG0R67BH82W duration=679.443594ms
ts=2023-02-01T13:10:15.490Z caller=head.go:1213 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=45.716946ms
ts=2023-02-01T13:10:16.376Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1675224000273 maxt=1675231200000 ulid=01GR6HJV620DD71W5XQKKW83V3 duration=885.662914ms
ts=2023-02-01T13:10:16.460Z caller=head.go:1213 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=84.165421ms
ts=2023-02-01T13:10:16.462Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=2 mint=1675231200000
ts=2023-02-01T13:10:16.597Z caller=head.go:1185 level=info component=tsdb msg="WAL checkpoint complete" first=0 last=2 duration=135.324346ms
ts=2023-02-01T13:10:16.598Z caller=db.go:1203 level=info component=tsdb msg="compact ooo head resulted in no blocks" duration=1.45µs
Prometheus Agent:
ts=2023-02-01T13:09:10.389Z caller=dedupe.go:112 component=remote level=warn remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Skipping resharding, last successful send was beyond threshold" lastSendTimestamp=1675225428 minSendTimestamp=1675256940
ts=2023-02-01T13:09:12.682Z caller=dedupe.go:112 component=remote level=warn remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Failed to send batch, retrying" err="Post \"https://<REDDACTED>/promtest/api/v1/write\": dial tcp: lookup <REDDACTED> on 10.80.18.10:53: server misbehaving"
ts=2023-02-01T13:09:20.389Z caller=dedupe.go:112 component=remote level=warn remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Skipping resharding, last successful send was beyond threshold" lastSendTimestamp=1675225428 minSendTimestamp=1675256950
ts=2023-02-01T13:09:30.389Z caller=dedupe.go:112 component=remote level=warn remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Skipping resharding, last successful send was beyond threshold" lastSendTimestamp=1675225428 minSendTimestamp=1675256960
ts=2023-02-01T13:09:40.389Z caller=dedupe.go:112 component=remote level=warn remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Skipping resharding, last successful send was beyond threshold" lastSendTimestamp=1675225428 minSendTimestamp=1675256970
ts=2023-02-01T13:09:50.388Z caller=dedupe.go:112 component=remote level=warn remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Skipping resharding, last successful send was beyond threshold" lastSendTimestamp=1675225428 minSendTimestamp=1675256980
ts=2023-02-01T13:10:00.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=1 to=200
ts=2023-02-01T13:10:02.512Z caller=dedupe.go:112 component=remote level=error remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="error tailing WAL" err="open data-agent/wal/00000009: no such file or directory"
ts=2023-02-01T13:10:07.515Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Replaying WAL" queue=cloud
ts=2023-02-01T13:10:08.978Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Done replaying WAL" duration=1.463466775s
ts=2023-02-01T13:11:20.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=200 to=105
ts=2023-02-01T13:11:30.389Z caller=dedupe.go:112 component=remote level=warn remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Skipping resharding, last successful send was beyond threshold" lastSendTimestamp=1675257076 minSendTimestamp=1675257080
ts=2023-02-01T13:11:40.388Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=105 to=61
ts=2023-02-01T13:11:50.389Z caller=dedupe.go:112 component=remote level=warn remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Skipping resharding, last successful send was beyond threshold" lastSendTimestamp=1675257090 minSendTimestamp=1675257100
ts=2023-02-01T13:12:00.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=61 to=38
ts=2023-02-01T13:12:20.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=38 to=24
ts=2023-02-01T13:12:40.388Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=24 to=16
ts=2023-02-01T13:13:00.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=16 to=11
ts=2023-02-01T13:13:20.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=11 to=7
ts=2023-02-01T13:13:50.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=7 to=4
ts=2023-02-01T13:14:20.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=4 to=2
ts=2023-02-01T13:15:00.389Z caller=dedupe.go:112 component=remote level=info remote_name=cloud url=https://<REDDACTED>/promtest/api/v1/write msg="Remote storage resharding" from=2 to=1

コントリビューターガイド

調査方針
Prometheusサーバーでのout of order time windowの適用方法と、エージェントモードでの保持最小/最大時間の処理方法を調査します。リモート書き込みパスとWALリプレイを確認し、順序外れのサンプルがなぜドロップまたは受け入れられないのかを確認します。TSDB取り込みコードをレビューして、時間ウィンドウが正しく適用されていることを確認します。
技術スタック
go
領域
observabilitybackend
Issue 種別
バグ
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
4
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
1-2日
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
アクティブ
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
明確
前提条件
GoPrometheus internalsTSDB
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
30