Errors created via relabeling leak to remote-write

What did you do?

Per docs prometheus works in this order:

• scrapes target
• do relabel
• drop duplicates (logs them: https://github.com/prometheus/prometheus/issues/11725)
• ingest the rest, there cannot be duplicates in tsdb by design
• replicate local tsdb from WAL via remote-write to target endpoint

Unfortunately we see that duplicates also leak to remote-write. Way to reproduce:

• We will start prometheus, which would scrape nginx and remote-write to mimir:

version: '2'
services:

  prometheus:
    container_name: prometheus
    hostname: prometheus
    image: prom/prometheus:v2.43.0
    command:
      - --config.file=/cfg/prometheus.yml
    volumes:
      - ./prometheus.yml:/cfg/prometheus.yml
    ports:
      - '9090:9090'

  nginx:
    container_name: nginx
    hostname: nginx
    image: nginx
    volumes:
      - ./metrics:/usr/share/nginx/html/metrics
    ports:
      - '8081:80'
      
  mimir:
    container_name: mimir
    hostname: mimir
    image: grafana/mimir:2.7.1
    command:
      - -auth.multitenancy-enabled=false
      - -ingester.ring.replication-factor=1
    ports:
      - '8080:8080'

• prometheus config:

global:
  scrape_interval: 1s
scrape_configs:
  - job_name: own
    static_configs:
      - targets:
          - prometheus:9090
          - mimir:8080

  - job_name: duplicates
    static_configs:
      - targets:
          - nginx:80

remote_write:
  - url: http://mimir:8080/api/v1/push

• duplicate metrics:

$ for i in `seq 100000`; do echo "same_metric_name $i"; done > metrics
$ head metrics
same_metric_name 1
same_metric_name 2
same_metric_name 3
same_metric_name 4
same_metric_name 5
same_metric_name 6
same_metric_name 7
same_metric_name 8
same_metric_name 9
same_metric_name 10

• start this via docker-compose, and let's see prometheus metrics:

same_metric_name{instance="nginx:80", job="duplicates"} 1 # all lines deduplicated to just one metric, correct
prometheus_tsdb_head_series{instance="prometheus:9090", job="own"} 5169 # no 100k series in tsdb, correct
scrape_samples_scraped{instance="nginx:80", job="duplicates"} 100000 # hm, ok
scrape_samples_post_metric_relabeling{instance="nginx:80", job="duplicates"} 100000 # shouldn't it be just 1 here?
# sending speed via remote-write in samples/s
rate(prometheus_remote_storage_samples_total[1m]) 102902 # why? there are only 5k samples in head
# mimir failed samples/s
rate(cortex_ingester_ingested_samples_failures_total[1m]) 6883.5292321782945 # that actually matters, see below

• logs on mimir side:

mimir       | ts=2023-04-12T18:02:44.403379149Z caller=push.go:130 level=error user=anonymous msg="push error" err="rpc error: code = Code(400) desc = failed pushing to ingester: user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2023-04-12T18:02:44.39Z and is from series {__name__=\"same_metric_name\", instance=\"nginx:80\", job=\"duplicates\"}"

That seems as a bug on prometheus side for me. If there cannot be duplicates in tsdb, how do they appear when tsdb is being replicated via remote-write?

What did you expect to see?

I would expect that all duplicates issues would be handled on prometheus side, and then only data passed all validations would be replicated further via remote-write.

What did you see instead? Under which circumstances?

That does not sound like a big issue, but unfortunately it is harmful for people writing to mimir in distributor/ingester mode (i believe thanos-router/receive scheme would have the same issue).

• one of tenants got around 20 targets in his prometheus, exposing 400k duplicated metrics each. Not a big deal as we can ingest 9M ok-series easily
• any non-ok remote-write request starts to increase latency on incoming processing in mimir. In this case it has been 9M tainted samples, so large amount of incoming packets start to get 4xx
• usual incoming latency in our cluster 10-30ms, in this case it spiked to 1-3s
• mimir distributor component reads incoming http request to memory, parse remote-write proto and then replicates request to ingesters, awaits quorum number write confirmation and only then answers to incoming http request
• in case of increased latency it has to hold 3s of incoming 150Mbit/s flow in memory, it does back-pressure with currently established connections from prometheuses, but proms just spawn new shards if current slows down. Customers manage their prom configs, we cannot dictate remote_write.queue_config to them
• distributors start massively die of OOM, affects other. It even has context_deadline_timeout of 2s, so if we scale mem and replicas quickly, we anyway getting massive amounts of 5xx because ingester latency grows even higher than that. And 5xx triggers re-sends from prometheuses, so situation escalates quickly

System information

No response

Prometheus version

No response

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response