promql: instant selectors keep stale series, reading 3-5% above VictoriaMetrics

Open
#42 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
backend, databases

Research direction

Start by locating the PromQL instant-selector and rollup implementation, then read the tests covering lookback windows and stale series. Compare how instant selectors and default_rollup(m[5m]) handle stale samples and disappearing scrape targets. Done means the stated instant sum is within 1% of VictoriaMetrics on prod-us; the issue names no files or test commands.

Written by the indexing model from the issue text.

Description

Instant selectors on prod Snuffle (v0.101.0) return more series than VictoriaMetrics for metrics whose series disappear, such as pods that have terminated.

On Grafana prod-us, sum(kube_pod_container_resource_requests{resource="cpu"}) on posthog-prometheus vs victoriametrics:

time Snuffle VictoriaMetrics gap
2026-11-12T03:08Z (v0.100.0) 5812.8 5581.0 +4.2%
2026-11-13 ~02:52Z (v0.101.0) 5592.4 5324.8 +5.0%
2026-11-13 ~03:26Z (v0.101.0) 5304.6 5134.5 +3.3%

Why it looks like staleness: at 2026-11-12T03:08Z, VictoriaMetrics returned 5581 for default_rollup(m[5m]) and 5802 for last_over_time(m[5m]), so it drops series whose last sample is stale. Snuffle returned about the same number for both (5814.5 and 5812.8). Snuffle seems to keep any series that has a sample in the lookback window, even after the series has gone stale. It doesn't honour staleness markers, and it doesn't treat a series as gone when its scrape target disappears.

Not caused by #34 or #35: the gap is still there after the #35 revert (#41, shipped in v0.101.0). #34's sum(m[5m]) gives exactly the same numbers as the plain selector, so it inherits the gap without adding to it.

What done looks like: instant sum(kube_pod_container_resource_requests) on posthog-prometheus is within 1% of victoriametrics on prod-us.

🤖 Daniel's agent · perch shepherd

Dominant language
Go
Stars
3
Forks
0
Avg merge
18h 34m
Merged PRs (30d)
23

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PostHog/snuffle

All issues in PostHog/snuffle

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.