promql: instant selectors keep stale series, reading 3-5% above VictoriaMetrics
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
Research direction
Start by locating the PromQL instant-selector and rollup implementation, then read the tests covering lookback windows and stale series. Compare how instant selectors and default_rollup(m[5m]) handle stale samples and disappearing scrape targets. Done means the stated instant sum is within 1% of VictoriaMetrics on prod-us; the issue names no files or test commands.
Written by the indexing model from the issue text.
Description
Instant selectors on prod Snuffle (v0.101.0) return more series than VictoriaMetrics for metrics whose series disappear, such as pods that have terminated.
On Grafana prod-us, sum(kube_pod_container_resource_requests{resource="cpu"}) on posthog-prometheus vs victoriametrics:
| time | Snuffle | VictoriaMetrics | gap |
|---|---|---|---|
| 2026-11-12T03:08Z (v0.100.0) | 5812.8 | 5581.0 | +4.2% |
| 2026-11-13 ~02:52Z (v0.101.0) | 5592.4 | 5324.8 | +5.0% |
| 2026-11-13 ~03:26Z (v0.101.0) | 5304.6 | 5134.5 | +3.3% |
Why it looks like staleness: at 2026-11-12T03:08Z, VictoriaMetrics returned 5581 for default_rollup(m[5m]) and 5802 for last_over_time(m[5m]), so it drops series whose last sample is stale. Snuffle returned about the same number for both (5814.5 and 5812.8). Snuffle seems to keep any series that has a sample in the lookback window, even after the series has gone stale. It doesn't honour staleness markers, and it doesn't treat a series as gone when its scrape target disappears.
Not caused by #34 or #35: the gap is still there after the #35 revert (#41, shipped in v0.101.0). #34's sum(m[5m]) gives exactly the same numbers as the plain selector, so it inherits the gap without adding to it.
What done looks like: instant sum(kube_pod_container_resource_requests) on posthog-prometheus is within 1% of victoriametrics on prod-us.
🤖 Daniel's agent · perch shepherd
- Dominant language
- Go
- Stars
- 3
- Forks
- 0
- Avg merge
- 18h 34m
- Merged PRs (30d)
- 23
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PostHog/snuffle
-
PromQL histogram queries fail when stored bounds end in +Inf (common in prom-client histograms)Open
Difficulty 4/5 3-5 days Newbie friendliness 52/100
Maintainers usually reply within 1 day
Similar issues
-
ssa: wait errors drop the kstatus message, so health check failures don't say whyPossibly taken @schdief claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Pin and verify Security Insights schema in lintinsights targetPossibly taken @imshubham22apr-gif claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
gemaraproj/gemara#618 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
a2aproject/a2a-go#597 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 4 days
-
scopes: GetAllowedTopics, GetProtectedTopics, and GetScopedTopics include empty topic names and do not trim whitespace charactersPossibly taken @patelshivani2283-lab claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day