PromQL histogram queries fail when stored bounds end in +Inf (common in prom-client histograms)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- clickhouse, go, prometheus
- Domain
- backend, observability
Research direction
Start with internal/snuffle/posthog_histogram.go and trace how postHogHistogramBuckets validates bounds and handles counts, then inspect the histogram query tests. Reproduce the failure with an explicit +Inf bound and confirm the query returns bucket series instead of a 422. The issue also describes count and bounds mismatches plus possible ingestion changes, so check their scope before deciding which fix to pursue.
Written by the indexing model from the issue text.
Description
Problem
PromQL queries over _bucket series fail with a 422 for histograms whose stored explicit bounds end in +Inf:
expanding series: handle clickhouse query row 1:
histogram "cdp_cyclotron_job_size_kb" bounds must be finite and strictly increasing
Prometheus accepts this data. Snuffle rejects the whole query, so any panel that uses histogram_quantile or a heatmap over this kind of histogram returns an error instead of data.
Why it happens
- Producers declare an explicit
Infinitybucket. Many PostHog nodejs histograms do this. For example,cdp_cyclotron_job_size_kbinnodejs/src/cdp/services/job-queue/shared.tsusesbuckets: [0, 16, 32, …, 4096, Infinity]. prom-client always adds its ownle="+Inf", so the scrape exposes+Inftwice. At least ten other nodejs histograms follow the same pattern (kafka consumer, hog executor and persons metrics). - OTLP ingestion stores the bounds as received.
rust/capture-logs/src/metric_record.rscopiesdp.explicit_boundsinto the row with no normalisation, so+Infcan end up as an explicit bound. The remote-write path (capture-apm-metrics/src/histogram_assembly.rs) already dedups a repeated+Inf, but OTLP doesn't go through it. - Snuffle is strict about it.
postHogHistogramBuckets(internal/snuffle/posthog_histogram.go) returnsinvalidPostHogHistogramfor any non-finite bound. A selector that names the metric exactly makes the series builder strict, so one bad sample fails the whole query instead of skipping that series.
Related: bounds and counts out of step
The same metric has also failed with:
histogram "cdp_cyclotron_job_size_kb" has 10 bounds but 12 bucket counts
metrics4_samples stores one anyLast(histogram_bounds) per series-hour but keeps every point's counts in histogram_counts_arr. If a series changes its bucket layout within an hour, the counts no longer match the bounds. PostHog/posthog#129793 adds per-point histogram_bounds_arr on the storage side. Snuffle needs a follow-up to read that column, using the length check the PR describes for older rows.
Reproduce
Against PostHog's own US project, where cdp_cyclotron_job_size_kb is ingested:
sum by (le) (increase(cdp_cyclotron_job_size_kb_bucket[1m]))
Expected: one series per le. Actual: a 422 with the error above. sum(rate(cdp_cyclotron_job_size_kb_sum[5m])) on the same metric works, so only the bucket expansion is affected.
To check the stored layout directly:
SELECT histogram_bounds, length(histogram_bounds), length(histogram_counts_arr[1])
FROM metrics4_samples
WHERE metric_name = 'cdp_cyclotron_job_size_kb'
ORDER BY time_bucket DESC
LIMIT 5
Potential fixes
- Snuffle (fixes data already stored): normalise before validating. If the last bound is
+Inf, drop it and add the final overflow count (always 0) to the+Infbucket. More generally, drop non-finite trailing bounds and fold their counts into the+Infbucket, then validate what's left. This matches how Prometheus treats a duplicate+Inf. - Snuffle (worth considering): when strict mode hits an invalid sample, skip it with a warning, as non-strict mode already does, instead of failing the whole query. Return a partial result with a warning rather than nothing.
- Ingestion (keeps new rows clean): apply the same normalisation in
capture-logsOTLP histogram handling thathistogram_assembly.rsapplies to remote-write. - Producers (hygiene): remove the explicit
Infinityfrom the nodejs histogram bucket lists, since prom-client adds+Infitself. - Layout mismatch: read per-point bounds once PostHog/posthog#129793 lands.
Fix 1 alone unblocks the affected queries. Fixes 3 and 4 stop the problem at the source.
Drafted from a Claude Code session investigating a broken dashboard panel.
- Dominant language
- Go
- Stars
- 3
- Forks
- 0
- Avg merge
- 18h 34m
- Merged PRs (30d)
- 23
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PostHog/snuffle
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
Maintainers usually reply within 1 day
Similar issues
-
ssa: wait errors drop the kstatus message, so health check failures don't say whyPossibly taken @schdief claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Pin and verify Security Insights schema in lintinsights targetPossibly taken @imshubham22apr-gif claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
gemaraproj/gemara#618 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
a2aproject/a2a-go#597 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 4 days
-
scopes: GetAllowedTopics, GetProtectedTopics, and GetScopedTopics include empty topic names and do not trim whitespace charactersPossibly taken @patelshivani2283-lab claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day