PromQL histogram queries fail when stored bounds end in +Inf (common in prom-client histograms)

Open
#21 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
clickhouse, go, prometheus

Research direction

Start with internal/snuffle/posthog_histogram.go and trace how postHogHistogramBuckets validates bounds and handles counts, then inspect the histogram query tests. Reproduce the failure with an explicit +Inf bound and confirm the query returns bucket series instead of a 422. The issue also describes count and bounds mismatches plus possible ingestion changes, so check their scope before deciding which fix to pursue.

Written by the indexing model from the issue text.

Description

Problem

PromQL queries over _bucket series fail with a 422 for histograms whose stored explicit bounds end in +Inf:

expanding series: handle clickhouse query row 1:
histogram "cdp_cyclotron_job_size_kb" bounds must be finite and strictly increasing

Prometheus accepts this data. Snuffle rejects the whole query, so any panel that uses histogram_quantile or a heatmap over this kind of histogram returns an error instead of data.

Why it happens

  1. Producers declare an explicit Infinity bucket. Many PostHog nodejs histograms do this. For example, cdp_cyclotron_job_size_kb in nodejs/src/cdp/services/job-queue/shared.ts uses buckets: [0, 16, 32, …, 4096, Infinity]. prom-client always adds its own le="+Inf", so the scrape exposes +Inf twice. At least ten other nodejs histograms follow the same pattern (kafka consumer, hog executor and persons metrics).
  2. OTLP ingestion stores the bounds as received. rust/capture-logs/src/metric_record.rs copies dp.explicit_bounds into the row with no normalisation, so +Inf can end up as an explicit bound. The remote-write path (capture-apm-metrics/src/histogram_assembly.rs) already dedups a repeated +Inf, but OTLP doesn't go through it.
  3. Snuffle is strict about it. postHogHistogramBuckets (internal/snuffle/posthog_histogram.go) returns invalidPostHogHistogram for any non-finite bound. A selector that names the metric exactly makes the series builder strict, so one bad sample fails the whole query instead of skipping that series.

Related: bounds and counts out of step

The same metric has also failed with:

histogram "cdp_cyclotron_job_size_kb" has 10 bounds but 12 bucket counts

metrics4_samples stores one anyLast(histogram_bounds) per series-hour but keeps every point's counts in histogram_counts_arr. If a series changes its bucket layout within an hour, the counts no longer match the bounds. PostHog/posthog#129793 adds per-point histogram_bounds_arr on the storage side. Snuffle needs a follow-up to read that column, using the length check the PR describes for older rows.

Reproduce

Against PostHog's own US project, where cdp_cyclotron_job_size_kb is ingested:

sum by (le) (increase(cdp_cyclotron_job_size_kb_bucket[1m]))

Expected: one series per le. Actual: a 422 with the error above. sum(rate(cdp_cyclotron_job_size_kb_sum[5m])) on the same metric works, so only the bucket expansion is affected.

To check the stored layout directly:

SELECT histogram_bounds, length(histogram_bounds), length(histogram_counts_arr[1])
FROM metrics4_samples
WHERE metric_name = 'cdp_cyclotron_job_size_kb'
ORDER BY time_bucket DESC
LIMIT 5

Potential fixes

  1. Snuffle (fixes data already stored): normalise before validating. If the last bound is +Inf, drop it and add the final overflow count (always 0) to the +Inf bucket. More generally, drop non-finite trailing bounds and fold their counts into the +Inf bucket, then validate what's left. This matches how Prometheus treats a duplicate +Inf.
  2. Snuffle (worth considering): when strict mode hits an invalid sample, skip it with a warning, as non-strict mode already does, instead of failing the whole query. Return a partial result with a warning rather than nothing.
  3. Ingestion (keeps new rows clean): apply the same normalisation in capture-logs OTLP histogram handling that histogram_assembly.rs applies to remote-write.
  4. Producers (hygiene): remove the explicit Infinity from the nodejs histogram bucket lists, since prom-client adds +Inf itself.
  5. Layout mismatch: read per-point bounds once PostHog/posthog#129793 lands.

Fix 1 alone unblocks the affected queries. Fixes 3 and 4 stop the problem at the source.


Drafted from a Claude Code session investigating a broken dashboard panel.

Dominant language
Go
Stars
3
Forks
0
Avg merge
18h 34m
Merged PRs (30d)
23

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PostHog/snuffle

All issues in PostHog/snuffle

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.