PromQL histogram queries fail when stored bounds end in +Inf (common in prom-client histograms)
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 52/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- clickhouse, go, prometheus
- Área
- backend, observability
Línea de trabajo
Start with internal/snuffle/posthog_histogram.go and trace how postHogHistogramBuckets validates bounds and handles counts, then inspect the histogram query tests. Reproduce the failure with an explicit +Inf bound and confirm the query returns bucket series instead of a 422. The issue also describes count and bounds mismatches plus possible ingestion changes, so check their scope before deciding which fix to pursue.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Problem
PromQL queries over _bucket series fail with a 422 for histograms whose stored explicit bounds end in +Inf:
expanding series: handle clickhouse query row 1:
histogram "cdp_cyclotron_job_size_kb" bounds must be finite and strictly increasing
Prometheus accepts this data. Snuffle rejects the whole query, so any panel that uses histogram_quantile or a heatmap over this kind of histogram returns an error instead of data.
Why it happens
- Producers declare an explicit
Infinitybucket. Many PostHog nodejs histograms do this. For example,cdp_cyclotron_job_size_kbinnodejs/src/cdp/services/job-queue/shared.tsusesbuckets: [0, 16, 32, …, 4096, Infinity]. prom-client always adds its ownle="+Inf", so the scrape exposes+Inftwice. At least ten other nodejs histograms follow the same pattern (kafka consumer, hog executor and persons metrics). - OTLP ingestion stores the bounds as received.
rust/capture-logs/src/metric_record.rscopiesdp.explicit_boundsinto the row with no normalisation, so+Infcan end up as an explicit bound. The remote-write path (capture-apm-metrics/src/histogram_assembly.rs) already dedups a repeated+Inf, but OTLP doesn't go through it. - Snuffle is strict about it.
postHogHistogramBuckets(internal/snuffle/posthog_histogram.go) returnsinvalidPostHogHistogramfor any non-finite bound. A selector that names the metric exactly makes the series builder strict, so one bad sample fails the whole query instead of skipping that series.
Related: bounds and counts out of step
The same metric has also failed with:
histogram "cdp_cyclotron_job_size_kb" has 10 bounds but 12 bucket counts
metrics4_samples stores one anyLast(histogram_bounds) per series-hour but keeps every point's counts in histogram_counts_arr. If a series changes its bucket layout within an hour, the counts no longer match the bounds. PostHog/posthog#129793 adds per-point histogram_bounds_arr on the storage side. Snuffle needs a follow-up to read that column, using the length check the PR describes for older rows.
Reproduce
Against PostHog's own US project, where cdp_cyclotron_job_size_kb is ingested:
sum by (le) (increase(cdp_cyclotron_job_size_kb_bucket[1m]))
Expected: one series per le. Actual: a 422 with the error above. sum(rate(cdp_cyclotron_job_size_kb_sum[5m])) on the same metric works, so only the bucket expansion is affected.
To check the stored layout directly:
SELECT histogram_bounds, length(histogram_bounds), length(histogram_counts_arr[1])
FROM metrics4_samples
WHERE metric_name = 'cdp_cyclotron_job_size_kb'
ORDER BY time_bucket DESC
LIMIT 5
Potential fixes
- Snuffle (fixes data already stored): normalise before validating. If the last bound is
+Inf, drop it and add the final overflow count (always 0) to the+Infbucket. More generally, drop non-finite trailing bounds and fold their counts into the+Infbucket, then validate what's left. This matches how Prometheus treats a duplicate+Inf. - Snuffle (worth considering): when strict mode hits an invalid sample, skip it with a warning, as non-strict mode already does, instead of failing the whole query. Return a partial result with a warning rather than nothing.
- Ingestion (keeps new rows clean): apply the same normalisation in
capture-logsOTLP histogram handling thathistogram_assembly.rsapplies to remote-write. - Producers (hygiene): remove the explicit
Infinityfrom the nodejs histogram bucket lists, since prom-client adds+Infitself. - Layout mismatch: read per-point bounds once PostHog/posthog#129793 lands.
Fix 1 alone unblocks the affected queries. Fixes 3 and 4 stop the problem at the source.
Drafted from a Claude Code session investigating a broken dashboard panel.
- Lenguaje dominante
- Go
- Estrellas
- 3
- Forks
- 0
- Merge medio
- 18 h 34 min
- PR fusionados (30 d)
- 23
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Sin plantilla de pull request
- Sin guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de PostHog/snuffle
-
Dificultad 4/5 3-5 días Aptitud para principiantes 52/100
Los mantenedores suelen responder en 1 día
Todos los issues de PostHog/snuffle
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
github/github-mcp-server#3865 ·
Los mantenedores suelen responder en 2 días
-
kind/features
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
TencentBlueKing/bk-nodemgr#4017 ·
Los mantenedores suelen responder en 1 día
-
ssa: wait errors drop the kstatus message, so health check failures don't say whyPosiblemente ocupada @schdief la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
equinor/radix-operator#2066 ·
Los mantenedores suelen responder en 2 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 67/100
microsoft/terraform-provider-msgraph#188 · 1 reacción ·
Los mantenedores suelen responder en 1 día