Delta exponential histogram never goes back to the maximum scale after a wide-range interval
Maintainers usually reply within 2 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 88/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- observability
Research direction
Start in opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/aggregation.py, at _ExponentialBucketHistogramAggregation.collect and its DELTA→DELTA branch. Reproduce the issue with the provided script, then inspect the metrics tests for a suitable regression-test location. Done means later single-value delta intervals return to maximum scale without breaking the existing metrics test suite.
Written by the indexing model from the issue text.
Description
Describe your environment
OS: macOS (arm64)
Python version: 3.12
SDK version: main at 18d7e24 (opentelemetry-sdk installed from source)
API version: main at 18d7e24
What happened?
With DELTA temporality, the exponential histogram aggregation lowers its scale when an interval's values span a wide range, but never raises it again. Collection resets the buckets but not self._mapping (_ExponentialBucketHistogramAggregation.collect, the DELTA→DELTA branch in opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/aggregation.py). So every later delta point keeps the lowest scale any earlier interval needed, even when it holds a single value.
The SDK spec says: "When the histogram contains not more than one value in either of the positive or negative ranges, the implementation SHOULD use the maximum scale." A delta point covers only its own interval.
Steps to Reproduce
from opentelemetry.sdk.metrics import Histogram, MeterProvider
from opentelemetry.sdk.metrics.export import AggregationTemporality, InMemoryMetricReader
from opentelemetry.sdk.metrics.view import ExponentialBucketHistogramAggregation, View
reader = InMemoryMetricReader(preferred_temporality={Histogram: AggregationTemporality.DELTA})
provider = MeterProvider(
metric_readers=[reader],
views=[View(instrument_name="latency", aggregation=ExponentialBucketHistogramAggregation())],
)
hist = provider.get_meter("demo").create_histogram("latency")
def point():
return reader.get_metrics_data().resource_metrics[0].scope_metrics[0].metrics[0].data.data_points[0]
hist.record(1e-6)
hist.record(1e6)
print(point().scale) # interval 1: 2
for _ in range(3):
hist.record(5.0)
print(point().scale) # each later interval holds one value: 2, 2, 2
Expected Result
Each interval after the first holds a single value, so it's exported at the maximum scale (20 by default), with positive.offset = 2434718.
Actual Result
Every later interval is exported at scale 2, with positive.offset = 9. That bucket is (4.757, 5.657], a relative error of about 8.6% instead of about 3.3e-7 at scale 20. It never recovers.
In a worse case, one interval that records 1e-300 and 1e300 pins later intervals at scale -4. After that, four measurements of 10, 12, 15 and 20 ms all land in the single bucket (2^-16, 1].
Additional context
Adding self._mapping = self._new_mapping(self._max_scale) in that DELTA→DELTA branch, next to the bucket reset, restores the maximum scale for every interval. With it, the reproduction above prints 20 for each later interval, and the outlier case goes back to scale 7 with the four values in separate buckets. The 335 tests in opentelemetry-sdk/tests/metrics pass with and without the change, so none of them currently covers this.
For comparison, jmacd noted during review of #2964 that when the structure goes from 0 measurements to 1, "the scale is supposed to reset to the maximum scale again".
Found while auditing the metrics SDK with Claude Code. The outputs above come from running the script against main at 18d7e24.
Would you like to implement a fix?
Not at the moment. The one-line change above is what I tested, and anyone is welcome to pick it up.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 1k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 41
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from open-telemetry/opentelemetry-python
-
The changelog workflow excludes otelbot but not dependabot, so every Dependabot pull request fails the changelog checkPossibly taken @ocelotl claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 82/100
open-telemetry/opentelemetry-python#5825 ·
Maintainers usually reply within 2 days
-
W3CBaggagePropagator.extract ignores all but the first baggage headerPossibly taken @knowprasad claimed this 3 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
open-telemetry/opentelemetry-python#5811 ·
Maintainers usually reply within 2 days
-
Difficulty 1/5 Under an hour Newbie friendliness 70/100
open-telemetry/opentelemetry-python#5700 · 1 comment ·
Maintainers usually reply within 2 days
-
Tests in test_metric_reader_storage.py assert on process-global mock state instead of on the storage under testPossibly taken @ocelotl claimed this 47 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
open-telemetry/opentelemetry-python#5638 ·
Maintainers usually reply within 2 days
-
`SpanProcessor.on_end` exceptions escape cleanup and replace application exceptionsPossibly taken @somuai claimed this 51 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
open-telemetry/opentelemetry-python#5624 ·
Maintainers usually reply within 2 days
All issues in open-telemetry/opentelemetry-python
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
yzhao062/awesome-auditable-ai#45 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
NVIDIA/flare-local-inference#90 ·
Maintainers usually reply within 1 day
-
area/streaming comp/gateway P2 platform/telegram type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#144897 ·
Maintainers usually reply within 1 day
-
request-theme
Difficulty 1/5 Under an hour Newbie friendliness 66/100
LizardByte/ThemerrDB#9575 ·
Maintainers usually reply within 1 day