Improve Summary quantiles with DataSketches
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 32/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- java
- Domain
- observability-sre
Research direction
Start at Summary.observe() and trace the current CKMS-based quantile path in client_java. Review the ZooKeeper DataSketches Summary PR and compare a KLL-based approach using benchmark data, including accuracy, memory, and quantile visibility. Done means a documented direction and evidence for whether an opt-in implementation is worthwhile.
Written by the indexing model from the issue text.
Description
Summary.observe() can become expensive when quantiles are recorded at high frequency. This can make the current quantile path visible on hot request paths.
We saw this in ZooKeeper's Prometheus metrics path. In an internal ZooKeeper 3.9.2 fork, a version inspired by ZooKeeper's unmerged DataSketches Summary PR improved peak throughput by about 2x.
DataSketches KLL may be a useful way to improve this in client_java. The goal would be to reduce the cost of the observe path while keeping the external Summary behavior as close as practical.
This would not have to replace the current CKMS-based Summary immediately. DataSketches KLL has a different accuracy model, memory cost, and quantile visibility behavior, so an explicit opt-in path may be a better first step.
Initial questions:
- Does using DataSketches for Summary quantiles sound like a direction worth exploring?
- If so, would a separate opt-in artifact be a reasonable way to introduce it?
- What behavior details and benchmark data would be most useful before going further?
- Dominant language
- Java
- Stars
- 2.3k
- Forks
- 833
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 86
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from prometheus/client_java
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
prometheus/client_java#2416 · 1 comment ·
-
Switch Micrometer compatibility workflow to upstream once typed-descriptor path becomes default Open
Difficulty 1/5 Under an hour Newbie friendliness 86/100
prometheus/client_java#2182 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
prometheus/client_java#2306 · 9 comments · 4 reactions ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
prometheus/client_java#2075 · 10 comments · 1 reaction ·
-
prometheus/client_java#1915 · 1 assignee ·
All issues in prometheus/client_java
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
opensearch-project/k-NN#3597 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100