histogram: numeric bars are a percentage of the bucketed values, the NA bar a percentage of all rows
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- data-visualization
Research direction
Start with numeric_histogram in histogram.py at the referenced line, and compare its population calculation with categorical_histogram; the issue notes that Polars shares this implementation. Check how the bucketed values and null count are calculated, then verify that bucket and NA bars use the same percentage base, including the example with 90% nulls; tail handling is also relevant to #1078.
Written by the indexing model from the issue text.
Description
Problem
numeric_histogram (histogram.py:148) draws each bucket as its share of the bucketed values (normalized_populations sums to 1 over the middle of the column) and the NA bar as nan_per * 100 (line 132), a share of all rows. The bars share one y axis, so a column that's 90% null shows ten buckets of about 10 beside an NA bar of 90, where each bucket holds about 1% of the rows. categorical_histogram divides both by length, so a categorical column with the same null share has its bars on one base.
import pandas as pd
from buckaroo.customizations.pd_stats_v2 import PD_ANALYSIS_V2
from buckaroo.pluggable_analysis_framework.stat_pipeline import StatPipeline
def histo(values):
sd, _ = StatPipeline(PD_ANALYSIS_V2, unit_test=False).process_df(pd.DataFrame({"a": values}))
return sd["a"]["histogram"]
h = histo([None] * 900 + [float(i) for i in range(100)])
print(sum(b.get("population", 0) for b in h), [b for b in h if "NA" in b])
# 98.0 [{'name': 'NA', 'NA': 90.0}]
print(histo([None] * 900 + ["x", "y"] * 50))
# [{'name': 'x', 'cat_pop': 5.0}, {'name': 'y', 'cat_pop': 5.0}, {'name': 'NA', 'NA': 90.0}]
Polars shares numeric_histogram and does the same.
Suggested fix
Scale the bucket populations by the share of rows that were bucketed, so every bar is a percentage of the column. The tails would need the same base (see #1078).
Context
Found while probing histogram edge cases after #1069. Reproduces on main at e88ace91.
- Dominant language
- Python
- Stars
- 685
- Forks
- 17
- Avg merge
- 21h 26m
- Merged PRs (30d)
- 22
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from buckaroo-data/buckaroo
-
Histograms JS
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
buckaroo-data/buckaroo#1089 · 1 comment ·
Maintainers usually reply within 1 day
-
Histograms
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
buckaroo-data/buckaroo#1087 · 1 comment ·
Maintainers usually reply within 1 day
-
Histograms
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
buckaroo-data/buckaroo#1086 · 1 comment ·
Maintainers usually reply within 1 day
-
Histograms
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
buckaroo-data/buckaroo#1083 · 1 comment ·
Maintainers usually reply within 1 day
-
Histograms
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
buckaroo-data/buckaroo#1082 · 1 comment ·
Maintainers usually reply within 1 day
All issues in buckaroo-data/buckaroo
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
UKGovernmentBEIS/optstop#10 ·
-
maintenance
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pollen-robotics/reachy_mini#1508 ·
Maintainers usually reply within 1 day
-
[BUG] `cloud-keys.env` values retain enclosing quotes causing 401 Unauthorized, and missing utf-8 encoding crashes on WindowsPossibly taken @divyanshu07-code claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
open-jarvis/OpenJarvis#1192 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sylvainmetayer/homelab#851 ·
Maintainers usually reply within 1 day