Profiling a file with a high-cardinality geography column takes quadratic time in _build_flags
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 84/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- javascript, pandas, python
- Domain
- data, performance
Research direction
Start with faircode/profiler.py _build_flags() and inspect the list membership check described in the issue; compare it with assets/profiler-engine.js buildFlags(). Read tests/test_profiler.py and add the suggested regression case with 50,000 distinct labels. Done when the lookup is efficient in both engines, the result still exposes a list, and the regression test completes within a few seconds.
Written by the indexing model from the issue text.
Description
Where: faircode/profiler.py _build_flags() (line 455 onward: g["label"] in d["under_represented"] is a list membership test inside a loop over groups), assets/profiler-engine.js buildFlags(), tests/test_profiler.py.
Current gap: geography columns are exempt from max_dimension_groups, so a zip column with tens of thousands of values is profiled in full, and flag building tests membership in a list for every group. With 200,000 rows and about 90,000 zip codes profile() spent 19.5 of 21 seconds in _build_flags; 500,000 rows took 26 seconds in total (the web engine 10).
Repro:
>>> import numpy as np, pandas as pd, time; from faircode import profile
>>> n = 200_000
>>> df = pd.DataFrame({"sex": np.random.choice(["m", "f"], n), "zip": np.random.randint(10000, 99999, n).astype(str)})
>>> t = time.time(); profile(df); round(time.time() - t, 1)
~20.0 # about 0.4 s without the zip column
Suggested approach: make under_represented a set for the lookup (keep the list in the result), do the same in JS, and add a regression test with 50,000 distinct labels that finishes within a few seconds.
- Dominant language
- HTML
- Stars
- 47
- Forks
- 45
- Avg merge
- 3h 24m
- Merged PRs (30d)
- 72
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from yakew7/Fair-Code
-
documentation good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
XLSX workbooks that use the 1904 date system get dates four years and a day early in the web engineOpenbug help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
enhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
Numeric column headers keep their type in Python (the integer 2020) and are text in the web enginePossibly taken @snehal1931 claimed this 1 day ago. Openbug help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
yakew7/Fair-Code#1045 · 2 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
JSON string cells "NA" and "" are missing in the web engine and ordinary values in PythonPossibly taken @snehal1931 claimed this 1 day ago. Openbug help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
yakew7/Fair-Code#1036 · 2 comments · 1 assignee ·
Maintainers usually reply within 1 day
All issues in yakew7/Fair-Code
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
nf-osi/nf-metadata-dictionary#1027 ·
Maintainers usually reply within 1 day
-
average_spectra doesn't normalize exact window functions (rows sum to the total weight instead of 1)Openarea: grouping type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
HERA-Team/hera_pspec#564 ·
Maintainers usually reply within 1 day
-
good first issue no hardware needed python
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
GageLawton/noise-cancellation#297 ·
Maintainers usually reply within 1 day
-
template-proposal
Difficulty 1/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
RobokopU24/ORION#445 ·
Maintainers usually reply within 1 day