BurntSushi/xsv

feature: binned histograms

Open

#84 opened on Jun 13, 2017

View on GitHub
 (3 comments) (2 reactions) (0 assignees)Rust (9,730 stars) (315 forks)batch import
enhancementhelp wanted

Description

Histograms could be done in numerous ways. Here are some thoughts:

  • like most of xsv, should operate over huge tables with a single pass
  • .idx files could store metadata helpful for binning? eg, min and max values per column
  • "chose N evenly spaced numerical bins" seems to require more than one pass (or keeping all values in memory). Keeping a tree of round-sized bins and merging them when the tree gets too big would avoid that
  • logarithmic bins
  • power-of-two or power-of-1024 (eg, for file sizes)
  • binning of strings or decimal numbers by prefix
  • "other" / "NaN" / null bins
  • csv/tsv output by default, then a separate mode like xsv table to pretty print bars

Contributor guide

Tech stack
rust
Domain
clidata
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
RustCSV parsinghistogram algorithmsmemory efficient processing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate single pass histogram binning approaches for large CSV data, considering memory constraints. Review comments in issue #84 for additional requirements like logarithmic bins and null handling. Existing .idx files may provide min/max metadata. The implementation should output CSV/TSV and support a pretty print mode similar to `xsv table`.