[Feature Request]: [Python] Add Variance, StandardDeviation, Covariance, and LinearRegression to apache_beam.transforms.combiners

Open
  • Is something wrong?
#41,171 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data

Research direction

Start by reading the Python SDK's apache_beam.transforms.combiners implementation and its existing combiner tests. The issue specifies four statistical combiners, stable distributed accumulator merging, multiple input shapes, and a structured regression result, so scope the work and tests around those requirements. Done means the new combiners handle the requested inputs and produce the stated statistics across distributed merges.

Written by the indexing model from the issue text.

Description

P2 python
What would you like to happen?

Currently, apache_beam.transforms.combiners provides standard aggregation transforms such as Count, Mean, Min, Max, Top, and Sample. However, common statistical and bivariate regression aggregations are missing from the Python SDK:

  • Variance (sample and population)
  • StandardDeviation (sample and population)
  • Covariance (sample and population for (x, y) pairs or structured records)
  • LinearRegression (ordinary least squares slope, intercept, r, r2, and means)

While users can compute sample means with combiners.Mean.Globally(), calculating variance, standard deviation, covariance, or linear regression currently requires writing custom multi-pass accumulators or pulling in NumPy/SciPy inside custom CombineFns, which can be tricky to make numerically stable across distributed worker combines.

We propose adding first-class, numerically stable combiners in apache_beam.transforms.combiners:

import apache_beam as beam
from apache_beam.transforms import combiners

# 1. Variance & Standard Deviation
with beam.Pipeline() as p:
  numbers = p | beam.Create([10.0, 20.0, 30.0, 40.0, 50.0])
  sample_var = numbers | 'SampleVar' >> combiners.Variance.Globally()
  pop_stddev = numbers | 'PopStdDev' >> combiners.StandardDeviation.Globally(sample=False)

# 2. Covariance
with beam.Pipeline() as p:
  points = p | beam.Create([(1.0, 2.0), (2.0, 4.0), (3.0, 5.0), (4.0, 7.0)])
  cov = points | combiners.Covariance.Globally()

# 3. Ordinary Least Squares Linear Regression
with beam.Pipeline() as p:
  ad_data = p | beam.Create([
      {'spend': 10, 'revenue': 25.0},
      {'spend': 20, 'revenue': 41.0},
      {'spend': 30, 'revenue': 62.0},
  ])
  model = ad_data | combiners.LinearRegression.Globally(x='spend', y='revenue')
  # Emits RegressionResult(count=3, slope=1.85, intercept=5.67, r=0.998, r2=0.997, mean_x=20.0, mean_y=42.67)
Key Technical Details
  1. Numerical Stability: Uses single-pass, numerically stable Welford updating equations for both variance and bivariate cross-products, and Chan et al. pairwise merging for distributed accumulator merges (merge_accumulators).
  2. Flexible Input Shapes:
    • Variance and StandardDeviation accept numeric types (int, float, np.number).
    • Covariance and LinearRegression accept 2-tuples/sequences (x, y) or key-value/dict mappings with x='col_x', y='col_y'.
  3. Structured Output: LinearRegression returns a typed NamedTuple (RegressionResult) containing (count, slope, intercept, r, r2, mean_x, mean_y).
  4. Runner & Streaming Parity: Fully pure Python, runner-agnostic, zero extra dependencies, and works out-of-the-box in batch, streaming, DirectRunner, Prism, and Dataflow.
Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components
  • Component: Python SDK
  • Component: beam-core
Dominant language
Java
Stars
8.7k
Forks
4.8k
Avg merge
2d 9h
Merged PRs (30d)
368

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/beam

All issues in apache/beam

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.