SQL API: aggregates don't skip NULL values (COUNT(col) counts them, SUM and AVG throw NPE)

Open
#891 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@AarushShetty346 is already working on this.

Since Nov 18, 2026.

  • #893 by @AarushShetty346 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
20/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
java, spark, sql
Domain
backend, databases

Research direction

Start with AggregateAddCols.java, AggregateFunction.java, and AggregateGetResult.java at the linked lines; trace how NULL values are handled through aggregation and how AVG gets its count. Reproduce the listed queries on the Java platform, including a GROUP BY case, and check the behavior on Spark. Done means COUNT(col), SUM, and AVG match the expected results while COUNT(*) still counts every row.

Written by the indexing model from the issue text.

Description

Aggregates in the SQL API treat NULL as a value. COUNT(col) counts NULL rows, and SUM / AVG throw a NullPointerException when a column has a NULL.

In SQL, every aggregate except COUNT(*) ignores NULL values. There are three spots where this goes wrong:

  1. AggregateAddCols.java:53 sets every COUNT column to 1, so COUNT(col) is always the same as COUNT(*).
  2. AggregateFunction.java:118-124 picks the type from whichever side isn't null, then still passes the null to the function. For example, Integer::sum is applied to (1, null) and fails on unboxing.
  3. AggregateGetResult.java:55-56 divides the AVG sum by the row count, not by the number of non-null values. AVG would therefore be wrong even without the NPE.
To reproduce
col1:int,col2:string
1;a
;b
3;c

On the Java platform:

query result expected
SELECT COUNT(*) FROM fs.exampleNull 3 3
SELECT COUNT(col1) FROM fs.exampleNull 3 2
SELECT SUM(col1) FROM fs.exampleNull NPE 4
SELECT AVG(col1) FROM fs.exampleNull NPE 2.0
java.lang.NullPointerException
    at org.apache.wayang.api.sql.calcite.converter.functions.AggregateFunction.castAndMap(AggregateFunction.java:127)
    at org.apache.wayang.api.sql.calcite.converter.functions.AggregateFunction.apply(AggregateFunction.java:58)

The same thing happens with GROUP BY when a group has both NULL and non-NULL values. It fails in JavaReduceBy, and in Spark too.

Tested on main at 713faa7. I have a fix with tests and will open a PR for it.

Dominant language
Java
Stars
284
Forks
147
Avg merge
2d 13h
Merged PRs (30d)
13

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/wayang

All issues in apache/wayang

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.