[VL] A broadcast anti join drops build rows that its filter can still match through coalesce or nvl

Open
#13,711 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

@LuciferYang is already working on this.

Since Nov 21, 2026.

  • #13712 by @LuciferYang — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
32/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
scala, sql
Domain
databases

Research direction

Start at SparkShims.isNullIntolerant(condition), which only inspects the root of the join filter; compare that decision with the native hash-table build path that drops null build rows. Use the issue's probe/build SQL to reproduce, then check both coalesce and nvl variants against Spark. Done when the broadcast anti join returns only (2, 1) with default config, and the driver-side build path agrees. Note linked PR #13712 is already open.

Written by the indexing model from the issue text.

Description

For a broadcast anti join with a filter, the native hash table build leaves out the build rows that are null in any build column the filter references, when it is told the filter is null-intolerant. Gluten decides that with SparkShims.isNullIntolerant(condition), which only checks the root of the condition. A filter like p.x > coalesce(b.y, 0) has a null-intolerant root (GreaterThan), so the rows with a null y are dropped, although coalesce(null, 0) is 0 and they can match.

With the default configuration:

-- probe(k, x) = (1, 5), (2, 1)
-- build(k, y) = (1, null), (2, 3)
SELECT * FROM probe WHERE NOT EXISTS
  (SELECT 1 FROM build WHERE probe.k = build.k AND probe.x > coalesce(build.y, 0))

Spark returns (2, 1). Gluten returns (1, 5) and (2, 1). nvl, LEFT ANTI JOIN and the driver-side hash table build behave the same. It reproduces on v1.8.0-rc1.

This issue was written with the assistance of AI (Claude Opus).

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.