[VL] A broadcast anti join drops build rows that its filter can still match through coalesce or nvl
Maintainers usually reply within 3 days
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 32/100
Research direction
Start at SparkShims.isNullIntolerant(condition), which only inspects the root of the join filter; compare that decision with the native hash-table build path that drops null build rows. Use the issue's probe/build SQL to reproduce, then check both coalesce and nvl variants against Spark. Done when the broadcast anti join returns only (2, 1) with default config, and the driver-side build path agrees. Note linked PR #13712 is already open.
Written by the indexing model from the issue text.
Description
For a broadcast anti join with a filter, the native hash table build leaves out the build rows that are null in any build column the filter references, when it is told the filter is null-intolerant. Gluten decides that with SparkShims.isNullIntolerant(condition), which only checks the root of the condition. A filter like p.x > coalesce(b.y, 0) has a null-intolerant root (GreaterThan), so the rows with a null y are dropped, although coalesce(null, 0) is 0 and they can match.
With the default configuration:
-- probe(k, x) = (1, 5), (2, 1)
-- build(k, y) = (1, null), (2, 3)
SELECT * FROM probe WHERE NOT EXISTS
(SELECT 1 FROM build WHERE probe.k = build.k AND probe.x > coalesce(build.y, 0))
Spark returns (2, 1). Gluten returns (1, 5) and (2, 1). nvl, LEFT ANTI JOIN and the driver-side hash table build behave the same. It reproduces on v1.8.0-rc1.
This issue was written with the assistance of AI (Claude Opus).
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day