[VL] Driver-side broadcast hash table build falls back when a build key is a generated alias
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- Half a day
- Newbie friendliness
- 62/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- scala, spark
- Domain
- data-engineering
Research direction
Start with buildHashTableWithRuntime in ColumnarBuildSideRelation and UnsafeColumnarBuildSideRelation, and compare it with the executor-side buildHashTable that #12407 already fixed by mapping each join key to the build output attribute with the same exprId. Trace the fallback through VeloxSparkPlanExecApi. Done when the json_tuple LATERAL VIEW query with driverSideBroadcastHashTableBuild=true builds on the driver without the 'falling back to executor-side build' log.
Written by the indexing model from the issue text.
Description
With spark.gluten.sql.columnar.backend.velox.driverSideBroadcastHashTableBuild=true, the driver-side hash table build fails when a join key reaches the build side under a generated alias, for example the json_tuple LATERAL VIEW query added in #12407. The native build reports Field not found: key#329. Available fields are: p_partkey#197, _post_11#329. VeloxSparkPlanExecApi catches the error, logs "Failed to build hash table on driver ... falling back to executor-side build", and each executor builds the table instead. Results are correct, but the driver-side build never takes effect for such joins.
#12407 fixed this for the executor-side buildHashTable in ColumnarBuildSideRelation and UnsafeColumnarBuildSideRelation by mapping each join key to the build output attribute with the same exprId. buildHashTableWithRuntime, which does the driver-side build, still passes the unmapped name.
This issue was written with the assistance of AI (Claude Opus).
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day