[CORE] Spark 3.3/3.2 leftovers needing an owner decision: bolt Makefile targets and a CH test fixture
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 38/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- clickhouse, cpp, spark
- Domain
- build-system, testing-qa
Research direction
Start with the Makefile targets and line references in the issue, comparing the working jar_spark34, jar_spark35, and test_spark35 targets with the obsolete profiles and artifact names. Then inspect cpp-ch/local-engine/tests/gtest_write_pipeline.cpp and its INCBIN loads to confirm whether the native_write_plan_1_spark33.json and sibling fixture are unused. Done requires owner decisions on the supported Makefile defaults and confirmation before removing fixtures.
Written by the indexing model from the issue text.
Description
Description
After the Spark 3.3 removal (#12807) and the follow-up cleanup (#13003), two Spark 3.2/3.3 references remain that I did not want to change unilaterally, because each needs a decision from the code's owner rather than a mechanical edit.
1. Makefile: dead -Pspark-3.3 / -Pspark-3.2 / -Pspark-3.2-las targets
The bolt Makefile (added with #12454, not wired into any CI workflow) still drives several targets off profiles that no longer exist:
jar(:158) builds with-Pspark-3.3jar-skip-check(:164),test(:211),clean(:224) build with-Pspark-3.2spark32-las(:169-170) builds with-Pspark-3.2-lasjar_spark33(:192-196) builds with-Pspark-3.3;jar_spark33is also in.PHONY(:83)fast-jar/zip/fast-zipand thejar/jar-skip-check/spark32-lasoutput names are hardcodedgluten-spark3.2_2.12-...(already inconsistent, sincejarbuilds 3.3 but names the artifact 3.2)
Because Maven only warns on an unknown -P, these fall through to the root-pom default properties (sparkbundle.version=3.5 / sparkshim=spark35 but spark.version=4.0.2) with -Denforcer.skip=true suppressing the guard, so make jar today produces a nonsense build rather than an error.
jar_spark34 / jar_spark35 / test_spark35 are the working targets. The fix is presumably to repoint the defaults to a supported version (3.5 matches the repo default and bolt builds on it) and drop the 3.2/3.3-only targets, but the default target and the output-naming scheme are bolt's call, so I am leaving it here rather than guessing. cc the bolt owners.
2. cpp-ch/local-engine/tests/json/native_write_plan_1_spark33.json
This fixture, and its sibling native_write_plan_1_spark35.json, appear unreferenced: the only INCBIN loads in gtest_write_pipeline.cpp are native_write_plan.json and native_write_one_partition.json, and a tree-wide grep for native_write_plan_1_spark finds no loader. If they are genuinely dead, the spark33 one is a 3.3 leftover and the spark35 one is orphaned too. I cannot build or run the ClickHouse tests locally, so I would rather a CH maintainer confirm these are unused before they are deleted. cc the ClickHouse owners.
Handled separately
The tools/workload/benchmark_velox/initialize.ipynb reference to spark-3.3.1 (it provisioned a Spark that Gluten can no longer run on) is a straightforward version bump and is fixed in the PR that references this issue, together with a stale Spark 3.3/3.4 comment in WriteFilesExecTransformer.
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day