[CORE] Spark 3.3/3.2 leftovers needing an owner decision: bolt Makefile targets and a CH test fixture

Open
#13,088 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
clickhouse, cpp, spark

Research direction

Start with the Makefile targets and line references in the issue, comparing the working jar_spark34, jar_spark35, and test_spark35 targets with the obsolete profiles and artifact names. Then inspect cpp-ch/local-engine/tests/gtest_write_pipeline.cpp and its INCBIN loads to confirm whether the native_write_plan_1_spark33.json and sibling fixture are unused. Done requires owner decisions on the supported Makefile defaults and confirmation before removing fixtures.

Written by the indexing model from the issue text.

Description

Description

After the Spark 3.3 removal (#12807) and the follow-up cleanup (#13003), two Spark 3.2/3.3 references remain that I did not want to change unilaterally, because each needs a decision from the code's owner rather than a mechanical edit.

1. Makefile: dead -Pspark-3.3 / -Pspark-3.2 / -Pspark-3.2-las targets

The bolt Makefile (added with #12454, not wired into any CI workflow) still drives several targets off profiles that no longer exist:

  • jar (:158) builds with -Pspark-3.3
  • jar-skip-check (:164), test (:211), clean (:224) build with -Pspark-3.2
  • spark32-las (:169-170) builds with -Pspark-3.2-las
  • jar_spark33 (:192-196) builds with -Pspark-3.3; jar_spark33 is also in .PHONY (:83)
  • fast-jar / zip / fast-zip and the jar/jar-skip-check/spark32-las output names are hardcoded gluten-spark3.2_2.12-... (already inconsistent, since jar builds 3.3 but names the artifact 3.2)

Because Maven only warns on an unknown -P, these fall through to the root-pom default properties (sparkbundle.version=3.5 / sparkshim=spark35 but spark.version=4.0.2) with -Denforcer.skip=true suppressing the guard, so make jar today produces a nonsense build rather than an error.

jar_spark34 / jar_spark35 / test_spark35 are the working targets. The fix is presumably to repoint the defaults to a supported version (3.5 matches the repo default and bolt builds on it) and drop the 3.2/3.3-only targets, but the default target and the output-naming scheme are bolt's call, so I am leaving it here rather than guessing. cc the bolt owners.

2. cpp-ch/local-engine/tests/json/native_write_plan_1_spark33.json

This fixture, and its sibling native_write_plan_1_spark35.json, appear unreferenced: the only INCBIN loads in gtest_write_pipeline.cpp are native_write_plan.json and native_write_one_partition.json, and a tree-wide grep for native_write_plan_1_spark finds no loader. If they are genuinely dead, the spark33 one is a 3.3 leftover and the spark35 one is orphaned too. I cannot build or run the ClickHouse tests locally, so I would rather a CH maintainer confirm these are unused before they are deleted. cc the ClickHouse owners.

Handled separately

The tools/workload/benchmark_velox/initialize.ipynb reference to spark-3.3.1 (it provisioned a Spark that Gluten can no longer run on) is a straightforward version bump and is fixed in the PR that references this issue, together with a stale Spark 3.3/3.4 comment in WriteFilesExecTransformer.

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.