[BUILD] `iceberg` and `iceberg-test` profiles collide on one build-helper execution id, silently dropping `src-iceberg/test` sources

Open Beginner friendly
#12,933 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
86/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java, scala

Research direction

Start in the root pom.xml at the iceberg and iceberg-test build-helper-maven-plugin executions around lines 1833 and 1884. Give the two executions distinct ids, then run the reported ClickHouse test-compile command with both profiles and inspect the effective POM to confirm both source lists remain; done means the command succeeds and both named classes are emitted.

Written by the indexing model from the issue text.

Description

What happens

Both profiles in the root pom declare a build-helper-maven-plugin execution with the id add-iceberg-test-sources: pom.xml:1833 under iceberg, pom.xml:1884 under iceberg-test. Maven merges plugin executions by id, so when both profiles are active only one <sources> list survives.

help:effective-pom with -Pbackends-clickhouse -Pspark-3.5 -Pspark-ut -Pdelta -Piceberg -Piceberg-test resolves that execution to:

add-iceberg-test-sources
    src-iceberg-spark35/test/scala
    src-iceberg-spark35/test/java

The iceberg profile's four entries (src-iceberg/test/{scala,java} and src-iceberg${iceberg.binary.version}/test/{scala,java}) are gone. No warning is printed; the sources are simply not compiled.

Effect on ClickHouse: the combination cannot build

ClickHouseIcebergHiveTableSupport lives in backends-clickhouse/src-iceberg/test/scala, and both Java tests under src-iceberg-spark35/test/java instantiate it. The class therefore leaves the source path at exactly the moment the tests join it:

$ ./build/mvn -Pbackends-clickhouse -Pspark-3.5 -Pscala-2.13 -Pspark-ut -Pdelta \
      -Piceberg -Piceberg-test -DskipTests test-compile -pl backends-clickhouse
[ERROR] .../src-iceberg-spark35/test/java/.../TestPositionDeletesTableGluten.java:[41,18] cannot find symbol
[ERROR]   symbol:   class ClickHouseIcebergHiveTableSupport

Giving the iceberg-test execution a distinct id (add-iceberg-spark-test-sources) and changing nothing else makes the same command exit 0 and emit both ClickHouseIcebergHiveTableSupport.class and TestPositionDeletesTableGluten.class.

Effect on Velox: harmless today, which is why nobody noticed

Nothing in backends-velox needs both directories at once. src-iceberg/test holds VeloxIcebergSuite and VeloxTPCHIcebergSuite, which run in the jobs that pass -Piceberg without -Piceberg-test (velox_backend_x86.yml:941 and :1013, whose wildcardSuites includes org.apache.gluten), plus a @EnhancedFeaturesTest variant that velox_backend_enhanced.yml covers, a JUnit test surefire picks up, and one helper object. The 52 files under src-iceberg-spark34/test run in the three groups at :817, :821, :825 that pass both profiles.

Suggested fix

Rename the iceberg-test profile's execution so the two lists no longer overwrite each other. Both profiles are always passed together in every in-repo invocation, so the merged list is what the plugin was meant to produce.

Two things a reviewer will want to know about the blast radius. The three x86 groups and the 4.0 test-compile at :1277 would newly compile src-iceberg/test/**, so those suites run twice across the matrix; expect longer jobs rather than new failures. And src-iceberg${iceberg.binary.version}/test is empty in every module, so nothing else appears on the source path.

Only backends-velox and backends-clickhouse hold files under src-iceberg*/test, so the change is confined to those two modules.

This blocks the ClickHouse Iceberg Java test work in #12934.

Dominant language
Scala
Stars
1.6k
Forks
661
Avg merge
4d 9h
Merged PRs (30d)
74

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.