Keep remaining connector source attachments and release checks module-local

Open
  • Is something wrong?
#658 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@liuyongvs is already working on this.

Since Nov 12, 2026.

  • #645 by @liuyongvs — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
java
Domain
build-system, testing

Research direction

Start with PR #645 at final candidate 1c5c08c2fcc0687d72db2949521fda7f7e3f05a6, then inspect the Iceberg and Doris module source configuration and the release artifact/source attachment checker. Run the host-only release and source checks for Flink 1.18/JDK 11 and Flink 2.2/JDK 17. Done means full hosted qualification completes and the combined PR is merged; keep the issue open until it ships.

Written by the indexing model from the issue text.

Description

Reproduced problem

At PR #645 head 0ffb67562, the real Flink 2.2/JDK 17 release source attachment for Iceberg contains 70 Java files although the module owns only 40. The extra 30 files come from the inherited root compatibility source directory, for example tech.streamfusion.compat.FlinkCompat and tech.streamfusion.planner.NativeCanonicalSinkExecNode. They belong to core and are not packaged in Iceberg's executable JAR.

Iceberg and Doris already store implementations locally, but still inherit the root host source directory and rely on compiler/Javadoc/source filters. Doris's source filter currently prevents the same leak; it does not remove the accidental source-root dependency. Separately, the release artifact guard does not compare Iceberg/Doris classes with their local sources and omits Hudi from its checked release-module set, even though Hudi is shipped in the Flink 1.18 archive.

This is a source-publication and build-validation gap, not an observed SQL result mismatch. The current Iceberg, Doris and Hudi binary class/SPI ownership checks pass when invoked directly.

Solution

  • Disable the inherited root compatibility source execution in common-only Iceberg and Doris modules. Remove obsolete production compiler/Javadoc/source include filters; retain their existing host/JDK gates and opt-in test-source profiles.
  • Check source attachments against the exact selected module source paths and bytes. Reject missing, foreign, stale, wrong-host or duplicate Java inputs and executable/native payloads. Keep binary class/SPI checks and missing-module fallback unchanged.
  • Extend the real release guard to Iceberg and Doris on Flink 2.2 and Hudi on Flink 1.18, retaining Java-only native-payload checks and existing deployment identities.
  • Verify clean release/source/Javadoc builds, unchanged binary class/SPI/native contents and focused local connector tests.
  • Complete full hosted qualification and merge the combined PR.

Append focused commits to PR #645 so the final module-boundary qualification includes these previously missed artifacts. Do not change Java packages, JNI/DSO ownership, SQL admission, native execution contracts or checkpoint/recovery assertions. Core/runtime source consolidation remains separate work.

Implementation and validation

Implemented in two focused commits appended to PR #645: 43d0aaeb3 removes accidental root source inheritance and 1c5c08c2f extends actual-artifact/source-attachment validation. Final candidate: 1c5c08c2fcc0687d72db2949521fda7f7e3f05a6.

Clean unsigned release/source/Javadoc builds and complete host-only artifact/consumer-POM/loader checks pass on Flink 1.18/JDK 11 and Flink 2.2/JDK 17. Iceberg now publishes exactly 40 owned Java sources, Doris 18 and Hudi 25. The new checker rejects the actual pre-fix Iceberg source JAR's 30 foreign sources. All five ownership regression test methods pass, including both-host negative controls.

All 155 Iceberg/Doris/Hudi binary class/SPI/native entries remain byte-identical. All 316 focused Doris/Iceberg tests and 13 Hudi tests pass without skips, including missing-stock-connector fallback, native sink readback, checkpoint/rescaling recovery, stock/native Hudi reader restoration and batch/cancellation ownership. Strict docs pass. No SQL algorithm, admission, deployment identity or native library ownership changed.

Hosted checks for this exact final head:

Keep this issue open until the change ships; queued/running hosted checks are not reported as passed.

Dominant language
Java
Stars
157
Forks
13
Avg merge
7h 56m
Merged PRs (30d)
153

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from datafusion-contrib/StreamFusion

All issues in datafusion-contrib/StreamFusion

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.