[BUG] openlineage-flink ships unrelocated Jackson SPI files (Jdk8Module): breaks Apache Beam and any ObjectMapper.findModules() caller

integration/flink/build.gradleを読み、issueが先例として挙げているintegration/spark/build.gradleのshadowJar設定と比較してください。FlinkのshadowJar設定にmergeServiceFiles()を追加し、ビルドしたjarのMETA-INF/servicesエントリを確認して、Jacksonモジュールの検出時に再現手順が失敗しなくなったことを検証してください。

OpenLineage integration/client

Flink integration

OpenLineage version

io.openlineage:openlineage-flink 1.55.0 (latest). The same services file is present in every release I checked: 1.25.0, 1.30.0, 1.35.0, 1.40.0, 1.45.0, 1.48.0, 1.50.0 to 1.55.0.

Technology and package versions

• Java 17 (eclipse-temurin)
• Jackson 2.18.2, as used by the host application
• Apache Beam 2.74.0 (beam-sdks-java-core, beam-runners-flink-1.19)
• Apache Flink 1.19.2

Environment configuration

openlineage-flink jar on the job classpath. Nothing else is needed to trigger the problem.

Deployment details

Found with an Apache Beam pipeline run by the Beam FlinkRunner (Flink Kubernetes Operator, Application mode). The minimal reproduction below needs neither Flink nor Kubernetes.

Problem details

openlineage-flink relocates Jackson (relocate 'com.fasterxml.jackson', 'io.openlineage.flink.shaded.com.fasterxml.jackson'), but its shadowJar task does not call mergeServiceFiles(). The jar therefore ships Jackson's unrelocated SPI registrations, pointing at classes that only exist relocated:

File in openlineage-flink-1.55.0.jar	Content	Class in jar
META-INF/services/com.fasterxml.jackson.databind.Module	com.fasterxml.jackson.datatype.jdk8.Jdk8Module	only io/openlineage/flink/shaded/com/fasterxml/jackson/datatype/jdk8/Jdk8Module.class
META-INF/services/com.fasterxml.jackson.core.JsonFactory	com.fasterxml.jackson.core.JsonFactory	only the io/openlineage/flink/shaded/... copy
META-INF/services/com.fasterxml.jackson.core.ObjectCodec	com.fasterxml.jackson.dataformat.yaml.YAMLMapper	only the io/openlineage/flink/shaded/... copy

Any code on the same classpath that discovers Jackson modules through ServiceLoader with the real Jackson now fails, for example ObjectMapper.findModules() or findAndRegisterModules():

java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: Provider com.fasterxml.jackson.datatype.jdk8.Jdk8Module not found

Apache Beam calls ObjectMapper.findModules(...) inside PipelineOptionsFactory, which every Beam pipeline uses to parse its arguments. Adding openlineage-flink to a Beam-on-Flink job therefore stops the pipeline from starting at all, before any OpenLineage code runs.

This is the same problem that was fixed for Spark in #4227 / #4243 ("Fix SPI conflicts caused by package relocation"). That fix added mergeServiceFiles() to integration/spark/build.gradle. The Flink shadowJar block in integration/flink/build.gradle has the same Jackson relocation but no mergeServiceFiles().

What you think should happen instead

Adding openlineage-flink to a classpath should not break ServiceLoader lookups for the host application's own (unrelocated) Jackson. The service files should be relocated together with the classes, as mergeServiceFiles() does:

META-INF/services/io.openlineage.flink.shaded.com.fasterxml.jackson.databind.Module
  -> io.openlineage.flink.shaded.com.fasterxml.jackson.datatype.jdk8.Jdk8Module

Proposed fix, mirroring #4243:

 shadowJar {
     ...
     relocate 'com.fasterxml.jackson', 'io.openlineage.flink.shaded.com.fasterxml.jackson'
     ...
+    mergeServiceFiles()
 }

I verified the expected effect by patching the 1.55.0 jar to contain the three relocated service files instead of the unrelocated ones:

• the host application's ObjectMapper.findModules() returns [] and Beam's PipelineOptionsFactory works;
• OpenLineage's own shaded ObjectMapper.findModules() still discovers io.openlineage.flink.shaded...Jdk8Module.

How to reproduce

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <groupId>repro</groupId><artifactId>ol-flink-spi</artifactId><version>1</version>
  <properties><maven.compiler.release>17</maven.compiler.release></properties>
  <dependencies>
    <dependency><groupId>io.openlineage</groupId><artifactId>openlineage-flink</artifactId><version>1.55.0</version></dependency>
    <dependency><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-databind</artifactId><version>2.18.2</version></dependency>
    <dependency><groupId>org.apache.beam</groupId><artifactId>beam-sdks-java-core</artifactId><version>2.74.0</version></dependency>
    <dependency><groupId>org.slf4j</groupId><artifactId>slf4j-simple</artifactId><version>1.7.36</version></dependency>
  </dependencies>
</project>

src/main/java/repro/Main.java:

package repro;

import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.beam.sdk.options.PipelineOptionsFactory;

public class Main {
  public static void main(String[] args) {
    try {
      System.out.println("findModules: " + ObjectMapper.findModules());
    } catch (Throwable t) {
      System.out.println("findModules FAILED: " + t);
    }
    try {
      System.out.println("Beam options: " + PipelineOptionsFactory.fromArgs("--jobName=x").create().getJobName());
    } catch (Throwable t) {
      System.out.println("Beam PipelineOptionsFactory FAILED: " + t);
    }
  }
}

mvn -q compile dependency:build-classpath -Dmdep.outputFile=cp.txt
java -cp target/classes:$(cat cp.txt) repro.Main

Output with openlineage-flink 1.55.0:

findModules FAILED: java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: Provider com.fasterxml.jackson.datatype.jdk8.Jdk8Module not found
Beam PipelineOptionsFactory FAILED: java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: Provider com.fasterxml.jackson.datatype.jdk8.Jdk8Module not found

Output with the openlineage-flink dependency removed:

findModules: []
Beam options: x

The bad entry can also be seen directly in the jar:

unzip -p openlineage-flink-1.55.0.jar META-INF/services/com.fasterxml.jackson.databind.Module
# com.fasterxml.jackson.datatype.jdk8.Jdk8Module
unzip -l openlineage-flink-1.55.0.jar | grep 'Jdk8Module.class'
# io/openlineage/flink/shaded/com/fasterxml/jackson/datatype/jdk8/Jdk8Module.class   (no unrelocated copy)

Additional details

The workaround for users who build a fat jar is to drop that file when shading, for example with the Maven Shade plugin:

<filter>
  <artifact>io.openlineage:openlineage-flink</artifact>
  <excludes><exclude>META-INF/services/com.fasterxml.jackson.databind.Module</exclude></excludes>
</filter>

That only works when the user controls the shading. It does not help when the jar is dropped onto the classpath as is, for example in /opt/flink/lib or usrlib.

Do you plan to make this contribution yourself?

• I am interested in doing this work