[BUG] [SPARK] Kafka streaming inputs empty on Spark 3.4/3.5: StreamingDataSourceV2ScanRelation builder only registered for Spark 4

Start by comparing Spark35DatasetBuilderFactory and Spark34DatasetBuilderFactory with Spark40DatasetBuilderFactory, then read StreamingDataSourceV2ScanRelationDatasetBuilder and the Spark 4 tests added for #5101. Check how the Spark 3.5 factory handles the reported StreamingDataSourceV2ScanRelation plan. Done means Kafka inputs are present alongside outputs on Spark 3.5, with tests covering the reported plan and Databricks-style shaded partition keys.

OpenLineage integration/client

Spark integration

OpenLineage version

• io.openlineage:openlineage-spark_2.13:1.55.0
• io.openlineage:transports-s3:1.55.0 (S3 transport)

Technology and package versions

Component	Version
Platform	Databricks
Apache Spark (failing case)	3.5.2 (processing_engine.version on emitted events)
Apache Spark (control — works)	4.0.0 (same OpenLineage JAR, same job)
Scala	2.13 (openlineage-spark_2.13 on driver)
OpenLineage Spark adapter	1.55.0
Kafka connector	Spark kafka / KafkaSourceProvider (KafkaTable, MicroBatchScan)
Databricks Runtime	e.g. 16.4 LTS (Spark 3.5.2) vs DBR with Spark 4.0

Environment configuration

spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener,com.databricks.backend.daemon.driver.DBCEventLoggingListener
spark.openlineage.transport.type=s3
spark.openlineage.transport.bucketName=bucket
spark.openlineage.transport.fileNamePrefix=openlineage/events/
spark.openlineage.namespace=workspace_name
spark.openlineage.version=v1

Deployment details

Init script: copies openlineage-spark_2.13-1.55.0.jar and transports-s3-1.55.0.jar to /databricks/jars.

Pipeline: readStream.format("kafka") → decode → writeStream.toTable(<catalog>.<schema>.<table>).

Problem details

On Databricks with Apache Spark 3.5.x, OpenLineage 1.55.0 emits streaming COMPLETE events with Unity Catalog / Delta outputs but inputs: [] for the Kafka source. The same Structured Streaming job on Spark 4.0 with the same OpenLineage version produces both Kafka inputs and UC/Delta outputs.

This is a follow-up to #5099: #5101 fixed shaded TopicPartition on the Spark 4 code path (StreamingDataSourceV2ScanRelationDatasetBuilder + KafkaMicroBatchStreamStrategy). On Spark 3.5.x, that builder is not registered in Spark35DatasetBuilderFactory, so #5101 never runs even though the logical plan already contains StreamingDataSourceV2ScanRelation.

On both Spark versions the analyzed logical plan uses the same shape: WriteToMicroBatchDataSourceV1 over StreamingDataSourceV2ScanRelation … KafkaTable (not legacy StreamingDataSourceV2Relation). See SPARK-46777, #4018.

Actual behavior

Spark 3.5.2 (failing):

• Stream and sink succeed; UC table is updated.
• OpenLineage COMPLETE has inputs: [] and populated outputs (openlineageAdapterVersion: 1.55.0).
• Driver Log4j: no Selected this strategy: KafkaMicroBatchStreamStrategy (input builder not invoked). No #5099-style TopicPartitionProxy error on 1.55.0 — strategy simply never runs.

Spark 4.0 (control — same job, same OL 1.55.0):

• COMPLETE includes Kafka inputs (subscribed topic) and UC/Delta outputs.
• Log4j may show Selected this strategy: KafkaMicroBatchStreamStrategy (after #5101).

Logical plan (representative)

Plan parity (redacted; same job on DBR Spark 3.5.2 vs 4.0):

Plan element	Spark 3.5.2	Spark 4.0
Sink	WriteToMicroBatchDataSourceV1 → UC/Delta DeltaSink	Same
Source logical node	StreamingDataSourceV2ScanRelation[...] KafkaTable	Same
Source physical node	MicroBatchScan[...] KafkaSourceProvider$KafkaScan	Same
Legacy StreamingDataSourceV2Relation	Not present	Not present

WriteToMicroBatchDataSourceV1 `<catalog>`.`<schema>`.`<table>`, DeltaSink[s3://<uc-storage>/...], ...
+- ...
    +- StreamingDataSourceV2ScanRelation[key#..., value#..., topic#..., partition#..., offset#..., ...] KafkaTable

Physical:
+- MicroBatchScan[...] class org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan

Root cause (analysis)

OpenLineage picks a DatasetBuilderFactory from SPARK_VERSION (DatasetBuilderFactoryProvider):

• 3.4.x → Spark34DatasetBuilderFactory
• 3.5.x → Spark35DatasetBuilderFactory
• 4.x → Spark40DatasetBuilderFactory

Spark34DatasetBuilderFactory / Spark35DatasetBuilderFactory register WriteToMicroBatchDataSourceV1DatasetBuilder (streaming outputs, #4018) but do not register io.openlineage.spark40.agent.lifecycle.plan.StreamingDataSourceV2ScanRelationDatasetBuilder, where KafkaMicroBatchStreamStrategy and #5101 run.

Spark40DatasetBuilderFactory registers both the micro-batch output builder and StreamingDataSourceV2ScanRelationDatasetBuilder. That matches observed behavior: outputs-only on 3.5.2, full lineage on 4.0 with the same plan nodes.

Spark3VisitorFactoryImpl still adds legacy StreamingDataSourceV2RelationVisitor, which does not match StreamingDataSourceV2ScanRelation — no fallback.

Spark	DatasetBuilderFactory	Kafka inputs on COMPLETE	UC/Delta outputs
3.5.2	Spark35DatasetBuilderFactory	Empty	Present
4.0.0	Spark40DatasetBuilderFactory	Present	Present

What you think should happen instead

Streaming COMPLETE events include a Kafka input dataset for the subscribed topic (namespace from bootstrap servers), plus UC/Delta output datasets — on Spark 3.4+ when the plan uses StreamingDataSourceV2ScanRelation, not only on Spark 4.

How to reproduce

1. On Databricks DBR with Spark 3.5.x, install openlineage-spark_2.13-1.55.0 on the driver classpath and enable the OpenLineage Spark listener.
2. Run Structured Streaming: Kafka source (subscribe, kafka.bootstrap.servers, SASL; use kafkashaded.* in JAAS per Databricks Kafka FAQ) → UC/Delta via writeStream.toTable(...).
3. Confirm the stream reads data and the UC table is updated.
4. Inspect a streaming COMPLETE event: outputs populated, inputs: [].
5. Repeat steps 1–4 on Databricks DBR with Spark 4.0 and the same OpenLineage version: COMPLETE includes Kafka inputs and UC/Delta outputs (compare df.explain(true) — same StreamingDataSourceV2ScanRelation / MicroBatchScan shape).

Additional details

Related work

• #5099 / #5101 — Databricks shaded TopicPartition (fixed on Spark 4 builder path)

Possible fix (suggestion)

• Register StreamingDataSourceV2ScanRelationDatasetBuilder (or move to a shared module) in Spark34DatasetBuilderFactory and Spark35DatasetBuilderFactory, mirroring Spark40DatasetBuilderFactory, or
• Register Kafka streaming input extraction for all Spark ≥ 3.4 when the plan contains StreamingDataSourceV2ScanRelation + KafkaMicroBatchStream.

Add unit/integration tests for Spark 3.5 (and 3.4 if supported) with StreamingDataSourceV2ScanRelation and Databricks-style kafkashaded.* partition keys (#5101 tests currently target the Spark 4 builder module).

Do you plan to make this contribution yourself?

• I am interested in doing this work