feat(connectors): add Apache Spark Structured Streaming source

Aperta
  • Qualcosa non va?
#4,127 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@rohankumardubey ci sta già lavorando.

Dal 11/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

connectors enhancement
Description

Add an Apache Spark Structured Streaming source integration so Spark
applications can consume Apache Iggy streams through the DataFrame and
Dataset APIs.

This integration is listed in #2753 under Stream Processing Integration,
but it does not currently have a dedicated issue or implementation.

The initial scope would include:

  • A Spark Structured Streaming source, rather than the legacy DStream
    receiver API.
  • A Java or Scala connector module, following the existing Flink
    external-processor structure where reusable.
  • Mapping Iggy stream partitions to Spark input partitions.
  • Representing Iggy offsets through Spark's streaming offset API.
  • Committing Iggy progress only after Spark commits the completed
    micro-batch.
  • Exposing the message payload and relevant Iggy metadata, including
    stream, topic, partition, offset, and timestamp.
  • Configuration for server addresses, authentication, TLS, stream,
    topic, consumer identity or group, starting offset, and batch limits.
  • Unit tests, documentation, and an integration example using local
    Spark and Iggy.

The initial implementation would not include a Spark sink, legacy
DStream support, or continuous-processing mode. Those can be addressed
separately after the source semantics are validated.

Affected area/component

Connectors and Java SDK.

Proposed solution

Implement a Spark Data Source V2 micro-batch source, potentially under:

foreign/java/external-processors/iggy-connector-spark

The connector would implement Spark's MicroBatchStream integration
surface. Iggy partitions would be represented as Spark input
partitions, and Iggy offsets would be represented as Spark streaming
offsets. Spark checkpointing and the micro-batch commit callback would
be used to coordinate recovery and offset advancement.

The existing Flink connector can provide reusable project structure,
configuration, Java SDK, serialization, testing, and example patterns.

The delivery and recovery guarantees should be explicitly documented
and tested, particularly around retries after a Spark executor,
application, or Iggy connection failure.

Alternatives considered
  • Using Iggy's Kafka compatibility layer. This would be less native and
    would couple the integration to Kafka-compatible behavior.
  • Implementing a legacy Spark Streaming DStream receiver. Structured
    Streaming is Spark's current integration model and provides the
    appropriate offset and checkpoint interfaces.
  • Using a raw TCP or socket source. This would not provide the necessary
    partition and offset semantics.
Contribution

I am willing to implement this after the scope and design are confirmed.

Affected area / component

No response

Proposed solution

No response

Alternatives considered

No response

Contribution
  • I'm willing to submit a pull request to implement this feature
Good first issue
  • I think this could be a good first issue for a new contributor
Lingua principale
Rust
Stelle
5.2k
Fork
502
Merge medio
3g 11h
PR unite (30g)
143

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/iggy

Tutte le issue di apache/iggy

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.