connectors: Delta Sink - production feasibility improvements

Aperta
  • Qualcosa non va?
#4,105 4 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@mmmmxa ci sta già lavorando.

Dal 9/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

connectors
Description

As mentioned a number of time on Discord, for the last few weeks I was working on trying to make Delta sink run on a real EC2 instance writing to S3. The implementation turned out to have a substantial amount of issues that render the sink not practical to use in production:

  1. S3 storage type mandates access key and secret key, use of which is highly discouraged by AWS. The modern practice is to assign a role to an instance that is going to perform the write, for which you should simply write into the destination bucket without any credentials.
  2. The existence of the destination delta table is not checked, which causes obscure errors when the table doesn't exist in the destination S3 URL. The table is not automatically created either, but this is likely to be a future feature.
  3. When set in production, aws_s3_endpoint_url is confusing S3 API to understand the region setting which doesn't allow the code to open the table for writing. This setting should be documented better (and studied as there seem to be a couple of other quirks) as removing it usually fixes the problem.
  4. If a connectors runtime instantiated with a config containing topics that do not exist yet, and then tries to connect to the iggy-server, it stalls indefinitely and the sink is not writing anymore, which seems to be an offset issue, but the cause of this is yet to be studied in more detail.
  5. Current implementation allows to have a setup when no fields from the JSONs inside of the topic and the fields (columns) of the table schema overlap, causing the system to write a table full of nulls. This needs to be studied further.
  6. The defaults for batch_length and poll_interval have to be adjusted for Delta tables. Writing the data with the standard batch_length == 100 and poll_interval == 5ms is going to create lots and lots of small files in Delta, which is causing very slow reading on the side of Delta readers. The default of batch_length == 10000 and poll_interval == 3s is much more adequate for most use cases as it's going to create appropriately-sized files.
  7. The underlying IggyError ID is not propagated for the user to understand what is the problem which leads to cryptic error messages: iggy-connect | 2026-09-10T14:17:06.835367Z ERROR iggy_connectors::sink: Failed to initialize sink connector with ID: 1: "Failed to set up sink consumers: Iggy error". Skipping...

All of these will likely be bundled in a number of different PRs but the grouping is still TBD.

Affected area / component

Connectors

Proposed solution

PR 1:

  1. Stop mandating the S3 access and secret keys. Mention the two access options in the docs. Suggest the sufficient policy configuration to be given to the instance.
  2. Check the table existence after opening, write a test for it.
  3. Study and document the usage of aws_s3_endpoint_url configuration setting.
  4. Change defaults in the documentation. One possibility could also be to have a cutoff by bytes as the recommended size of an individual file for a Delta table is 128 - 1024 MB, but this is a larger feature that needs to be thought through way more.

Future PRs (exact split is TBD)

  1. The stalling after failure to find a stream / topic by the connectors runtime has to be studied and most likely there needs to be an error with a retry period or something like this.
  2. Make the error message more elaborate in case of IggyError returned from the client.
  3. Study the schema coercion process further to see the exact nature of the problem. I think it would make sense to implement an error on no fields overlapping as this results in all-null records made into the delta table.
Alternatives considered

No response

Contribution
  • I'm willing to submit a pull request to implement this feature
Good first issue
  • I think this could be a good first issue for a new contributor
Lingua principale
Rust
Stelle
5.2k
Fork
502
Merge medio
3g 11h
PR unite (30g)
143

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/iggy

Tutte le issue di apache/iggy

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.