Decode the remaining Kafka key and metadata shapes natively

Open
  • Is something wrong?
#655 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
apache, java, kafka, rust
Domain
backend, data-engineering, stream-processing

Research direction

Read the connector admission policy, docs/connectors/kafka.md, and the work tracked in #657; this issue depends on #539 and #540. Start by choosing one listed Kafka shape and add parity tests against Flink's deserializers plus end-to-end decode-counter evidence on both lines. Done means the admission policy's measurements pass and the fallback list is updated for each shape that lands.

Written by the indexing model from the issue text.

Description

Priority: P3.

Problem

After #539 and #540, these Kafka source shapes still stay on Flink's decoder at planning time, so a native pipeline over them pays Flink's decode and a RowData-to-Arrow transpose:

  • key and value formats decoded by different native libraries, such as a JSON key over an Avro value or a bare Avro key over an Avro Confluent value;
  • Protobuf keys, and any key over a Protobuf value;
  • connector metadata over Debezium Avro Confluent and Protobuf values;
  • a CDC format's own value.* metadata, such as Canal's value.database;
  • a TIMESTAMP(3) metadata rowtime, which Flink derives through the session time zone;
  • tables whose columns are all metadata, and keyed tables whose every physical column is a key field under EXCEPT_KEY.

Direction

Admit each shape only with parity tests against Flink's own deserializers and end-to-end decode-counter evidence on both lines, plus measurements against stock Flink and the current route, per the connector admission policy. A cross-library key could decode in its own library and cross into the value library's composition as Arrow C Data, so no Rust object crosses a library boundary. Update the fallback list in docs/connectors/kafka.md as each shape lands.

Depends on #539 and #540. Tracked in #657.

Dominant language
Java
Stars
157
Forks
13
Avg merge
7h 56m
Merged PRs (30d)
153

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from datafusion-contrib/StreamFusion

All issues in datafusion-contrib/StreamFusion

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.