confluentinc/ksql

Topic <--> SQL Conversion Tool

Open

#8,444 opened on Dec 3, 2021

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Java (1,048 forks)batch import
enhancementgood first issue

Repository metrics

Stars
 (5,739 stars)
PR merge metrics
 (Avg merge 4d 20h) (6 merged PRs in 30d)

Description

Is your feature request related to a problem? Please describe.

Often we run into situations when we try to replicate a user's environment and data, and we end up using mock data that we create which is similar-ish to what the user is seeing. This isn't always sufficient to help us get to the root cause of an issue - it would be great if we could have a way to reliably "load" a topic into ksqlDB from a topic that isn't in the same kafka cluster (and can be sent over public internet, such as a file)

Describe the solution you'd like

It'd be cool if we could take a topic and dump it in standard SQL format (e.g. using INSERT INTO ... VALUES). That way, we could take the file and just run it as a script into ksqlDB and set up the data with exactly the correct timestamps, keys and values (though perhaps not offsets) agnostic to any format (e.g. AVRO/JSON/PROTOBUF).

Describe alternatives you've considered

If such a tool exists, I'd love to hear about it!

Contributor guide