CLI-3553: Avro JSON Encoding Union Type Fails to Decode

Open
#2,822 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
go, kafka
Domain
cli

Research direction

Reproduce the failure with the provided Avro schema and sample record, comparing the CLI with /usr/bin/kafka-avro-console-producer in the Confluent Schema Registry container. Trace the CLI's JSON-encoded Avro decoding path and verify that the timestamp union value follows the Avro JSON encoding specification without producing the reported error.

Written by the indexing model from the issue text.

Description

enhancement

When attempting to write records to Confluent Cloud using the CLI, we sometimes get errors related JSON Encoded Avro records that deal with union types.

According to the JSON Encoding spec from Avro, when using a union type

  • if its type is null, then it is encoded as a JSON null;
  • otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value. For Avro’s named types (record, fixed or enum) the user-specified name is used, for other types the type name is used.

We can do this locally using the included /usr/bin/kafka-avro-console-producer inside of the CP Schema Registry docker container, but not using the CLI. Instead, we get

Error: cannot decode textual record "<record-namespace>.<record-name>": cannot decode textual union: cannot decode textual map: cannot determine codec: "<type>" for key: "<field-name>"

Below is the full schema (with some sensitive bits removed) and the sample record we are using

{
  "type" : "record",
  "name" : "Updated",
  "namespace" : "namespace",
  "doc" : "Updated Event",
  "fields" : [ {
    "name" : "subjectivity_id",
    "type" : {
      "type" : "string",
      "logicalType" : "string"
    },
    "doc" : "Identifier"
  }, {
    "name" : "topic",
    "type" : "string",
    "doc" : "Topic name"
  }, {
    "name" : "created_at",
    "type" : [ "null", {
      "type" : "long",
      "logicalType" : "timestamp-millis"
    } ],
    "doc" : "Date indicating when the record was created",
    "default" : null
  }, {
    "name" : "updated_at",
    "type" : [ "null", {
      "type" : "long",
      "logicalType" : "timestamp-millis"
    } ],
    "doc" : "Date indicating when the record was updated",
    "default" : null
  }]
}
{ "subjectivity_id": "subj-1234","topic": "subjectivity_updated","created_at": null,"updated_at": { "long": 1696613296 }}

Let me know what other data we can provide!

Dominant language
Go
Stars
82
Forks
37
Avg merge
1d 13h
Merged PRs (30d)
22

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from confluentinc/cli

All issues in confluentinc/cli

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.