Feature request: Add Avro serialization for Kafka keys and values

Use Case

Our current telemetry pipeline is:

Telegraf -> Apache NiFi -> Kafka

Telegraf collects and enriches the metrics. Apache NiFi then serializes the Kafka message key and value using Avro and publishes the messages to Kafka.

The source systems and Kafka are located in different network zones. Using Avro is important because it reduces the amount of data transferred between these zones.

We want to remove Apache NiFi and use Telegraf as the only component in the pipeline:

Telegraf -> Kafka

This would allow Telegraf to collect, enrich, serialize, and publish the metrics directly to Kafka.

Expected behavior

The outputs.kafka plugin should support Avro serialization for both:

• the Kafka message key;
• the Kafka message value.

The serialized key and value should use binary Avro encoding and the Confluent wire format.

The key and value should preferably support separate Avro schemas and schema IDs, since they may use different schemas.

The Avro schema is defined once and reused for serialization.

A possible configuration could look like this:

[[outputs.kafka]]
  brokers = ["kafka:9092"]
  topic = "metrics"

  key_data_format = "avro"
  data_format = "avro"

  schema_registry_url = "https://schema-registry.example.com"

  key_schema_id = 10
  value_schema_id = 11

  # Optional TLS configuration
  # tls_ca = "/etc/telegraf/certs/ca.pem"
  # insecure_skip_verify = false

Telegraf should:

1. Read the configured schema ID for the Kafka key and value.
2. Check whether the corresponding schema is available in the local in-memory cache.
3. Fetch the schema from Schema Registry over HTTPS on a cache miss.
4. Store the retrieved schema in the cache.
5. Reuse the cached schema for subsequent messages.
6. Serialize the Kafka key and value using the corresponding schemas.
7. Publish the resulting Confluent-compatible payload to Kafka.

The exact configuration names are illustrative and can be adjusted to match the existing Telegraf conventions.

Actual behavior

Telegraf currently supports Avro as an input data format, but it does not support Avro serialization for Kafka output messages.

In particular, outputs.kafka cannot currently:

• serialize the Kafka message key as Avro;
• serialize the Kafka message value as Avro;
• produce Kafka key and value payloads in the Confluent wire format.

As a result, the current pipeline cannot be migrated completely to:

Telegraf -> Kafka

Using JSON or another text-based format instead of Avro would increase the amount of data transferred between the network zones.

Additional info

The main goal is to remove Apache NiFi and reduce the operational and network overhead of the current pipeline.

A minimal first implementation could support:

• the Kafka output plugin only;
• binary Avro encoding;
• user-provided schemas;
• separately configured key and value schema IDs;
• fetching existing schemas from Schema Registry over HTTPS;
• in-memory schema caching;
• Confluent wire format.

Schema registration, dynamic schema generation, and advanced schema evolution support could be added later.

This request is related to issue #1630:

https://github.com/influxdata/telegraf/issues/1630

That issue requested Avro support for Kafka producer and consumer. The current implementation provides Avro parsing, but Avro output serialization is still required for this use case.