Kafka gateway: keep repeated header names and header order

Abierto
  • ¿Algo no está bien?
#4,517 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
30/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
backend

Línea de trabajo

Start with gateways/kafka/docs/BRIDGE_MAPPING.md to see how headers are currently mapped and sorted on Fetch. The issue says kafka_protocol keeps only the last value of a repeated header name, so the raw record bytes need to be read for headers. The open question on whether every record or only repeated-name records keep order must be settled before changing the envelope. Done means the KafkaJS array-header producer succeeds and a record with a repeated header name matches after Produce and Fetch.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

gateways

Summary

A record with a repeated header name gets INVALID_RECORD (87), and Fetch returns headers sorted by name (gateways/kafka/docs/BRIDGE_MAPPING.md). Kafka allows both.

Who hits it:

  • KafkaJS writes array header values as repeated keys, a documented feature. If it forwards a consumed message, it writes them again. The KafkaJS layer of confluent-kafka-javascript does the same.
  • Apps that append a trace header twice (https://github.com/spring-projects/spring-kafka/issues/3786).
  • Error 87 fails the whole partition batch in librdkafka, franz-go and Sarama. It puts a kafka-python 3.0 producer in a fatal state, so every later send fails until a restart. A kafka-go writer at its default acks=0 loses the batch and the connection, with no error.

Open question

Does every record keep its header order, at the cost of a larger stored envelope, or only records with repeated names?

Scope

  • Read headers from the raw record bytes, because kafka_protocol keeps only the last value of a name
  • Store a record with repeated names in the envelope, which keeps a list
  • Return headers in their original order on Fetch, as the open question decides
  • Update gateways/kafka/docs/BRIDGE_MAPPING.md

Acceptance criteria

  • A KafkaJS producer with array header values succeeds, and a consumer reads every value in order
  • A record with the same header name twice is the same after Produce and Fetch

References

Lenguaje dominante
Rust
Estrellas
5.2k
Forks
502
Merge medio
3 d 3 h
PR fusionados (30 d)
163

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/iggy

Todos los issues de apache/iggy

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.