Producer.poll(timeout) blocks for full timeout after delivery callback fires (regression in 2.13.0)

Abierto Apto para principiantes
#2,215 1 comentario 3 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

  • #2297 de @wgu9 — cerrado sin fusionar

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
78/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
c, python
Área
api, backend

Línea de trabajo

Comienza en src/confluent_kafka/src/Producer.c, en Producer_poll0, y compara su bucle que puede despertarse con la implementación de Consumer.poll(). Ejecuta la reproducción proporcionada de producer.poll(5.0) con las versiones indicadas y verifica que poll retorne poco después de que se ejecute el callback de entrega, en lugar de consumir todo el tiempo de espera.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

component:producer

Description

Producer.poll(timeout) no longer returns early after processing delivery callbacks when timeout > 200ms. Instead it blocks for the entire timeout duration, regardless of whether events were served.

This is a regression introduced in 2.13.0 by the "wakeable poll" refactor (PR #2126 ). The original single-call implementation of rd_kafka_poll() returned as soon as events were processed. The new chunked loop replicates this correctly for Consumer.poll() but not for Producer.poll().

The producer loop accumulates chunk_result but never breaks when chunk_result > 0, causing it to exhaust all remaining chunks even after the delivery callback has fired.

Step to reproduce

import time
from confluent_kafka import Producer, __version__

KAFKA_BOOTSTRAP = "localhost:9092"
POLL_TIMEOUT = 5.0

print(f"confluent-kafka version: {__version__}")

producer = Producer({"bootstrap.servers": KAFKA_BOOTSTRAP})

def on_delivery(err, msg):
    print(f"  [{time.time():.3f}] on_delivery fired — err={err}, topic={msg.topic()}")

print(f"[{time.time():.3f}] produce()")
producer.produce("test-topic", value=b"hello", on_delivery=on_delivery)

print(f"[{time.time():.3f}] poll({POLL_TIMEOUT}) start")
t0 = time.time()
producer.poll(POLL_TIMEOUT)
elapsed = time.time() - t0
print(f"[{time.time():.3f}] poll() returned after {elapsed:.3f}s")

Output in 2.12.2

confluent-kafka version: 2.12.2
[1733305268.206] produce()
[1733305268.206] poll(5.0) start
  [1733305268.532] on_delivery fired — err=None, topic=test-topic
[1733305268.532] poll() returned after 0.325s

Output in 2.13.2

confluent-kafka version: 2.13.2
[1733305281.264] produce()
[1733305281.265] poll(5.0) start
  [1733305281.276] on_delivery fired — err=None, topic=test-topic
[1733305286.078] poll() returned after 4.813s

Potential Cause

In Producer_poll0 (src/confluent_kafka/src/Producer.c), the wakeable loop does not break when chunk_result > 0

Potential fix

Add an early exit in Producer_poll0 after events are served, matching the original rd_kafka_poll() single-call behaviour and the consumer implementation:

int chunk_result = rd_kafka_poll(self->rk, chunk_timeout_ms);
if (chunk_result < 0) { r = chunk_result; break; }

r += chunk_result;
if (chunk_result > 0) break;  // ← return early once events are served

The new chunked implementation was intended to preserve this early-exit behaviour while adding signal interruptibility. It does so correctly for Consumer.poll() (if (rkm) break;) but the equivalent guard was omitted in Producer_poll0, making poll(5.0) in 2.13.x strictly worse than poll(5.0) in 2.12.x when the broker is fast.

Environment

  • confluent-kafka-python: 2.13.0, 2.13.1, 2.13.2
  • Python: 3.11+
  • OS: Linux

Thanks to the maintainers for the great work on confluent kafka. Happy to open a PR with the proposed fix if the issue is
confirmed.

Lenguaje dominante
Python
Estrellas
528
Forks
977
Merge medio
2 d 19 h
PR fusionados (30 d)
19

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de confluentinc/confluent-kafka-python

Todos los issues de confluentinc/confluent-kafka-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.