AdminClient and DNS lookup issue

Abierto
  • ¿Algo no está bien?
#3,138 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

  • #3300 de @yashdotdev13 — cerrado sin fusionar

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Estancado
Stack tecnológico
java, kafka, kubernetes, spring
Área
backend, distributed-systems, networking, observability

Línea de trabajo

Comienza con el issue relacionado 2914 y la configuración del binder de Kafka Streams; después, inspecciona cómo la creación de AdminClient y /actuator/health/binders/kstream determinan su estado. Compara la accesibilidad del broker desde el endpoint con el comportamiento descrito ante un fallo de DNS. Se considera completado cuando se documenta la causa raíz y se valida un enfoque de recuperación o liveness para la interrupción descrita.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

@sobychacko

We are using Spring cloud stream and using Kafka Streams binder configuration. This is what our configuration looks like:

 spring:
  cloud:
   stream:
    bindings:
     <input>:
      <destination>
    kafka:
     bindings:
      <input>:
       consumer:
        dlq-name: <name>
        configuration: 
         offset-reset-strategy: latest
         group.id: <id>
     streams:
      binder:
       auto-create-topics: false
      configuration:
       session.timeout.ms: 12000
       max.poll.interval.ms: 5000
       heartbeat.interval.ms: 40000
       reconnect.backoff.ms: 500
       reconnect.backoff.max.ms: 2000
       retry.backoff.ms: 1000
       retry.backoff.max.ms: 2000

  1. We are still seeing the AdminClient trying to get instantiated and hence seeing authentication related issues. We are using SASL/SSL authentication methodology. What is the root cause of this happening internally? Why is an adminClient getting instantiated?

  2. Also, we are seeing DNS lookup issues from our K8s clusters when trying to resolve kafka broker hostnames to its IP address when our consumer application is running. This connection gets broken when theres a DNS maintenance or change, and the consumer goes into idle/limbo state and stops consuming any messages from the kafka topic, even though the application seems like its running without any error logs. We are finding ourselves having to restart the AKS pods manually and this is causing significant delays and slowing productivity.

We have tried an auto-healing mechanism but couldn't test this out in lower environments since we were unable to replicate the unique scenario of DNS being unable to resolve broker host IPs. So we're kind of waiting for the issue to show up in prod to test out if auto-heal mechanism actually works. The way we implemented it is by making a liveness health check to the /actuator/health/binders/kstream endpoint, and checking the binder status. We restart the pod if it remains down status for 6 mins. We are currently not sure if the K8s liveness check actually checks the connection to the broker or it only hits the endpoint and gets its status, in which case I believe the pod wouldn't get restarted.

Do you have any thoughts on these concerns? I would like to hear your thoughts on these and share any troubleshooting or any other implementations we could consider for our application to build resilience when theres a network outage.

Please let me know if there are any questions or clarifications you need.

Thanks,
Santhosh

Lenguaje dominante
Java
Estrellas
1.1k
Forks
653
Merge medio
2 d 15 h
PR fusionados (30 d)
8

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de spring-cloud/spring-cloud-stream

Todos los issues de spring-cloud/spring-cloud-stream

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.