AdminClient and DNS lookup issue

未关闭
  • 哪里不对？
#3,138 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

  • #3300 来自 @yashdotdev13 —— 已关闭，未合并

评估

难度
5/5
预计耗时
一周以上
新手友好度
25/100
Issue 类型
缺陷
描述清晰度
需要澄清
活跃度
停滞
技术栈
java, kafka, kubernetes, spring
领域
backend, distributed-systems, networking, observability

调研方向

从相关 issue 2914 和 Kafka Streams binder 配置开始，然后检查 AdminClient 创建和 /actuator/health/binders/kstream 如何推导其状态。将 endpoint 对 broker 的可达性与所描述的 DNS 失败行为进行比较。完成的标准是记录根因，并验证针对所述中断的 recovery 或 liveness 方案。

由索引模型根据 Issue 内容生成。

描述

@sobychacko

We are using Spring cloud stream and using Kafka Streams binder configuration. This is what our configuration looks like:

 spring:
  cloud:
   stream:
    bindings:
     <input>:
      <destination>
    kafka:
     bindings:
      <input>:
       consumer:
        dlq-name: <name>
        configuration: 
         offset-reset-strategy: latest
         group.id: <id>
     streams:
      binder:
       auto-create-topics: false
      configuration:
       session.timeout.ms: 12000
       max.poll.interval.ms: 5000
       heartbeat.interval.ms: 40000
       reconnect.backoff.ms: 500
       reconnect.backoff.max.ms: 2000
       retry.backoff.ms: 1000
       retry.backoff.max.ms: 2000

  1. We are still seeing the AdminClient trying to get instantiated and hence seeing authentication related issues. We are using SASL/SSL authentication methodology. What is the root cause of this happening internally? Why is an adminClient getting instantiated?

  2. Also, we are seeing DNS lookup issues from our K8s clusters when trying to resolve kafka broker hostnames to its IP address when our consumer application is running. This connection gets broken when theres a DNS maintenance or change, and the consumer goes into idle/limbo state and stops consuming any messages from the kafka topic, even though the application seems like its running without any error logs. We are finding ourselves having to restart the AKS pods manually and this is causing significant delays and slowing productivity.

We have tried an auto-healing mechanism but couldn't test this out in lower environments since we were unable to replicate the unique scenario of DNS being unable to resolve broker host IPs. So we're kind of waiting for the issue to show up in prod to test out if auto-heal mechanism actually works. The way we implemented it is by making a liveness health check to the /actuator/health/binders/kstream endpoint, and checking the binder status. We restart the pod if it remains down status for 6 mins. We are currently not sure if the K8s liveness check actually checks the connection to the broker or it only hits the endpoint and gets its status, in which case I believe the pod wouldn't get restarted.

Do you have any thoughts on these concerns? I would like to hear your thoughts on these and share any troubleshooting or any other implementations we could consider for our application to build resilience when theres a network outage.

Please let me know if there are any questions or clarifications you need.

Thanks,
Santhosh

主要语言
Java
星标
1.1k
派生
653
平均合并
2 天 15 小时
30 天内合并 PR
8

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

spring-cloud/spring-cloud-stream 的其他 Issue

查看 spring-cloud/spring-cloud-stream 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。