bug: fluent operator is consuming lot of cpu cores (~27 of the 32 available cores on a Node)

未关闭
  • 哪里不对？
#1,990 9 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
48/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
冷清
技术栈
go, helm, kubernetes
领域
devops, infrastructure, observability-sre

调研方向

首先复现 issue 中的 v3.4.0 部署，并检查 controller-runtime 跟踪信息和 CPU 行为。阅读 manifests/setup/fluent-operator-deployment.yaml，并将症状与旧 issue #1717 进行比较；当确定并修复原因，使重新创建的 operator pod 不再消耗几乎所有可用 CPU 核心时，即表示完成。

由索引模型根据 Issue 内容生成。

描述

Describe the issue

When deploying Fluent Operator on a Kubernetes cluster, the operator process consumes a very high number of CPU cores (~27 out of 32 available cores on a single Node). This is far beyond expected usage and causes resource contention.

Deleting the fluent pod does not change the high cpu behavior once new pod is recreated. Need assistance identifying cause of the high cpu load, thanks.

fluent-operator-df4555f76-ht6f2.log

2025-09-15T04:49:19Z INFO setup starting manager
2025-09-15T04:49:19Z INFO controller-runtime.metrics Starting metrics server
2025-09-15T04:49:19Z INFO starting server {"name": "health probe", "addr": "[::]:8081"}
2025-09-15T04:49:19Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
2025-09-15T04:49:19Z INFO Starting EventSource {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.FluentBit"}

Trace[833216769]: [208.550494ms] [208.550494ms] END
I0915 04:49:45.438100 1 trace.go:236] Trace[442998739]: "DeltaFIFO Pop Process" ID:XXXXX,Depth:16,Reason:slow event handlers blocking the queue (15-Sep-2025 04:49:45.204) (total time: 186ms):
Trace[442998739]: [186.678635ms] [186.678635ms] END
I0915 04:49:45.830454 1 trace.go:236] Trace[989326455]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:232 (15-Sep-2025 04:49:19.799) (total time: 25965ms):
Trace[989326455]: ---"Objects listed" error: 25832ms (04:49:45.631)
Trace[989326455]: [25.965573843s] [25.965573843s] END
2025-09-15T04:57:02Z INFO controllers.FluentBitConfig Fluent Bit main configuration has updated {"logging-control-plane": "monitoring", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}

Deployed fluent-operator 3.4.0 in "operators" namespace using fluent-operator helm charts. Below is the configurations:

Fluent Operator configurations
fluent-operator:
  enabled: true
  fluent-operator:
    containerRuntime: containerd
    operator:
       # The init container is to get the actual storage path of the docker log files so that it can be mounted to collect the logs.
       # see https://github.com/fluent/fluent-operator/blob/master/manifests/setup/fluent-operator-deployment.yaml#L26
      logPath:
        # The operator currently assumes a Docker container runtime path for the logs as the default, for other container runtimes you can set the location explicitly below.
        containerd: /var/log/containers/*.log
      disableComponentControllers: "fluentd"
      

    fluentbit:
       # Installs a sub chart carrying the CRDs for the fluent-bit controller. The sub chart is enabled by default.
      crdsEnable: true
      enable: false

    fluentd:
       # Installs a sub chart carrying the CRDs for the fluentd controller. The sub chart is enabled by default.
      crdsEnable: false

Deployed fluent-bit in "monitoring" namespace using fluent-operator helm charts. Below is the configurations:

Fluent bit configurations
fluent-operator:
  containerRuntime: containerd
  operator:
    enable: false
    disableComponentControllers: "fluentd"
      

    fluentbit:
       # Installs a sub chart carrying the CRDs for the fluent-bit controller. The sub chart is enabled by default.
      enable: true

Please let me if you want me provide any other details.
Old bug ref : https://github.com/fluent/fluent-operator/issues/1717

To Reproduce

Deploy Fluent Operator version v3.4.0 on a Kubernetes cluster.

Observe CPU usage on the Node where the operator is running.

Notice that the operator consumes ~27 cores out of 32.

Expected behavior

Fluent Operator should run normally with minimal CPU usage, not consuming nearly all available cores.

Your Environment
Fluent Operator version: 3.4.0

Container Runtime: containerd

Operating System: Ubuntu 22.04



Kubernetes version: 1.35
How did you install fluent operator?

No response

Additional context

No response

主要语言
Go
星标
689
派生
333
平均合并
2 天 6 小时
30 天内合并 PR
43

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

fluent/fluent-operator 的其他 Issue

查看 fluent/fluent-operator 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。