feat: support storage.total_limit_size for the fluent-bit forward output
维护者通常 1 天内回复
评估
调研方向
The issue names plugins/output/http_types.go as the reference for the field and rendering, and a linked pull request (#2127) is already open. Read that file and the forward output type and renderer to understand the matching pattern; done means the forward output exposes totalLimitSize and renders it as storage.total_limit_size, with coverage following the existing output tests.
由索引模型根据 Issue 内容生成。
描述
Is your feature request related to a problem? Please describe.
With storage.type filesystem on inputs, fluent-bit only bounds the on-disk buffer per output via
storage.total_limit_size. The forward output (ClusterOutput.spec.forward) does not expose this
setting, so there is no way to cap the filesystem buffer for the common fluent-bit → fluentd setup.
We hit this in production: the downstream (fluentd → OpenSearch) was unavailable, fluent-bit kept
buffering with retry_limit: no_limits, the buffer grew to ~40 GB on the host path and filled
the node disks.
Today TotalLimitSize exists on 8 output plugins (es, opensearch, kafka, loki, syslog, http,
opentelemetry, splunk — added one by one, e.g. in #1196, #1318, #1823, #1865, #2009), but not on
forward and 16 others (checked on master 404b514).
Current workaround: replace forward with customPlugin.config and write the whole output section
by hand. This loses the typed CRD fields (tls, networking, secrets) and has to be duplicated in
every overlay that patches the output.
Describe the solution you'd like
Add totalLimitSize to the forward output, rendered as storage.total_limit_size, same as
plugins/output/http_types.go:
// Limit the maximum number of Chunks in the filesystem for the current output logical destination.
TotalLimitSize string `json:"totalLimitSize,omitempty"`
...
plugins.InsertKVString(kvs, "storage.total_limit_size", f.TotalLimitSize)
Alternative: since storage.total_limit_size is a generic option of every fluent-bit output (like
Retry_Limit, which is already on OutputSpec), it could be added once to OutputSpec instead
of per plugin. Happy to go either way.
Additional context
- fluent-operator v3.6.0, fluent-bit 4.2.2, downstream OpenSearch 3.9.0
- fluent-bit docs: https://docs.fluentbit.io/manual/pipeline/buffering#per-output-settings
- 主要语言
- Go
- 星标
- 689
- 派生
- 333
- 平均合并
- 2 天 6 小时
- 30 天内合并 PR
- 43
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
fluent/fluent-operator 的其他 Issue
-
feat: support the Fluent Bit type_converter filter可能已有人在做 @NopeCharon 今天认领。 未关闭
难度 4/5 3-5 天 新手友好度 65/100
fluent/fluent-operator#2131 ·
维护者通常 1 天内回复
-
bug: fluent-bit image still ships vulnerable libbrotli1 (CVE-2026-111497)可能已有人在做 @joshuabaird 于 9 天前认领。 未关闭fluent-bit security
难度 2/5 1-3 小时 新手友好度 55/100
fluent/fluent-operator#2082 · 2 条评论 ·
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 48/100
fluent/fluent-operator#1990 · 9 条评论 ·
维护者通常 1 天内回复
-
enhancement fluent-bit good first issue
难度 4/5 3-5 天 新手友好度 35/100
fluent/fluent-operator#1854 · 2 条评论 · 1 个 reaction ·
维护者通常 1 天内回复
-
Remove tag consideration from Fluentd ClusterFilter / Filter order可能已有人在做 @cvas22 于 1 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 45/100
fluent/fluent-operator#1832 ·
维护者通常 1 天内回复
查看 fluent/fluent-operator 的全部 Issue
相似的 Issue
-
automated-triage type/bug type/ci
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 75/100
hashicorp/terraform-provider-google#30678 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 83/100
ethereum-optimism/optimism#24120 ·
维护者通常 1 天内回复
-
area/docs
难度 1/5 1 小时以内 新手友好度 86/100
meshery/meshery#23375 · 1 条评论 ·
维护者通常 1 天内回复
-
ApplicationSet: BitBucket Cloud SCM provider returns corrupted error (%!w(<nil>)) when a branch has no commit hash可能已有人在做 @AP24110011695 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复