check-alerts worker: 13 error spans (mongodb.create, NamespaceExists) every minute because mongoose autoCreate is still on
维护者通常 1 天内回复
还没有人认领这个 Issue。
- #3546 来自 @CG-5228 —— 已关闭,未合并
评估
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 72/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- mongodb, typescript
- 领域
- backend, observability
调研方向
先阅读 check-alerts worker 中的 DefaultAlertProvider.init() 和 asyncDispose(),然后检查 connectDB 如何传递 Mongoose 选项;PR #2887 是有关 autoIndex 的相关背景。确认 autoCreate: false 已应用于该 worker 的连接,然后运行 worker 或相关测试,并检查重复运行时不再产生 13 个 mongodb.create 错误跨度。
由索引模型根据 Issue 内容生成。
描述
What happened?
Our self-hosted HyperDX sends its own telemetry to a collector, so we see what the check-alerts worker does. Every minute its trace contains 13 failed spans named mongodb.create with the status message "Collection already exists", one per model: hyperdx.alerts, alerthistories, connections, dashboards, favorites, pinnedfilters, presetdashboardfilters, savedsearches, sources, teams, teaminvites, users and webhooks. On an idle instance that adds up to roughly 19 000 error spans a day, and the Service Dashboard shows a 55 % client error rate for the hyperdx service. Nothing is actually broken, alerts still run.
The cause, as far as I can tell from the code: DefaultAlertProvider.init() opens a new connection on every run (connectDB with autoIndex false) and asyncDispose() closes it again. Mongoose still has autoCreate on, so on every fresh connection Model.init() calls createCollection for each model. The collections exist, MongoDB answers NamespaceExists (code 48), Mongoose ignores that error, but the OpenTelemetry mongodb instrumentation has already recorded a failed command.
PR #2887 turned autoIndex off in that worker for the same reason (the API service owns the schema). Passing autoCreate: false alongside it should remove the 13 commands per run. Happy to open that PR if you want it.
Steps to reproduce
- Run the hyperdx image with HYPERDX_API_KEY and OTEL_EXPORTER_OTLP_ENDPOINT pointing at a collector.
- Wait a minute for the check-alerts cron.
- Search traces for
StatusCode:Error– 13 mongodb.create spans per minute.
How are you running HyperDX?
Self-hosted, image hyperdx/hyperdx:2.40.0 with mongo:5.0.32 as rootless Podman containers; own ClickHouse 26.8 and OpenTelemetry Collector 0.162.0 on separate hosts.
Where does it show up?
In the traces of the hyperdx service itself (Search with source Traces, Service Dashboard). Not a UI bug.
Debug info
N/A (worker process, not the UI)
Logs
SpanName: mongodb.create (Client)
StatusCode: Error
StatusMessage: Collection already exists. NS: hyperdx.teams
ScopeName: @opentelemetry/instrumentation-mongodb 0.52.0
service.version: 2.40.0
- 主要语言
- TypeScript
- 星标
- 10.1k
- 派生
- 527
- 平均合并
- 1 天 20 小时
- 30 天内合并 PR
- 186
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
ClickHouse/ClickStack 的其他 Issue
-
external
难度 2/5 1-3 小时 新手友好度 72/100
ClickHouse/ClickStack#3761 ·
维护者通常 1 天内回复
-
Metric chart with aggFn increase and a multi-column group-by fails: toString() gets several arguments可能已有人在做 @CG-5228 于 26 天前认领。 未关闭external
难度 2/5 1-3 小时 新手友好度 76/100
ClickHouse/ClickStack#3477 · 1 条评论 ·
维护者通常 1 天内回复
-
Slack notifications for inline alerts fail when the chart-explorer link passes Slack's 3000-character block limit可能已有人在做 @Tyagiquamar 于 27 天前认领。 未关闭alerting bug external
难度 2/5 1-3 小时 新手友好度 76/100
ClickHouse/ClickStack#3452 ·
维护者通常 1 天内回复
-
count_distinct aggFn on metric tiles ignores valueExpression, always counts distinct metric Values可能已有人在做 @Harshul1484 于 54 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
ClickHouse/ClickStack#2311 ·
维护者通常 1 天内回复
-
enhancement Needs Feedback
难度 5/5 一周以上 新手友好度 25/100
ClickHouse/ClickStack#3687 ·
维护者通常 1 天内回复
查看 ClickHouse/ClickStack 的全部 Issue
相似的 Issue
-
Add: GTrekker未关闭check:passed streams:add
难度 1/5 1 小时以内 新手友好度 72/100
维护者通常 1 天内回复
-
area: ui Bug Payload OSS
难度 1/5 1 小时以内 新手友好度 88/100
payloadcms/payload#19839 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
fix(auth): add the email OTP verification step to Pollar login可能已有人在做 @akansnoro 今天认领。 未关闭Stellar Wave
难度 2/5 1-3 小时 新手友好度 74/100
safetrustcr/dApp-SafeTrust#584 · 10 条评论 ·
维护者通常 1 天内回复
-
The manager preset saves launch definitions without the required file and digest pair可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭bug
难度 2/5 1-3 小时 新手友好度 80/100
维护者通常 1 天内回复