check-alerts worker: 13 error spans (mongodb.create, NamespaceExists) every minute because mongoose autoCreate is still on

Aberta Para iniciantes
#3,308 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

  • #3546 de @CG-5228 — fechado sem integrar

Avaliação

Dificuldade
2/5
Tempo estimado
1-3 horas
Facilidade para iniciantes
72/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
mongodb, typescript

Direção de pesquisa

Comece lendo DefaultAlertProvider.init() e asyncDispose() no worker check-alerts; em seguida, inspecione como connectDB passa as opções do Mongoose. PR #2887 é um contexto relevante para autoIndex. Confirme que autoCreate: false é aplicado à conexão desse worker; depois, execute o worker ou os testes pertinentes e verifique se execuções repetidas não emitem mais os 13 spans de erro de mongodb.create.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug external Needs Feedback

What happened?

Our self-hosted HyperDX sends its own telemetry to a collector, so we see what the check-alerts worker does. Every minute its trace contains 13 failed spans named mongodb.create with the status message "Collection already exists", one per model: hyperdx.alerts, alerthistories, connections, dashboards, favorites, pinnedfilters, presetdashboardfilters, savedsearches, sources, teams, teaminvites, users and webhooks. On an idle instance that adds up to roughly 19 000 error spans a day, and the Service Dashboard shows a 55 % client error rate for the hyperdx service. Nothing is actually broken, alerts still run.

The cause, as far as I can tell from the code: DefaultAlertProvider.init() opens a new connection on every run (connectDB with autoIndex false) and asyncDispose() closes it again. Mongoose still has autoCreate on, so on every fresh connection Model.init() calls createCollection for each model. The collections exist, MongoDB answers NamespaceExists (code 48), Mongoose ignores that error, but the OpenTelemetry mongodb instrumentation has already recorded a failed command.

PR #2887 turned autoIndex off in that worker for the same reason (the API service owns the schema). Passing autoCreate: false alongside it should remove the 13 commands per run. Happy to open that PR if you want it.

Steps to reproduce

  1. Run the hyperdx image with HYPERDX_API_KEY and OTEL_EXPORTER_OTLP_ENDPOINT pointing at a collector.
  2. Wait a minute for the check-alerts cron.
  3. Search traces for StatusCode:Error – 13 mongodb.create spans per minute.

How are you running HyperDX?

Self-hosted, image hyperdx/hyperdx:2.40.0 with mongo:5.0.32 as rootless Podman containers; own ClickHouse 26.8 and OpenTelemetry Collector 0.162.0 on separate hosts.

Where does it show up?

In the traces of the hyperdx service itself (Search with source Traces, Service Dashboard). Not a UI bug.

Debug info

N/A (worker process, not the UI)

Logs

SpanName: mongodb.create (Client)
StatusCode: Error
StatusMessage: Collection already exists. NS: hyperdx.teams
ScopeName: @opentelemetry/instrumentation-mongodb 0.52.0
service.version: 2.40.0
Linguagem predominante
TypeScript
Estrelas
10.1k
Forks
527
Merge médio
1d 20h
PRs com merge (30d)
186

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de ClickHouse/ClickStack

Todas as issues de ClickHouse/ClickStack

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.