check-alerts worker: 13 error spans (mongodb.create, NamespaceExists) every minute because mongoose autoCreate is still on

Abierto Apto para principiantes
  • ¿Algo no está bien?
#3,308 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
72/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
mongodb, typescript
Área
backend, observability

Línea de trabajo

Empieza leyendo DefaultAlertProvider.init() y asyncDispose() en el worker check-alerts; después, inspecciona cómo connectDB pasa las opciones de Mongoose. PR #2887 es un contexto relevante para autoIndex. Confirma que autoCreate: false se aplica a la conexión de ese worker; luego ejecuta el worker o las pruebas pertinentes y comprueba que las ejecuciones repetidas ya no emitan los 13 spans de error de mongodb.create.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug external Needs Feedback

What happened?

Our self-hosted HyperDX sends its own telemetry to a collector, so we see what the check-alerts worker does. Every minute its trace contains 13 failed spans named mongodb.create with the status message "Collection already exists", one per model: hyperdx.alerts, alerthistories, connections, dashboards, favorites, pinnedfilters, presetdashboardfilters, savedsearches, sources, teams, teaminvites, users and webhooks. On an idle instance that adds up to roughly 19 000 error spans a day, and the Service Dashboard shows a 55 % client error rate for the hyperdx service. Nothing is actually broken, alerts still run.

The cause, as far as I can tell from the code: DefaultAlertProvider.init() opens a new connection on every run (connectDB with autoIndex false) and asyncDispose() closes it again. Mongoose still has autoCreate on, so on every fresh connection Model.init() calls createCollection for each model. The collections exist, MongoDB answers NamespaceExists (code 48), Mongoose ignores that error, but the OpenTelemetry mongodb instrumentation has already recorded a failed command.

PR #2887 turned autoIndex off in that worker for the same reason (the API service owns the schema). Passing autoCreate: false alongside it should remove the 13 commands per run. Happy to open that PR if you want it.

Steps to reproduce

  1. Run the hyperdx image with HYPERDX_API_KEY and OTEL_EXPORTER_OTLP_ENDPOINT pointing at a collector.
  2. Wait a minute for the check-alerts cron.
  3. Search traces for StatusCode:Error – 13 mongodb.create spans per minute.

How are you running HyperDX?

Self-hosted, image hyperdx/hyperdx:2.40.0 with mongo:5.0.32 as rootless Podman containers; own ClickHouse 26.8 and OpenTelemetry Collector 0.162.0 on separate hosts.

Where does it show up?

In the traces of the hyperdx service itself (Search with source Traces, Service Dashboard). Not a UI bug.

Debug info

N/A (worker process, not the UI)

Logs

SpanName: mongodb.create (Client)
StatusCode: Error
StatusMessage: Collection already exists. NS: hyperdx.teams
ScopeName: @opentelemetry/instrumentation-mongodb 0.52.0
service.version: 2.40.0
Lenguaje dominante
TypeScript
Estrellas
10k
Forks
508
Merge medio
2 d 9 h
PR fusionados (30 d)
156

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de hyperdxio/hyperdx

Todos los issues de hyperdxio/hyperdx

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.