Workspace sync re-broadcasts hundreds of duplicate "mobile notification limit reached" cloud announcements every 12h, flooding GET /api/v1/banners and restarting the server

Abierto
#43,564 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@Kulraj69 ya está trabajando en esto.

Desde el 6/11/2026.

  • #43568 de @Kulraj69 — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
47/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
kubernetes, mongodb, node.js, typescript

Línea de trabajo

Start with apps/meteor/server/lib/cloud/syncWorkspace/handleCommsSync.ts and trace how announcements in create reach Banner.create(), Banners.createOrUpdate(), and sendToUsers(). Check the related banner and workspace sync tests, then reproduce a sync with repeated announcements if the test setup allows it. Done means unchanged announcements no longer trigger a burst of broadcasts; the issue also raises preventing duplicate announcements and cleaning up expired banners as possible scope.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

type: bug
Description:

When the workspace's monthly mobile push quota is exceeded (from around the 22nd of each month until the quota resets), Rocket.Chat Cloud sends a new cloud-announcements-core announcement many times a day (200–400 per day). They all say the same thing:

Workspace members are not receiving notifications on mobile — Your plan's monthly mobile notification limit has been reached.

Each announcement lasts about 48h, so several hundred identical banners are active at the same time.

On every workspace sync (every 12h, at 00:xx and 12:xx UTC), handleAnnouncementsOnWorkspaceSync (apps/meteor/server/lib/cloud/syncWorkspace/handleCommsSync.ts) calls Banner.create() for each announcement in create. That runs Banners.createOrUpdate() and then sendToUsers(), which broadcasts banner.enabled once per announcement, even when the banner has not changed.

Connected admin clients then send thousands of GET /api/v1/banners?platform=web requests within a minute. Each response contains all the active duplicates (around 440 KB with about 490 active banners). The Node.js event loop saturates, /health stops responding, and Kubernetes' liveness probe restarts the server.

Steps to reproduce:
  1. Use a registered workspace on a plan with a mobile push quota, and exceed the quota.
  2. Wait a few days so that Cloud creates many "notification limit reached" announcements.
  3. Keep several admin users connected on the web client.
  4. Watch GET /api/v1/banners requests and nodejs_eventloop_lag_max_seconds during the next workspace sync.
Expected behavior:
  • An announcement that has not changed should not be broadcast again on every sync, or at least the broadcasts should be grouped (one event per sync rather than one per banner).
  • Cloud should not create hundreds of identical announcements per month for the same condition. One active announcement per condition should be enough.
  • Optionally, expired cloud announcements could be removed or disabled during sync.
Actual behavior:

Thousands of /api/v1/banners requests per minute at each sync, event loop saturation, /health timeouts, and the pod restarted by liveness. All users are disconnected at the end of every month.

The 00:xx bursts are much smaller than the 12:xx ones, which fits fewer admin clients being connected at night. The volume seems to grow with (number of active duplicate announcements) × (number of connected admin clients).

Server Setup Information:
  • Version of Rocket.Chat Server: 8.9.0 when observed (also seen on 8.2.0 and 8.5.x); now on 8.9.1, where the code paths are unchanged
  • License Type: Community
  • Number of Users: 180
  • Operating System: Linux (official Docker image)
  • Deployment Method: Kubernetes (RKE2), official Helm chart 6.27.1
  • Number of Running Instances: 1 (microservices disabled)
  • DB Replicaset Oplog: Yes (replica set)
  • NodeJS Version: 24.22.1 (8.9.1 image)
  • MongoDB Version: 8.0.13
Client Setup Information
  • Desktop App or Browser Version: web client (current browsers), admins only (the announcements target roles: ["admin"])
  • Operating System: various
Additional context

Prometheus data (rocketchat_rest_api_count{entrypoint="banners"}):

Sync (UTC) /banners calls/min Notes
baseline ~0.3
2026-09-30 12:41 1,814 27 active REST requests, event loop lag 0.85 s, pod restarted by liveness
2026-10-29 12:32 1,670 50 active REST requests, event loop lag 0.68 s, pod restarted
2026-10-29 12:48 959 new container, restarted again
2026-10-28 00:xx / 12:xx 291 / 1,352
2026-10-30 00:xx / 12:xx 1,990 / 3,454
2026-11-01 onwards 0–8 quota reset, no more announcements

Heap stays flat (~600 MiB) and MongoDB has no slow queries during the bursts. Only the event loop is affected.

Database observations (rocketchat_banner, 2026-09-30):

  • 6,035 documents with createdBy: "cloud", view.appId: "cloud-announcements-core", roles: ["admin"], platform: ["web","mobile"]
  • About 578 not yet expired. The rest have expired but are still active: true and are never removed. They seem harmless because findActiveByRoleOrId filters on expireAt and createOrUpdate refuses expired banners, but they pile up month after month.
  • Same pattern each month since June 2026: from the ~22nd until the end of the month, nothing in between.
  • Deleting the documents only helps until the next sync, because announcements that have not expired are upserted and broadcast again.

Workaround in place: we relaxed the liveness and readiness probes so that a burst of one or two minutes no longer restarts the pod. Admins still get slow responses during each sync.

Related: #32018 (banners causing client-side performance problems), #31501, #19877 (push-limit alerts repeated).

Relevant logs:

The server logs no error or warning during the bursts: the workspace sync succeeds. The problem only shows up in the metrics above (REST request rate, active requests, event loop lag) and in the Kubernetes liveness probe failures.

Lenguaje dominante
TypeScript
Estrellas
46.5k
Forks
14.1k
Merge medio
1 d 21 h
PR fusionados (30 d)
381

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de RocketChat/Rocket.Chat

Todos los issues de RocketChat/Rocket.Chat

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.