Workspace sync re-broadcasts hundreds of duplicate "mobile notification limit reached" cloud announcements every 12h, flooding GET /api/v1/banners and restarting the server

Aperta
  • Qualcosa non va?
#43,564 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Kulraj69 ci sta già lavorando.

Dal 6/11/2026.

  • #43568 di @Kulraj69 — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
47/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
kubernetes, mongodb, node.js, typescript
Ambito
backend, observability, performance

Direzione di ricerca

Start with apps/meteor/server/lib/cloud/syncWorkspace/handleCommsSync.ts and trace how announcements in create reach Banner.create(), Banners.createOrUpdate(), and sendToUsers(). Check the related banner and workspace sync tests, then reproduce a sync with repeated announcements if the test setup allows it. Done means unchanged announcements no longer trigger a burst of broadcasts; the issue also raises preventing duplicate announcements and cleaning up expired banners as possible scope.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

type: bug
Description:

When the workspace's monthly mobile push quota is exceeded (from around the 22nd of each month until the quota resets), Rocket.Chat Cloud sends a new cloud-announcements-core announcement many times a day (200–400 per day). They all say the same thing:

Workspace members are not receiving notifications on mobile — Your plan's monthly mobile notification limit has been reached.

Each announcement lasts about 48h, so several hundred identical banners are active at the same time.

On every workspace sync (every 12h, at 00:xx and 12:xx UTC), handleAnnouncementsOnWorkspaceSync (apps/meteor/server/lib/cloud/syncWorkspace/handleCommsSync.ts) calls Banner.create() for each announcement in create. That runs Banners.createOrUpdate() and then sendToUsers(), which broadcasts banner.enabled once per announcement, even when the banner has not changed.

Connected admin clients then send thousands of GET /api/v1/banners?platform=web requests within a minute. Each response contains all the active duplicates (around 440 KB with about 490 active banners). The Node.js event loop saturates, /health stops responding, and Kubernetes' liveness probe restarts the server.

Steps to reproduce:
  1. Use a registered workspace on a plan with a mobile push quota, and exceed the quota.
  2. Wait a few days so that Cloud creates many "notification limit reached" announcements.
  3. Keep several admin users connected on the web client.
  4. Watch GET /api/v1/banners requests and nodejs_eventloop_lag_max_seconds during the next workspace sync.
Expected behavior:
  • An announcement that has not changed should not be broadcast again on every sync, or at least the broadcasts should be grouped (one event per sync rather than one per banner).
  • Cloud should not create hundreds of identical announcements per month for the same condition. One active announcement per condition should be enough.
  • Optionally, expired cloud announcements could be removed or disabled during sync.
Actual behavior:

Thousands of /api/v1/banners requests per minute at each sync, event loop saturation, /health timeouts, and the pod restarted by liveness. All users are disconnected at the end of every month.

The 00:xx bursts are much smaller than the 12:xx ones, which fits fewer admin clients being connected at night. The volume seems to grow with (number of active duplicate announcements) × (number of connected admin clients).

Server Setup Information:
  • Version of Rocket.Chat Server: 8.9.0 when observed (also seen on 8.2.0 and 8.5.x); now on 8.9.1, where the code paths are unchanged
  • License Type: Community
  • Number of Users: 180
  • Operating System: Linux (official Docker image)
  • Deployment Method: Kubernetes (RKE2), official Helm chart 6.27.1
  • Number of Running Instances: 1 (microservices disabled)
  • DB Replicaset Oplog: Yes (replica set)
  • NodeJS Version: 24.22.1 (8.9.1 image)
  • MongoDB Version: 8.0.13
Client Setup Information
  • Desktop App or Browser Version: web client (current browsers), admins only (the announcements target roles: ["admin"])
  • Operating System: various
Additional context

Prometheus data (rocketchat_rest_api_count{entrypoint="banners"}):

Sync (UTC) /banners calls/min Notes
baseline ~0.3
2026-09-30 12:41 1,814 27 active REST requests, event loop lag 0.85 s, pod restarted by liveness
2026-10-29 12:32 1,670 50 active REST requests, event loop lag 0.68 s, pod restarted
2026-10-29 12:48 959 new container, restarted again
2026-10-28 00:xx / 12:xx 291 / 1,352
2026-10-30 00:xx / 12:xx 1,990 / 3,454
2026-11-01 onwards 0–8 quota reset, no more announcements

Heap stays flat (~600 MiB) and MongoDB has no slow queries during the bursts. Only the event loop is affected.

Database observations (rocketchat_banner, 2026-09-30):

  • 6,035 documents with createdBy: "cloud", view.appId: "cloud-announcements-core", roles: ["admin"], platform: ["web","mobile"]
  • About 578 not yet expired. The rest have expired but are still active: true and are never removed. They seem harmless because findActiveByRoleOrId filters on expireAt and createOrUpdate refuses expired banners, but they pile up month after month.
  • Same pattern each month since June 2026: from the ~22nd until the end of the month, nothing in between.
  • Deleting the documents only helps until the next sync, because announcements that have not expired are upserted and broadcast again.

Workaround in place: we relaxed the liveness and readiness probes so that a burst of one or two minutes no longer restarts the pod. Admins still get slow responses during each sync.

Related: #32018 (banners causing client-side performance problems), #31501, #19877 (push-limit alerts repeated).

Relevant logs:

The server logs no error or warning during the bursts: the workspace sync succeeds. The problem only shows up in the metrics above (REST request rate, active requests, event loop lag) and in the Kubernetes liveness probe failures.

Lingua principale
TypeScript
Stelle
46.5k
Fork
14.1k
Merge medio
1g 21h
PR unite (30g)
381

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di RocketChat/Rocket.Chat

Tutte le issue di RocketChat/Rocket.Chat

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.