Workspace sync re-broadcasts hundreds of duplicate "mobile notification limit reached" cloud announcements every 12h, flooding GET /api/v1/banners and restarting the server

Open
  • Is something wrong?
#43,564 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Kulraj69 is already working on this.

Since Nov 6, 2026.

  • #43568 by @Kulraj69 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
47/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
kubernetes, mongodb, node.js, typescript
Domain
backend, observability, performance

Research direction

Start with apps/meteor/server/lib/cloud/syncWorkspace/handleCommsSync.ts and trace how announcements in create reach Banner.create(), Banners.createOrUpdate(), and sendToUsers(). Check the related banner and workspace sync tests, then reproduce a sync with repeated announcements if the test setup allows it. Done means unchanged announcements no longer trigger a burst of broadcasts; the issue also raises preventing duplicate announcements and cleaning up expired banners as possible scope.

Written by the indexing model from the issue text.

Description

type: bug
Description:

When the workspace's monthly mobile push quota is exceeded (from around the 22nd of each month until the quota resets), Rocket.Chat Cloud sends a new cloud-announcements-core announcement many times a day (200–400 per day). They all say the same thing:

Workspace members are not receiving notifications on mobile — Your plan's monthly mobile notification limit has been reached.

Each announcement lasts about 48h, so several hundred identical banners are active at the same time.

On every workspace sync (every 12h, at 00:xx and 12:xx UTC), handleAnnouncementsOnWorkspaceSync (apps/meteor/server/lib/cloud/syncWorkspace/handleCommsSync.ts) calls Banner.create() for each announcement in create. That runs Banners.createOrUpdate() and then sendToUsers(), which broadcasts banner.enabled once per announcement, even when the banner has not changed.

Connected admin clients then send thousands of GET /api/v1/banners?platform=web requests within a minute. Each response contains all the active duplicates (around 440 KB with about 490 active banners). The Node.js event loop saturates, /health stops responding, and Kubernetes' liveness probe restarts the server.

Steps to reproduce:
  1. Use a registered workspace on a plan with a mobile push quota, and exceed the quota.
  2. Wait a few days so that Cloud creates many "notification limit reached" announcements.
  3. Keep several admin users connected on the web client.
  4. Watch GET /api/v1/banners requests and nodejs_eventloop_lag_max_seconds during the next workspace sync.
Expected behavior:
  • An announcement that has not changed should not be broadcast again on every sync, or at least the broadcasts should be grouped (one event per sync rather than one per banner).
  • Cloud should not create hundreds of identical announcements per month for the same condition. One active announcement per condition should be enough.
  • Optionally, expired cloud announcements could be removed or disabled during sync.
Actual behavior:

Thousands of /api/v1/banners requests per minute at each sync, event loop saturation, /health timeouts, and the pod restarted by liveness. All users are disconnected at the end of every month.

The 00:xx bursts are much smaller than the 12:xx ones, which fits fewer admin clients being connected at night. The volume seems to grow with (number of active duplicate announcements) × (number of connected admin clients).

Server Setup Information:
  • Version of Rocket.Chat Server: 8.9.0 when observed (also seen on 8.2.0 and 8.5.x); now on 8.9.1, where the code paths are unchanged
  • License Type: Community
  • Number of Users: 180
  • Operating System: Linux (official Docker image)
  • Deployment Method: Kubernetes (RKE2), official Helm chart 6.27.1
  • Number of Running Instances: 1 (microservices disabled)
  • DB Replicaset Oplog: Yes (replica set)
  • NodeJS Version: 24.22.1 (8.9.1 image)
  • MongoDB Version: 8.0.13
Client Setup Information
  • Desktop App or Browser Version: web client (current browsers), admins only (the announcements target roles: ["admin"])
  • Operating System: various
Additional context

Prometheus data (rocketchat_rest_api_count{entrypoint="banners"}):

Sync (UTC) /banners calls/min Notes
baseline ~0.3
2026-09-30 12:41 1,814 27 active REST requests, event loop lag 0.85 s, pod restarted by liveness
2026-10-29 12:32 1,670 50 active REST requests, event loop lag 0.68 s, pod restarted
2026-10-29 12:48 959 new container, restarted again
2026-10-28 00:xx / 12:xx 291 / 1,352
2026-10-30 00:xx / 12:xx 1,990 / 3,454
2026-11-01 onwards 0–8 quota reset, no more announcements

Heap stays flat (~600 MiB) and MongoDB has no slow queries during the bursts. Only the event loop is affected.

Database observations (rocketchat_banner, 2026-09-30):

  • 6,035 documents with createdBy: "cloud", view.appId: "cloud-announcements-core", roles: ["admin"], platform: ["web","mobile"]
  • About 578 not yet expired. The rest have expired but are still active: true and are never removed. They seem harmless because findActiveByRoleOrId filters on expireAt and createOrUpdate refuses expired banners, but they pile up month after month.
  • Same pattern each month since June 2026: from the ~22nd until the end of the month, nothing in between.
  • Deleting the documents only helps until the next sync, because announcements that have not expired are upserted and broadcast again.

Workaround in place: we relaxed the liveness and readiness probes so that a burst of one or two minutes no longer restarts the pod. Admins still get slow responses during each sync.

Related: #32018 (banners causing client-side performance problems), #31501, #19877 (push-limit alerts repeated).

Relevant logs:

The server logs no error or warning during the bursts: the workspace sync succeeds. The problem only shows up in the metrics above (REST request rate, active requests, event loop lag) and in the Kubernetes liveness probe failures.

Dominant language
TypeScript
Stars
46.5k
Forks
14.1k
Avg merge
1d 21h
Merged PRs (30d)
381

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from RocketChat/Rocket.Chat

All issues in RocketChat/Rocket.Chat

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.