Workspace sync re-broadcasts hundreds of duplicate "mobile notification limit reached" cloud announcements every 12h, flooding GET /api/v1/banners and restarting the server
メンテナーはふだん 1 日以内に返信
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 47/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- kubernetes, mongodb, node.js, typescript
調査の方向性
Start with apps/meteor/server/lib/cloud/syncWorkspace/handleCommsSync.ts and trace how announcements in create reach Banner.create(), Banners.createOrUpdate(), and sendToUsers(). Check the related banner and workspace sync tests, then reproduce a sync with repeated announcements if the test setup allows it. Done means unchanged announcements no longer trigger a burst of broadcasts; the issue also raises preventing duplicate announcements and cleaning up expired banners as possible scope.
索引モデルが issue の本文から書いたものです。
説明
Description:
When the workspace's monthly mobile push quota is exceeded (from around the 22nd of each month until the quota resets), Rocket.Chat Cloud sends a new cloud-announcements-core announcement many times a day (200–400 per day). They all say the same thing:
Workspace members are not receiving notifications on mobile — Your plan's monthly mobile notification limit has been reached.
Each announcement lasts about 48h, so several hundred identical banners are active at the same time.
On every workspace sync (every 12h, at 00:xx and 12:xx UTC), handleAnnouncementsOnWorkspaceSync (apps/meteor/server/lib/cloud/syncWorkspace/handleCommsSync.ts) calls Banner.create() for each announcement in create. That runs Banners.createOrUpdate() and then sendToUsers(), which broadcasts banner.enabled once per announcement, even when the banner has not changed.
Connected admin clients then send thousands of GET /api/v1/banners?platform=web requests within a minute. Each response contains all the active duplicates (around 440 KB with about 490 active banners). The Node.js event loop saturates, /health stops responding, and Kubernetes' liveness probe restarts the server.
Steps to reproduce:
- Use a registered workspace on a plan with a mobile push quota, and exceed the quota.
- Wait a few days so that Cloud creates many "notification limit reached" announcements.
- Keep several admin users connected on the web client.
- Watch
GET /api/v1/bannersrequests andnodejs_eventloop_lag_max_secondsduring the next workspace sync.
Expected behavior:
- An announcement that has not changed should not be broadcast again on every sync, or at least the broadcasts should be grouped (one event per sync rather than one per banner).
- Cloud should not create hundreds of identical announcements per month for the same condition. One active announcement per condition should be enough.
- Optionally, expired cloud announcements could be removed or disabled during sync.
Actual behavior:
Thousands of /api/v1/banners requests per minute at each sync, event loop saturation, /health timeouts, and the pod restarted by liveness. All users are disconnected at the end of every month.
The 00:xx bursts are much smaller than the 12:xx ones, which fits fewer admin clients being connected at night. The volume seems to grow with (number of active duplicate announcements) × (number of connected admin clients).
Server Setup Information:
- Version of Rocket.Chat Server: 8.9.0 when observed (also seen on 8.2.0 and 8.5.x); now on 8.9.1, where the code paths are unchanged
- License Type: Community
- Number of Users: 180
- Operating System: Linux (official Docker image)
- Deployment Method: Kubernetes (RKE2), official Helm chart 6.27.1
- Number of Running Instances: 1 (microservices disabled)
- DB Replicaset Oplog: Yes (replica set)
- NodeJS Version: 24.22.1 (8.9.1 image)
- MongoDB Version: 8.0.13
Client Setup Information
- Desktop App or Browser Version: web client (current browsers), admins only (the announcements target
roles: ["admin"]) - Operating System: various
Additional context
Prometheus data (rocketchat_rest_api_count{entrypoint="banners"}):
| Sync (UTC) | /banners calls/min |
Notes |
|---|---|---|
| baseline | ~0.3 | |
| 2026-09-30 12:41 | 1,814 | 27 active REST requests, event loop lag 0.85 s, pod restarted by liveness |
| 2026-10-29 12:32 | 1,670 | 50 active REST requests, event loop lag 0.68 s, pod restarted |
| 2026-10-29 12:48 | 959 | new container, restarted again |
| 2026-10-28 00:xx / 12:xx | 291 / 1,352 | |
| 2026-10-30 00:xx / 12:xx | 1,990 / 3,454 | |
| 2026-11-01 onwards | 0–8 | quota reset, no more announcements |
Heap stays flat (~600 MiB) and MongoDB has no slow queries during the bursts. Only the event loop is affected.
Database observations (rocketchat_banner, 2026-09-30):
- 6,035 documents with
createdBy: "cloud",view.appId: "cloud-announcements-core",roles: ["admin"],platform: ["web","mobile"] - About 578 not yet expired. The rest have expired but are still
active: trueand are never removed. They seem harmless becausefindActiveByRoleOrIdfilters onexpireAtandcreateOrUpdaterefuses expired banners, but they pile up month after month. - Same pattern each month since June 2026: from the ~22nd until the end of the month, nothing in between.
- Deleting the documents only helps until the next sync, because announcements that have not expired are upserted and broadcast again.
Workaround in place: we relaxed the liveness and readiness probes so that a burst of one or two minutes no longer restarts the pod. Admins still get slow responses during each sync.
Related: #32018 (banners causing client-side performance problems), #31501, #19877 (push-limit alerts repeated).
Relevant logs:
The server logs no error or warning during the bursts: the workspace sync succeeds. The problem only shows up in the metrics above (REST request rate, active requests, event loop lag) and in the Kubernetes liveness probe failures.
- 主要言語
- TypeScript
- スター
- 46.5k
- フォーク
- 14.1k
- 平均マージ
- 1日 21時間
- マージ済み PR(30日)
- 381
環境構築
このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
RocketChat/Rocket.Chat のほかの issue
-
Mark the empty state busy while a new search loads, like the table */}対応中かも @aditya25bcy10173-a11y が 2 日前に担当しました。 オープンtodo type: feature
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
RocketChat/Rocket.Chat#43699 · コメント 1 件 · リアクション 1 件 ·
メンテナーはふだん 1 日以内に返信
-
Timestamp with a time but no timezone shows as 1 January 1970対応中かも @Sarthak-agrawal-tech が 4 日前に担当しました。 オープンtype: bug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
RocketChat/Rocket.Chat#43637 · コメント 6 件 ·
メンテナーはふだん 1 日以内に返信
-
Move this mapping from useQuery's old `select` into a usePaginatedQuery `combine`対応中かも @aditya25bcy10173-a11y が 3 日前に担当しました。 オープンtodo type: feature
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
RocketChat/Rocket.Chat#43609 · コメント 4 件 · リアクション 1 件 ·
メンテナーはふだん 1 日以内に返信
-
evaluate switching to the other tab when the selected tab's permission is revoked mid-session対応中かも @schizophrenicmaniac が 4 日前に担当しました。 オープンtodo type: feature
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
RocketChat/Rocket.Chat#43579 · コメント 3 件 · リアクション 1 件 ·
メンテナーはふだん 1 日以内に返信
-
Docs say packages can be installed with `npm` but use unsupported `patch:` URL対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープンtype: bug
難易度 1/5 1〜3時間 初心者へのやさしさ 72/100
RocketChat/Rocket.Chat#43554 · コメント 4 件 ·
メンテナーはふだん 1 日以内に返信
RocketChat/Rocket.Chat の issue をすべて見る
似ている issue
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
langchain-ai/langchainjs#12239 ·
メンテナーはふだん 1 日以内に返信
-
Board view: the Monitoring column is always empty because compact issue rows drop displayStatusオープン
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
paperclipai/paperclip#21837 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 73/100
code-yeongyu/senpi#6813 ·
メンテナーはふだん 1 日以内に返信
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
難易度 1/5 1時間未満 初心者へのやさしさ 93/100
lingdojo/kana-dojo#34169 · コメント 1 件 · リアクション 5 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100