Send-only broadcasts do not restart event sampling after idle shutdown

Open
  • Is something wrong?
#2,465 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@hsusul is already working on this.

Since Oct 30, 2026.

  • #2466 by @hsusul — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
54/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
elixir, postgresql
Domain
backend, observability

Research direction

Start with BroadcastHandler.increment_rate_counter/1 and trace how receiving channels wake event samplers; then read RateCounter.init/1 and its idle shutdown behavior. Run the reproduction with the repository test databases and confirm the sampler is absent after send-only traffic. Done means accepted broadcasts after the shutdown cache entry expires restart sampling using current tenant configuration and produce a sampled sum of 3.

Written by the indexing model from the issue text.

Description

Bug report

  • I confirm this is a bug with Supabase, not with my own application.
  • I searched issues, PRs, discussions, branches, and local history for this behavior.

Describe the bug

An existing WebSocket channel that sends Broadcast messages with self: false and no other listeners cannot restart its tenant events sampler after idle shutdown. Messages are acknowledged successfully and increment the raw counter, but no sampler consumes those increments or emits the corresponding channel event telemetry. Continued send-only traffic leaves the sampler absent even after its shutdown cache entry expires.

To reproduce

On upstream main f5f0dab9, set up the documented repository test databases. Save the following as /tmp/broadcast_idle_repro_test.exs and run mix test /tmp/broadcast_idle_repro_test.exs.

This uses a real channel and valid Broadcast messages. It sends the sampler's normal :idle_shutdown timer message while its sampled bucket is empty, avoiding a five-minute wait. It observes normal process termination and waits for the existing one-second shutdown cache grace to expire before resuming traffic.

defmodule BroadcastIdleReproductionTest do
  use RealtimeWeb.ChannelCase, async: false
  use Mimic
  setup :set_mimic_from_context

  alias Realtime.{RateCounter, Tenants}
  alias RealtimeWeb.UserSocket

  test "a send-only channel wakes its event sampler after ordinary idle shutdown" do
    tenant = TestTenantDb.checkout_tenant(run_migrations: true)
    jwt = Generators.generate_jwt_token(tenant)

    {:ok, socket} =
      connect(UserSocket, %{},
        connect_info: %{
          uri: URI.parse("https://#{tenant.external_id}.localhost:4000/socket/websocket"),
          x_headers: [{"x-api-key", jwt}]
        }
      )

    socket = subscribe_and_join!(socket, "realtime:idle", %{"config" => %{"broadcast" => %{"ack" => true}}})
    args = Tenants.events_per_second_rate(tenant)
    [{counter_pid, _}] = Registry.lookup(Realtime.Registry.Unique, {RateCounter, :rate_counter, args.id})
    {:ok, %{sum: 0}} = RateCounterHelper.tick!(args)
    monitor = Process.monitor(counter_pid)
    send(counter_pid, :idle_shutdown)
    assert_receive {:DOWN, ^monitor, :process, ^counter_pid, :normal}, 1000
    assert_eventually RateCounter.find(args.id) == {:error, :not_found}, timeout: 3000

    for i <- 1..3 do
      ref = push(socket, "broadcast", %{"event" => "message", "payload" => %{"value" => i}})
      assert_reply ref, :ok, %{}, 1000
    end

    Phoenix.Channel.Server.socket(socket.channel_pid)
    assert {:ok, %RateCounter{}} = RateCounter.find(args.id)
    assert {:ok, %RateCounter{sum: 3}} = RateCounterHelper.tick!(args)
  end
end

Upstream fails at RateCounter.find(args.id) with {:error, :not_found}, after all three success acknowledgements. With the fix, the sampler exists and its sampled sum is 3.

Expected behavior

The first accepted Broadcast after the sampler's shutdown cache entry expires restarts event sampling. The resumed messages contribute to the tenant's sampled events and telemetry, including when the sender has no receiving clients.

Actual behavior

The sampler stays absent. All three acknowledged messages, and further send-only traffic, accumulate outside the sampling process. A later channel join or a receiving channel's counter check can restart it, but this supported send-only path cannot.

Root cause and impact

BroadcastHandler.increment_rate_counter/1 only calls GenCounter.add/1. The dispatcher wakes samplers through receiving channels. With self: false and no other listeners, there is no receiving channel to request that wake-up. RateCounter deliberately stops after five minutes with an empty bucket, and its transient supervisor does not restart a normal idle exit.

The fix must ensure the sampler exists before incrementing because RateCounter.init/1 resets the raw counter. Restart arguments must use current tenant configuration, preserving #2275. This differs from #2322's concurrent reset window and #2381's quota cache ordering.

The reproduced impact is missing tenant event sampling and telemetry. This report does not claim billing impact or add sender-side rate enforcement. The existing one-second shutdown cache grace is unchanged; this reproduction and proposed fix cover traffic after that entry expires.

System information

  • macOS arm64.
  • Upstream main f5f0dab9.
  • Elixir 1.19.5 and OTP 28.5.0.8.
  • PostgreSQL 17.11.
  • Phoenix ChannelTest rather than a client SDK.
Dominant language
Elixir
Stars
7.7k
Forks
479
Avg merge
2d 13h
Merged PRs (30d)
74

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from supabase/realtime

All issues in supabase/realtime

Similar issues

More Elixir issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.