AdmissionControl permits leak when HTTP/2 stream processes terminate with their connection
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- elixir
- Domain
- backend, networking
Research direction
Start with Electric.Plug.ServeShapePlug.call/2 and the AdmissionControl permit storage and ETS counter updates, then inspect Bandit's HTTP/2 stream termination behavior. Add the suggested externally terminated stream regression test, including the GOAWAY path from #4771 and a normal-completion race. Done means permits are reclaimed exactly once and a subsequent request can acquire the limit.
Written by the indexing model from the issue text.
Description
Versions
- Electric: current
main - HTTP server: Bandit 1.12.0
- Protocol: HTTP/2
Bug description
AdmissionControl permits are released by an after block in Electric.Plug.ServeShapePlug.call/2:
try do
# Run the shape request
after
release_admission_permit()
end
The acquired permit is stored only in the request process dictionary:
Process.put(@admission_permit_key, {stack_id, kind})
This handles normal returns, exceptions, throws, and exits raised within the request process. It does not handle the request process being terminated by an external exit signal, because the process stops without executing the after block.
This matters under Bandit's HTTP/2 process model:
- Each HTTP/2 stream runs its Plug pipeline in a separate
Bandit.HTTP2.StreamProcess. - The stream process is linked to the HTTP/2 connection process.
- The stream process does not trap exits.
- When the connection process terminates with a non-normal reason, its active stream processes terminate immediately.
- Any AdmissionControl permits held by those processes remain counted in the AdmissionControl ETS table.
HTTP/2 connection termination can happen when:
- The peer closes the connection while streams are active.
- The socket encounters an error.
- The server closes the connection.
- The HTTP/2
max_requestsbehavior described in #4771 sends GOAWAY and closes the connection.
Electric's live requests may spend up to long_poll_timeout waiting for changes, so it is normal for an HTTP/2 connection to have active request processes when the connection disappears.
Because the AdmissionControl ETS table is owned independently of the request and connection processes, leaked counts do not recover when those processes terminate. Repeated connection closures can ratchet the :initial or :existing counter upward until the configured concurrency limit is exhausted. At that point, subsequent requests receive 503 concurrent_request_limit_exceeded despite there being fewer—or no—corresponding live request processes.
The counters only recover when AdmissionControl or the VM restarts.
Expected behavior
An AdmissionControl permit should be released exactly once when its owning request process terminates, regardless of whether the request:
- Returns normally.
- Raises or exits internally.
- Is terminated by an external exit signal.
- Is killed because its HTTP/2 connection closes.
AdmissionControl's counters should represent live permit holders rather than relying exclusively on cooperative cleanup by those holders.
Suggested fix
Associate each acquired permit with its owning request process and monitor that process.
Both normal release and owner-process termination should attempt to atomically claim the permit lease before decrementing the counter. Only the path that successfully claims the lease should perform the decrement. This provides exactly-once release when normal cleanup races with a :DOWN notification.
The current direct ETS counter updates are intentionally cheap, so the implementation should avoid serializing every stack through one global GenServer. Possible implementations include partitioned lease monitors or another ownership mechanism that preserves concurrency across stacks.
Making HTTP/2 stream processes trap connection exits would be a narrower workaround, but it would only defer cleanup until the request returns and would couple Electric to Bandit's process model. Owner-monitored permits address the underlying AdmissionControl invariant regardless of why a request process terminates.
Suggested regression test
- Start Bandit with HTTP/2 enabled.
- Configure an AdmissionControl limit of one.
- Open a live request that remains in its long-poll wait and verify the counter is one.
- Close the underlying HTTP/2 connection while the stream remains active.
- Wait for the stream process to terminate.
- Assert that the AdmissionControl counter returns to zero.
- Assert that another request can acquire the permit.
A second test should cover the GOAWAY path from #4771 by configuring a very small HTTP/2 max_requests value.
The test should also verify that normal request completion does not cause a double release if process-death cleanup races with the existing after block.
Acceptance criteria
- Admission permits are associated with their owning request processes.
- A permit is reclaimed when its owner terminates unexpectedly.
- Normal completion and owner-death cleanup cannot release the same permit twice.
- HTTP/2 connection closure with active streams does not leave AdmissionControl counters elevated.
- A regression test covers externally terminated HTTP/2 stream processes.
- #4771 remains the mitigation for avoiding request-count-driven HTTP/2 connection cycling; this issue does not reintroduce an HTTP/2 request cap.
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 399
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryOpen
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error valuePossibly taken @Tyagiquamar claimed this 19 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)Possibly taken @mohanedmoh claimed this 37 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4826 · 1 comment ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wakePossibly taken @adityavkk claimed this 133 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
electric-sql/electric#4687 ·
-
Listen IP addressOpenfeature request triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
electric-sql/electric#4150 · 1 comment ·
All issues in electric-sql/electric
Similar issues
-
area:cli area:security bug priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
"code source" and "numerique.gouv" links: "new window" mention duplicated in the accessible nameOpenaccessibility
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
numerique-gouv/lasuite-landingpage#315 ·
Maintainers usually reply within 1 day
-
needs:triage
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
HyeokjaeChoi/vaultme#585 ·
Maintainers usually reply within 1 day
-
enhancement priority/P2
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
beettlle/pi-smart-router#200 ·