[Bug]: Docker server deadlocks on browser recycle: get_page() leaks context refcounts on cancellation, then waits forever for a drain slot

Read crawl4ai/browser_manager.py, especially BrowserManager.get_page() and _maybe_bump_browser_version(), to trace how context refcounts and pending cleanup interact during cancellation and recycling. Reproduce with wall_clock_s: 3, max_pages_before_recycle: 5, and the slow local paths described; compare behavior against the reported soak results. Done means cancelled or failed calls release acquired resources, recycling does not block callers indefinitely, and the reproducer continues serving requests.

crawl4ai version

0.9.5 (Docker image unclecode/crawl4ai:0.9.5). crawl4ai/browser_manager.py is unchanged on develop.

Expected Behavior

With browser recycling on (max_pages_before_recycle), a cancelled or failed get_page() gives back the context refcount it took, old browser versions drain, and the server keeps serving.

Current Behavior

The Docker server deadlocks after a few hours of normal traffic. Every /crawl returns 504 at wall_clock_s, the log repeats Waiting for old browser to drain (pending: 3), and PIDs climb until the container hits its limit. Only a restart clears it. We saw this in production every 10 to 14 hours.

Cause, in BrowserManager.get_page() (v0.9.5 browser_manager.py lines 1593 to 1768):

1. The context refcount is incremented inside _contexts_lock, then several awaits follow before the page is returned: closing an LRU-evicted context, context.new_page(), _apply_stealth_to_page(), and _maybe_bump_browser_version().
2. The refcount is given back only on an Exception from new_page(). Anything else that raises after the increment leaks it, and so does any cancellation (asyncio.CancelledError is a BaseException). The Docker server cancels any crawl still running at wall_clock_s, so in production this happens constantly.
3. After a recycle bump, a context with a leaked refcount never reaches 0, so its entry stays in _pending_cleanup forever.
4. Once 3 entries are stuck (_max_pending_browsers = 3, which counts context signatures, not browser versions), _maybe_bump_browser_version() loops waiting for a slot. It does so inside get_page(), after the page is created and before it is returned, so the caller never gets a page it could release. Every crawl then waits until the deadline cancels it inside that wait, which leaks more refcounts.

This is separate from #2205 / #2236. Those fix the finally cleanup in _crawl_web (and add crawl_timeout). The leak and the blocking wait above are in browser_manager.py, which #2236 does not touch. We checked: stock 0.9.5 with #2236's library changes applied (async_configs.py, async_crawler_strategy.py) fails the same way in the soak below: every request from round 2 on is a 504, the final fast crawl is a 504, and the container ends at 11,060 PIDs (we did not count drain lines on that run).

Is this reproducible?

Yes

Inputs Causing the Bug

Any steady /crawl traffic where some requests hit wall_clock_s, with recycling on. A low wall_clock_s and max_pages_before_recycle make it quick.

Steps to Reproduce

1. Copy /app/config.yml from the 0.9.5 image and set wall_clock_s: 3 and max_pages_before_recycle: 5. Mount it over /app/config.yml.
2. Run the container with CRAWL4AI_ALLOW_INTERNAL_URLS=true and a local HTTP server inside it, where a third of the paths sleep 20 s before answering.
3. Send 8 rounds of 150 single-URL /crawl requests at 30 concurrent.
4. Then send one fast crawl to a page that answers at once.

Results (all on the same host, throwaway containers):

Build	Rounds 2 to 8	Final fast crawl	PIDs at end
stock 0.9.5	every request 504, drain loop in the log	504	10,161
stock 0.9.5 + #2236 (library files)	every request 504	504	11,060
0.9.5 + our patch	every round serves, no drain loop	200	about 200

Code snippets

# v0.9.5 BrowserManager.get_page(), simplified
async with self._contexts_lock:
    ...
    self._context_refcounts[config_signature] = self._context_refcounts.get(config_signature, 0) + 1
...
try:
    page = await context.new_page()
except Exception:            # CancelledError is not caught; nor is anything after this line
    self._context_refcounts[config_signature] -= 1
    raise
await self._apply_stealth_to_page(page)
await self._maybe_bump_browser_version()   # can wait indefinitely for a drain slot
return page, context

OS

Linux (Docker, host kernel 7.0)

Python version

3.12.15 (image default)

Browser

Chromium headless shell (image default)

Browser version

Playwright 1.64.0, chromium_headless_shell-1248

Error logs & Screenshots (if applicable)

Waiting for old browser to drain (pending: 3)

(BROWSER tag, debug level) repeating, with every /crawl returning 504 at wall_clock_s.

We have a fix running in production since 2026-10-17 17:06Z, with no drain loop and no deadlock restarts since. In short:

• The recycle bump never blocks. When 3 old versions are still draining, it first cleans pending entries whose refcount is already 0, then defers and lets a later request retry.
• get_page() records what it took (refcount, page, whether it created the page) and gives back exactly that on any BaseException, with no await.
• new_page() / new_context() are awaited shielded, so an object Playwright creates after the caller was cancelled is closed when it arrives.

It comes with 16 tests (15 with fake contexts, 1 with a real browser). 15 of them fail on stock 0.9.5. Happy to open a PR against develop if that is useful.