Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Bug]: Docker server deadlocks on browser recycle: get_page() leaks context refcounts on cancellation, then waits forever for a drain slot

オープン
  • 何か問題がありますか？
#2,419 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
49/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
docker, playwright, python
領域
backend, devtools

調査の方向性

Read crawl4ai/browser_manager.py, especially BrowserManager.get_page() and _maybe_bump_browser_version(), to trace how context refcounts and pending cleanup interact during cancellation and recycling. Reproduce with wall_clock_s: 3, max_pages_before_recycle: 5, and the slow local paths described; compare behavior against the reported soak results. Done means cancelled or failed calls release acquired resources, recycling does not block callers indefinitely, and the reproducer continues serving requests.

索引モデルが issue の本文から書いたものです。

説明

🐞 Bug 🩺 Needs Triage
crawl4ai version

0.9.5 (Docker image unclecode/crawl4ai:0.9.5). crawl4ai/browser_manager.py is unchanged on develop.

Expected Behavior

With browser recycling on (max_pages_before_recycle), a cancelled or failed get_page() gives back the context refcount it took, old browser versions drain, and the server keeps serving.

Current Behavior

The Docker server deadlocks after a few hours of normal traffic. Every /crawl returns 504 at wall_clock_s, the log repeats Waiting for old browser to drain (pending: 3), and PIDs climb until the container hits its limit. Only a restart clears it. We saw this in production every 10 to 14 hours.

Cause, in BrowserManager.get_page() (v0.9.5 browser_manager.py lines 1593 to 1768):

  1. The context refcount is incremented inside _contexts_lock, then several awaits follow before the page is returned: closing an LRU-evicted context, context.new_page(), _apply_stealth_to_page(), and _maybe_bump_browser_version().
  2. The refcount is given back only on an Exception from new_page(). Anything else that raises after the increment leaks it, and so does any cancellation (asyncio.CancelledError is a BaseException). The Docker server cancels any crawl still running at wall_clock_s, so in production this happens constantly.
  3. After a recycle bump, a context with a leaked refcount never reaches 0, so its entry stays in _pending_cleanup forever.
  4. Once 3 entries are stuck (_max_pending_browsers = 3, which counts context signatures, not browser versions), _maybe_bump_browser_version() loops waiting for a slot. It does so inside get_page(), after the page is created and before it is returned, so the caller never gets a page it could release. Every crawl then waits until the deadline cancels it inside that wait, which leaks more refcounts.

This is separate from #2205 / #2236. Those fix the finally cleanup in _crawl_web (and add crawl_timeout). The leak and the blocking wait above are in browser_manager.py, which #2236 does not touch. We checked: stock 0.9.5 with #2236's library changes applied (async_configs.py, async_crawler_strategy.py) fails the same way in the soak below: every request from round 2 on is a 504, the final fast crawl is a 504, and the container ends at 11,060 PIDs (we did not count drain lines on that run).

Is this reproducible?

Yes

Inputs Causing the Bug

Any steady /crawl traffic where some requests hit wall_clock_s, with recycling on. A low wall_clock_s and max_pages_before_recycle make it quick.

Steps to Reproduce
  1. Copy /app/config.yml from the 0.9.5 image and set wall_clock_s: 3 and max_pages_before_recycle: 5. Mount it over /app/config.yml.
  2. Run the container with CRAWL4AI_ALLOW_INTERNAL_URLS=true and a local HTTP server inside it, where a third of the paths sleep 20 s before answering.
  3. Send 8 rounds of 150 single-URL /crawl requests at 30 concurrent.
  4. Then send one fast crawl to a page that answers at once.

Results (all on the same host, throwaway containers):

Build Rounds 2 to 8 Final fast crawl PIDs at end
stock 0.9.5 every request 504, drain loop in the log 504 10,161
stock 0.9.5 + #2236 (library files) every request 504 504 11,060
0.9.5 + our patch every round serves, no drain loop 200 about 200
Code snippets
# v0.9.5 BrowserManager.get_page(), simplified
async with self._contexts_lock:
    ...
    self._context_refcounts[config_signature] = self._context_refcounts.get(config_signature, 0) + 1
...
try:
    page = await context.new_page()
except Exception:            # CancelledError is not caught; nor is anything after this line
    self._context_refcounts[config_signature] -= 1
    raise
await self._apply_stealth_to_page(page)
await self._maybe_bump_browser_version()   # can wait indefinitely for a drain slot
return page, context
OS

Linux (Docker, host kernel 7.0)

Python version

3.12.15 (image default)

Browser

Chromium headless shell (image default)

Browser version

Playwright 1.64.0, chromium_headless_shell-1248

Error logs & Screenshots (if applicable)
Waiting for old browser to drain (pending: 3)

(BROWSER tag, debug level) repeating, with every /crawl returning 504 at wall_clock_s.

We have a fix running in production since 2026-10-17 17:06Z, with no drain loop and no deadlock restarts since. In short:

  • The recycle bump never blocks. When 3 old versions are still draining, it first cleans pending entries whose refcount is already 0, then defers and lets a later request retry.
  • get_page() records what it took (refcount, page, whether it created the page) and gives back exactly that on any BaseException, with no await.
  • new_page() / new_context() are awaited shielded, so an object Playwright creates after the caller was cancelled is closed when it arrives.

It comes with 16 tests (15 with fake contexts, 1 with a real browser). 15 of them fail on stock 0.9.5. Happy to open a PR against develop if that is useful.

主要言語
Python
スター
85.9k
フォーク
8.9k
平均マージ
5日 6時間
マージ済み PR（30日）
24

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

unclecode/crawl4ai のほかの issue

unclecode/crawl4ai の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。