Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Bug]: Post-navigation page operations have no timeout, and page cleanup is skipped on task cancellation — pages can leak and crawls can hang forever

未关闭
  • 哪里不对？
#2,205 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

@SohamKukreti 已经在做这个了。

开始于 2026年9月8日。

  • #2236 来自 @SohamKukreti —— 未关闭

评估

难度
4/5
预计耗时
3-5 天
新手友好度
52/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
backend

调研方向

Start in async_crawler_strategy.py at the post-navigation operations and the cleanup finally block, then inspect browser_adapter.py and browser_manager.py for timeout behavior. Reproduce the busy-page hang and cancellation path described in the issue. Done means post-navigation work is bounded by page_timeout and cancelled tasks still close their pages.

由索引模型根据 Issue 内容生成。

描述

⚙️ In-progress 🐞 Bug

Summary

Found while investigating #2202 (Docker containers accumulating renderer processes for weeks). Two related defects in async_crawler_strategy.py make it possible for a crawl to hang indefinitely and for its page to never be closed.

1. Nothing after navigation has a timeout

page_timeout only bounds page.goto() (async_crawler_strategy.py:762-764). Every page interaction after navigation is an un-timed call into the page's JS engine:

  • overlay/consent removal — remove_overlay_elements (:1550)
  • body-visibility check, css_selector extraction (:1102), image-dimension updates
  • page.content() (:1115)
  • shadow-DOM flattening, iframe processing, user js_code

The adapter is a plain pass-through to page.evaluate (browser_adapter.py:61-65), which has no timeout in Playwright. The repo never calls set_default_timeout except inside if self.config.accept_downloads: (browser_manager.py:1215-1217) — and default timeouts would not cover evaluate anyway. Only scan_full_page is wrapped in asyncio.wait_for.

A page whose main thread goes busy after DOMContentLoaded (bad loop, broken ad script, hostile page) therefore hangs the crawl forever, holding its page (= one renderer process) open. Verified with a page that starts a busy loop right after DCL: navigation succeeds, then arun() never returns, far past page_timeout.

2. The page-closing finally doesn't survive cancellation

The cleanup block (async_crawler_strategy.py:1216-1231) guards with except Exception. asyncio.CancelledError is a BaseException, so a task cancelled during the first await in that block (release_page_with_context) propagates out and page.close() is never reached — the page and its context refcount leak. Reachable from dispatcher/stream teardown paths that cancel in-flight tasks.

Impact

In long-lived processes (the Docker server's browser pool, any SDK user reusing a crawler), leaked pages accumulate indefinitely. Combined with the launch flags that disable background throttling, each leaked page also burns CPU forever. This is the core mechanism behind the multi-week renderer accumulation in #2202.

Proposed fix

  • Wrap the post-navigation phase (or at minimum every evaluate/content() call) in asyncio.wait_for with a budget derived from page_timeout, so a crawl always terminates.
  • Make the cleanup finally cancellation-safe: catch BaseException (re-raising CancelledError after the page is closed) or shield the close.
主要语言
Python
星标
85.9k
派生
8.9k
平均合并
5 天 6 小时
30 天内合并 PR
21

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。