Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: Post-navigation page operations have no timeout, and page cleanup is skipped on task cancellation — pages can leak and crawls can hang forever

Aperta
  • Qualcosa non va?
#2,205 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@SohamKukreti ci sta già lavorando.

Dal 8/9/2026.

  • #2236 di @SohamKukreti — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start in async_crawler_strategy.py at the post-navigation operations and the cleanup finally block, then inspect browser_adapter.py and browser_manager.py for timeout behavior. Reproduce the busy-page hang and cancellation path described in the issue. Done means post-navigation work is bounded by page_timeout and cancelled tasks still close their pages.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

⚙️ In-progress 🐞 Bug

Summary

Found while investigating #2202 (Docker containers accumulating renderer processes for weeks). Two related defects in async_crawler_strategy.py make it possible for a crawl to hang indefinitely and for its page to never be closed.

1. Nothing after navigation has a timeout

page_timeout only bounds page.goto() (async_crawler_strategy.py:762-764). Every page interaction after navigation is an un-timed call into the page's JS engine:

  • overlay/consent removal — remove_overlay_elements (:1550)
  • body-visibility check, css_selector extraction (:1102), image-dimension updates
  • page.content() (:1115)
  • shadow-DOM flattening, iframe processing, user js_code

The adapter is a plain pass-through to page.evaluate (browser_adapter.py:61-65), which has no timeout in Playwright. The repo never calls set_default_timeout except inside if self.config.accept_downloads: (browser_manager.py:1215-1217) — and default timeouts would not cover evaluate anyway. Only scan_full_page is wrapped in asyncio.wait_for.

A page whose main thread goes busy after DOMContentLoaded (bad loop, broken ad script, hostile page) therefore hangs the crawl forever, holding its page (= one renderer process) open. Verified with a page that starts a busy loop right after DCL: navigation succeeds, then arun() never returns, far past page_timeout.

2. The page-closing finally doesn't survive cancellation

The cleanup block (async_crawler_strategy.py:1216-1231) guards with except Exception. asyncio.CancelledError is a BaseException, so a task cancelled during the first await in that block (release_page_with_context) propagates out and page.close() is never reached — the page and its context refcount leak. Reachable from dispatcher/stream teardown paths that cancel in-flight tasks.

Impact

In long-lived processes (the Docker server's browser pool, any SDK user reusing a crawler), leaked pages accumulate indefinitely. Combined with the launch flags that disable background throttling, each leaked page also burns CPU forever. This is the core mechanism behind the multi-week renderer accumulation in #2202.

Proposed fix

  • Wrap the post-navigation phase (or at minimum every evaluate/content() call) in asyncio.wait_for with a budget derived from page_timeout, so a crawl always terminates.
  • Make the cleanup finally cancellation-safe: catch BaseException (re-raising CancelledError after the page is closed) or shield the close.
Lingua principale
Python
Stelle
85.4k
Fork
8.9k
Merge medio
5g 13h
PR unite (30g)
28

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.