[Bug]: PDF scrape thread outlives Docker crawl deadline; no total download/parse timeout
维护者通常 1 天内回复
评估
调研方向
Read crawl4ai/processors/pdf/__init__.py, starting with PDFContentScrapingStrategy.ascrap, _fetch_with_redirect_checks, and _get_pdf_path, then compare their worker lifetime with the Docker /crawl timeout in deploy/docker/api.py. The issue proposes slow-stream and delayed-parser regression tests, including repeated cancellations. Done means a configurable total download and parse budget is enforced, workers and temporary PDFs are cleaned up after timeout or cancellation, and existing PDF limits still work.
由索引模型根据 Issue 内容生成。
描述
Summary
Source-inspected cancellation/lifetime gap, observed in upstream main and v0.9.5; not yet a measured live exploit.
PDFContentScrapingStrategy.ascrap() calls await asyncio.to_thread(self.scrap, url, html, **kwargs). scrap() synchronously downloads and parses the PDF. The downloader's requests.Session.get(..., stream=True, timeout=(20, 600), allow_redirects=False) uses a 600-second per-read timeout, not a total elapsed deadline. It iterates chunks and checks max_pdf_bytes, but does not check a monotonic elapsed deadline.
The Docker /crawl endpoint wraps the coroutine in asyncio.wait_for(..., wall_clock_s). Cancelling that coroutine cannot forcibly stop a thread already running the synchronous download or PDF parser. A source that steadily streams below the configured byte cap may keep the worker active well beyond the API/bridge timeout; a stalled parser may similarly persist. Repeated operations can accumulate lingering work, defeating the expected isolation of a long-lived self-hosted Docker crawler.
Relevant locations (v0.9.5 and current main)
crawl4ai/processors/pdf/__init__.pyascrap: offloads synchronous download + parse usingasyncio.to_thread._fetch_with_redirect_checks:timeout=(20, 600)per socket operation._get_pdf_path: streamed byte cap checked but no elapsed-time cap.- Docker API:
asyncio.wait_foris coroutine-scoped, not killable-thread-scoped.
Expected
A configurable, enforced PDF total elapsed download + parse budget, with no lingering worker or temporary PDF after timeout/cancellation. This should coexist with the existing (valuable) download-byte/page caps, redirect validation, and peer-IP/egress protections.
Proposed focused regression tests
- Local loopback HTTP fixture that streams one small chunk periodically, always within the 600-second per-read timeout, while exceeding a short total PDF budget. Once the budget expires, verify the worker has terminated and temporary file has been removed—not merely that the HTTP client received a 504.
- Deliberately delayed PDF parsing on a local fixture; cancel the API crawl and verify no persistent processing thread/process or temporary file remains after the budget.
- Repeat the timed-out request a few times and assert bounded worker count and resource cleanup. Verify normal small PDFs and byte/page/redirect limits still work.
An elapsed-time check inside the chunk loop plus a shorter socket read timeout helps the download path but does not guarantee prompt cancellation of an already-blocked read or synchronous parsing. If the parser cannot be interrupted safely in a thread, a killable subprocess or another lifecycle-bound worker may be needed. The aim is a narrow upstream fix, not general queue/supervisor infrastructure.
Context / related work
- The open #2236 adds page-specific crawl timeouts and cancellation-safe page cleanup, but its code path is for browser pages, not
PDFContentScrapingStrategysynchronous worker threads. - #2319 reports a different concurrent PDF page-numbering race; it is not a solution to cancellation.
- This was found when prototyping bounded native PDF extraction in a guarded personal MCP deployment. We closed the prototype unmerged because the current PDF processing lifetime cannot be bounded, even with 12 MiB / 100-page caps and 0 redirects.
Impact: potential long-running worker/resource exhaustion in self-hosted Docker research deployments. This report is based on source inspection; a live slow-stream reproduction has not yet been performed.
- 主要语言
- Python
- 星标
- 87.2k
- 派生
- 9.1k
- 平均合并
- 1 天 6 小时
- 30 天内合并 PR
- 25
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
unclecode/crawl4ai 的其他 Issue
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stack可能已有人在做 @HhH666-ux 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.config可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 70/100
unclecode/crawl4ai#2494 · 1 条评论 ·
维护者通常 1 天内回复
-
⚙ Done ✨ Enhancement
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 1 天内回复
-
⚙ Done 🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.media未关闭⚙ Done 🐞 Bug
难度 2/5 1-3 小时 新手友好度 78/100
unclecode/crawl4ai#2469 · 1 条评论 ·
维护者通常 1 天内回复
查看 unclecode/crawl4ai 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
documentation
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
Install the skills as a Claude Code plugin可能已有人在做 @ruban-24 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 6 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
data
难度 2/5 1-3 小时 新手友好度 66/100