[Bug]: PDF scrape thread outlives Docker crawl deadline; no total download/parse timeout

未关闭
#2,460 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@prassu017 已经在做这个了。

开始于 2026年10月28日。

  • #2546 来自 @ntohidi —— 未关闭

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
docker, python
领域
backend, devops

调研方向

Read crawl4ai/processors/pdf/__init__.py, starting with PDFContentScrapingStrategy.ascrap, _fetch_with_redirect_checks, and _get_pdf_path, then compare their worker lifetime with the Docker /crawl timeout in deploy/docker/api.py. The issue proposes slow-stream and delayed-parser regression tests, including repeated cancellations. Done means a configurable total download and parse budget is enforced, workers and temporary PDFs are cleaned up after timeout or cancellation, and existing PDF limits still work.

由索引模型根据 Issue 内容生成。

描述

⚙ Done 🐞 Bug 📌 Root caused

Summary

Source-inspected cancellation/lifetime gap, observed in upstream main and v0.9.5; not yet a measured live exploit.

PDFContentScrapingStrategy.ascrap() calls await asyncio.to_thread(self.scrap, url, html, **kwargs). scrap() synchronously downloads and parses the PDF. The downloader's requests.Session.get(..., stream=True, timeout=(20, 600), allow_redirects=False) uses a 600-second per-read timeout, not a total elapsed deadline. It iterates chunks and checks max_pdf_bytes, but does not check a monotonic elapsed deadline.

The Docker /crawl endpoint wraps the coroutine in asyncio.wait_for(..., wall_clock_s). Cancelling that coroutine cannot forcibly stop a thread already running the synchronous download or PDF parser. A source that steadily streams below the configured byte cap may keep the worker active well beyond the API/bridge timeout; a stalled parser may similarly persist. Repeated operations can accumulate lingering work, defeating the expected isolation of a long-lived self-hosted Docker crawler.

Relevant locations (v0.9.5 and current main)
Expected

A configurable, enforced PDF total elapsed download + parse budget, with no lingering worker or temporary PDF after timeout/cancellation. This should coexist with the existing (valuable) download-byte/page caps, redirect validation, and peer-IP/egress protections.

Proposed focused regression tests
  1. Local loopback HTTP fixture that streams one small chunk periodically, always within the 600-second per-read timeout, while exceeding a short total PDF budget. Once the budget expires, verify the worker has terminated and temporary file has been removed—not merely that the HTTP client received a 504.
  2. Deliberately delayed PDF parsing on a local fixture; cancel the API crawl and verify no persistent processing thread/process or temporary file remains after the budget.
  3. Repeat the timed-out request a few times and assert bounded worker count and resource cleanup. Verify normal small PDFs and byte/page/redirect limits still work.

An elapsed-time check inside the chunk loop plus a shorter socket read timeout helps the download path but does not guarantee prompt cancellation of an already-blocked read or synchronous parsing. If the parser cannot be interrupted safely in a thread, a killable subprocess or another lifecycle-bound worker may be needed. The aim is a narrow upstream fix, not general queue/supervisor infrastructure.

Context / related work
  • The open #2236 adds page-specific crawl timeouts and cancellation-safe page cleanup, but its code path is for browser pages, not PDFContentScrapingStrategy synchronous worker threads.
  • #2319 reports a different concurrent PDF page-numbering race; it is not a solution to cancellation.
  • This was found when prototyping bounded native PDF extraction in a guarded personal MCP deployment. We closed the prototype unmerged because the current PDF processing lifetime cannot be bounded, even with 12 MiB / 100-page caps and 0 redirects.

Impact: potential long-running worker/resource exhaustion in self-hosted Docker research deployments. This report is based on source inspection; a live slow-stream reproduction has not yet been performed.

主要语言
Python
星标
87.2k
派生
9.1k
平均合并
1 天 6 小时
30 天内合并 PR
25

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。