[Bug]: PDF scrape thread outlives Docker crawl deadline; no total download/parse timeout

Abierto
#2,460 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@prassu017 ya está trabajando en esto.

Desde el 28/10/2026.

  • #2546 de @ntohidi — abierto

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
docker, python
Área
backend, devops

Línea de trabajo

Read crawl4ai/processors/pdf/__init__.py, starting with PDFContentScrapingStrategy.ascrap, _fetch_with_redirect_checks, and _get_pdf_path, then compare their worker lifetime with the Docker /crawl timeout in deploy/docker/api.py. The issue proposes slow-stream and delayed-parser regression tests, including repeated cancellations. Done means a configurable total download and parse budget is enforced, workers and temporary PDFs are cleaned up after timeout or cancellation, and existing PDF limits still work.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

⚙ Done 🐞 Bug 📌 Root caused

Summary

Source-inspected cancellation/lifetime gap, observed in upstream main and v0.9.5; not yet a measured live exploit.

PDFContentScrapingStrategy.ascrap() calls await asyncio.to_thread(self.scrap, url, html, **kwargs). scrap() synchronously downloads and parses the PDF. The downloader's requests.Session.get(..., stream=True, timeout=(20, 600), allow_redirects=False) uses a 600-second per-read timeout, not a total elapsed deadline. It iterates chunks and checks max_pdf_bytes, but does not check a monotonic elapsed deadline.

The Docker /crawl endpoint wraps the coroutine in asyncio.wait_for(..., wall_clock_s). Cancelling that coroutine cannot forcibly stop a thread already running the synchronous download or PDF parser. A source that steadily streams below the configured byte cap may keep the worker active well beyond the API/bridge timeout; a stalled parser may similarly persist. Repeated operations can accumulate lingering work, defeating the expected isolation of a long-lived self-hosted Docker crawler.

Relevant locations (v0.9.5 and current main)
Expected

A configurable, enforced PDF total elapsed download + parse budget, with no lingering worker or temporary PDF after timeout/cancellation. This should coexist with the existing (valuable) download-byte/page caps, redirect validation, and peer-IP/egress protections.

Proposed focused regression tests
  1. Local loopback HTTP fixture that streams one small chunk periodically, always within the 600-second per-read timeout, while exceeding a short total PDF budget. Once the budget expires, verify the worker has terminated and temporary file has been removed—not merely that the HTTP client received a 504.
  2. Deliberately delayed PDF parsing on a local fixture; cancel the API crawl and verify no persistent processing thread/process or temporary file remains after the budget.
  3. Repeat the timed-out request a few times and assert bounded worker count and resource cleanup. Verify normal small PDFs and byte/page/redirect limits still work.

An elapsed-time check inside the chunk loop plus a shorter socket read timeout helps the download path but does not guarantee prompt cancellation of an already-blocked read or synchronous parsing. If the parser cannot be interrupted safely in a thread, a killable subprocess or another lifecycle-bound worker may be needed. The aim is a narrow upstream fix, not general queue/supervisor infrastructure.

Context / related work
  • The open #2236 adds page-specific crawl timeouts and cancellation-safe page cleanup, but its code path is for browser pages, not PDFContentScrapingStrategy synchronous worker threads.
  • #2319 reports a different concurrent PDF page-numbering race; it is not a solution to cancellation.
  • This was found when prototyping bounded native PDF extraction in a guarded personal MCP deployment. We closed the prototype unmerged because the current PDF processing lifetime cannot be bounded, even with 12 MiB / 100-page caps and 0 redirects.

Impact: potential long-running worker/resource exhaustion in self-hosted Docker research deployments. This report is based on source inspection; a live slow-stream reproduction has not yet been performed.

Lenguaje dominante
Python
Estrellas
87.2k
Forks
9.1k
Merge medio
1 d 6 h
PR fusionados (30 d)
25

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.