[Bug]: PDF scrape thread outlives Docker crawl deadline; no total download/parse timeout
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 35/100
Línea de trabajo
Read crawl4ai/processors/pdf/__init__.py, starting with PDFContentScrapingStrategy.ascrap, _fetch_with_redirect_checks, and _get_pdf_path, then compare their worker lifetime with the Docker /crawl timeout in deploy/docker/api.py. The issue proposes slow-stream and delayed-parser regression tests, including repeated cancellations. Done means a configurable total download and parse budget is enforced, workers and temporary PDFs are cleaned up after timeout or cancellation, and existing PDF limits still work.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Summary
Source-inspected cancellation/lifetime gap, observed in upstream main and v0.9.5; not yet a measured live exploit.
PDFContentScrapingStrategy.ascrap() calls await asyncio.to_thread(self.scrap, url, html, **kwargs). scrap() synchronously downloads and parses the PDF. The downloader's requests.Session.get(..., stream=True, timeout=(20, 600), allow_redirects=False) uses a 600-second per-read timeout, not a total elapsed deadline. It iterates chunks and checks max_pdf_bytes, but does not check a monotonic elapsed deadline.
The Docker /crawl endpoint wraps the coroutine in asyncio.wait_for(..., wall_clock_s). Cancelling that coroutine cannot forcibly stop a thread already running the synchronous download or PDF parser. A source that steadily streams below the configured byte cap may keep the worker active well beyond the API/bridge timeout; a stalled parser may similarly persist. Repeated operations can accumulate lingering work, defeating the expected isolation of a long-lived self-hosted Docker crawler.
Relevant locations (v0.9.5 and current main)
crawl4ai/processors/pdf/__init__.pyascrap: offloads synchronous download + parse usingasyncio.to_thread._fetch_with_redirect_checks:timeout=(20, 600)per socket operation._get_pdf_path: streamed byte cap checked but no elapsed-time cap.- Docker API:
asyncio.wait_foris coroutine-scoped, not killable-thread-scoped.
Expected
A configurable, enforced PDF total elapsed download + parse budget, with no lingering worker or temporary PDF after timeout/cancellation. This should coexist with the existing (valuable) download-byte/page caps, redirect validation, and peer-IP/egress protections.
Proposed focused regression tests
- Local loopback HTTP fixture that streams one small chunk periodically, always within the 600-second per-read timeout, while exceeding a short total PDF budget. Once the budget expires, verify the worker has terminated and temporary file has been removed—not merely that the HTTP client received a 504.
- Deliberately delayed PDF parsing on a local fixture; cancel the API crawl and verify no persistent processing thread/process or temporary file remains after the budget.
- Repeat the timed-out request a few times and assert bounded worker count and resource cleanup. Verify normal small PDFs and byte/page/redirect limits still work.
An elapsed-time check inside the chunk loop plus a shorter socket read timeout helps the download path but does not guarantee prompt cancellation of an already-blocked read or synchronous parsing. If the parser cannot be interrupted safely in a thread, a killable subprocess or another lifecycle-bound worker may be needed. The aim is a narrow upstream fix, not general queue/supervisor infrastructure.
Context / related work
- The open #2236 adds page-specific crawl timeouts and cancellation-safe page cleanup, but its code path is for browser pages, not
PDFContentScrapingStrategysynchronous worker threads. - #2319 reports a different concurrent PDF page-numbering race; it is not a solution to cancellation.
- This was found when prototyping bounded native PDF extraction in a guarded personal MCP deployment. We closed the prototype unmerged because the current PDF processing lifetime cannot be bounded, even with 12 MiB / 100-page caps and 0 redirects.
Impact: potential long-running worker/resource exhaustion in self-hosted Docker research deployments. This report is based on source inspection; a live slow-stream reproduction has not yet been performed.
- Lenguaje dominante
- Python
- Estrellas
- 87.2k
- Forks
- 9.1k
- Merge medio
- 1 d 6 h
- PR fusionados (30 d)
- 25
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackPosiblemente ocupada @HhH666-ux la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
unclecode/crawl4ai#2494 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAbierto⚙ Done ✨ Enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 1 día
-
⚙ Done 🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAbierto⚙ Done 🐞 Bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
unclecode/crawl4ai#2469 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de unclecode/crawl4ai
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 1 día
-
documentation
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
Install the skills as a Claude Code pluginPosiblemente ocupada @ruban-24 la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 6 días
-
Make the usage of time-zone aware UTC timestamps the norm across the pynxtools NOMAD ecosystemAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationAbiertodata
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100