[Bug]: Crawl4aiDockerClient cannot send 0.9 declarative hooks, and returns result.pdf as base64 text
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 70/100
Línea de trabajo
Start in the Crawl4aiDockerClient class, in its crawl() method, where hooks are currently wrapped as {"hooks": {"code": ..., "timeout": N}} and where CrawlResult(**r) is built on both /crawl and /crawl/stream. Check GET /hooks/info and deploy/docker/api.py for the 0.9 declarative format and the base64 pdf encoding. Done means: a list of {action, params} is sent as hooks.hooks, dict hooks still use the legacy path, the server's hooks warning is raised as a UserWarning, and result.pdf is decoded to raw PDF bytes on both paths.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
crawl4ai version
0.9.5 (also on develop)
Expected Behavior
Crawl4aiDockerClient.crawl(..., hooks=...)can use the hooks the 0.9 server supports: declarative actions such as{"action": "block_resources", "params": {...}}(listed byGET /hooks/info).- With
CrawlerRunConfig(pdf=True),result.pdfholds the raw PDF bytes (b'%PDF-1.4 ...'), the same asAsyncWebCrawler.
Current Behavior
Hooks. The client only knows the pre-0.9 inline-code hooks. It always sends {"hooks": {"code": {...}, "timeout": N}}, which the 0.9 server never runs:
- Hooks disabled (the default):
RequestError: Server error 403: Inline hook code (hooks.code) was removed in 0.9.0 and cannot be enabled; it was not executed. ... - Hooks enabled (
CRAWL4AI_HOOKS_ENABLED=true): the crawl returnssuccess=True, the hook does not run, and the server's warning (results["hooks"]["warning"], or theX-Hooks-Warningheader on/crawl/stream) is dropped by the client. The user is never told.
Passing the declarative format crashes in the client before any request:
AttributeError: 'list' object has no attribute 'values'
The crawl() docstring still shows Python function hooks as the example.
PDF. The server base64-encodes pdf for JSON (deploy/docker/api.py, b64encode(result_dict['pdf']), since 0.7.2), and the client passes that string straight into CrawlResult(**r). Pydantic turns the string into bytes without decoding, so result.pdf is base64 text (b'JVBERi0xLjQK...'). A file written from it is not a valid PDF, and no error is raised. Both /crawl and /crawl/stream are affected.
Is this reproducible?
Yes
Code snippets
from crawl4ai import CrawlerRunConfig
from crawl4ai.docker_client import Crawl4aiDockerClient
async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
client._http_client.headers["Authorization"] = f"Bearer {token}"
# 0.9 declarative hooks: crashes in the client
await client.crawl(
["https://httpbin.org/headers"],
hooks=[{"action": "set_headers", "params": {"headers": {"X-Test": "1"}}}],
)
# AttributeError: 'list' object has no attribute 'values'
# Legacy function hooks on a server with hooks enabled: success, hook never runs, no warning
async def before_goto(page, **kwargs):
await page.set_extra_http_headers({"X-Test": "1"})
return page
result = await client.crawl(["https://httpbin.org/headers"], hooks={"before_goto": before_goto})
print(result.success, "X-Test" in result.html) # True False
# PDF comes back as base64 text
result = await client.crawl(["https://example.com"], crawler_config=CrawlerRunConfig(pdf=True))
print(result.pdf[:10]) # b'JVBERi0xLj' (local AsyncWebCrawler: b'%PDF-1.4\n%')
Proposed fix
In Crawl4aiDockerClient:
- send a list of
{"action", "params"}as{"hooks": {"hooks": [...], "timeout": hooks_timeout}}; - keep sending a dict the legacy way, so servers older than 0.9 still run it;
- raise the server's hooks warning as a
UserWarningon both/crawland/crawl/stream; - decode
pdfwith strict base64 before buildingCrawlResult, on both paths, leaving any non-base64 value unchanged (screenshotstays as it is: it is base64 text in local crawls too).
I'll open a PR.
OS
macOS 26.6 (platform-independent)
Python version
3.13
- Lenguaje dominante
- Python
- Estrellas
- 87.2k
- Forks
- 9.1k
- Merge medio
- 1 d 6 h
- PR fusionados (30 d)
- 25
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackPosiblemente ocupada @HhH666-ux la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
unclecode/crawl4ai#2494 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAbierto⚙ Done ✨ Enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 1 día
-
⚙ Done 🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAbierto⚙ Done 🐞 Bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
unclecode/crawl4ai#2469 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de unclecode/crawl4ai
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 1 día
-
documentation
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
Install the skills as a Claude Code pluginPosiblemente ocupada @ruban-24 la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 6 días
-
Make the usage of time-zone aware UTC timestamps the norm across the pynxtools NOMAD ecosystemAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationAbiertodata
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100