[Bug]: Crawl4aiDockerClient cannot send 0.9 declarative hooks, and returns result.pdf as base64 text
Les mainteneurs répondent en général sous 1 jour
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Accessibilité débutants
- 70/100
Piste de recherche
Start in the Crawl4aiDockerClient class, in its crawl() method, where hooks are currently wrapped as {"hooks": {"code": ..., "timeout": N}} and where CrawlResult(**r) is built on both /crawl and /crawl/stream. Check GET /hooks/info and deploy/docker/api.py for the 0.9 declarative format and the base64 pdf encoding. Done means: a list of {action, params} is sent as hooks.hooks, dict hooks still use the legacy path, the server's hooks warning is raised as a UserWarning, and result.pdf is decoded to raw PDF bytes on both paths.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
crawl4ai version
0.9.5 (also on develop)
Expected Behavior
Crawl4aiDockerClient.crawl(..., hooks=...)can use the hooks the 0.9 server supports: declarative actions such as{"action": "block_resources", "params": {...}}(listed byGET /hooks/info).- With
CrawlerRunConfig(pdf=True),result.pdfholds the raw PDF bytes (b'%PDF-1.4 ...'), the same asAsyncWebCrawler.
Current Behavior
Hooks. The client only knows the pre-0.9 inline-code hooks. It always sends {"hooks": {"code": {...}, "timeout": N}}, which the 0.9 server never runs:
- Hooks disabled (the default):
RequestError: Server error 403: Inline hook code (hooks.code) was removed in 0.9.0 and cannot be enabled; it was not executed. ... - Hooks enabled (
CRAWL4AI_HOOKS_ENABLED=true): the crawl returnssuccess=True, the hook does not run, and the server's warning (results["hooks"]["warning"], or theX-Hooks-Warningheader on/crawl/stream) is dropped by the client. The user is never told.
Passing the declarative format crashes in the client before any request:
AttributeError: 'list' object has no attribute 'values'
The crawl() docstring still shows Python function hooks as the example.
PDF. The server base64-encodes pdf for JSON (deploy/docker/api.py, b64encode(result_dict['pdf']), since 0.7.2), and the client passes that string straight into CrawlResult(**r). Pydantic turns the string into bytes without decoding, so result.pdf is base64 text (b'JVBERi0xLjQK...'). A file written from it is not a valid PDF, and no error is raised. Both /crawl and /crawl/stream are affected.
Is this reproducible?
Yes
Code snippets
from crawl4ai import CrawlerRunConfig
from crawl4ai.docker_client import Crawl4aiDockerClient
async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
client._http_client.headers["Authorization"] = f"Bearer {token}"
# 0.9 declarative hooks: crashes in the client
await client.crawl(
["https://httpbin.org/headers"],
hooks=[{"action": "set_headers", "params": {"headers": {"X-Test": "1"}}}],
)
# AttributeError: 'list' object has no attribute 'values'
# Legacy function hooks on a server with hooks enabled: success, hook never runs, no warning
async def before_goto(page, **kwargs):
await page.set_extra_http_headers({"X-Test": "1"})
return page
result = await client.crawl(["https://httpbin.org/headers"], hooks={"before_goto": before_goto})
print(result.success, "X-Test" in result.html) # True False
# PDF comes back as base64 text
result = await client.crawl(["https://example.com"], crawler_config=CrawlerRunConfig(pdf=True))
print(result.pdf[:10]) # b'JVBERi0xLj' (local AsyncWebCrawler: b'%PDF-1.4\n%')
Proposed fix
In Crawl4aiDockerClient:
- send a list of
{"action", "params"}as{"hooks": {"hooks": [...], "timeout": hooks_timeout}}; - keep sending a dict the legacy way, so servers older than 0.9 still run it;
- raise the server's hooks warning as a
UserWarningon both/crawland/crawl/stream; - decode
pdfwith strict base64 before buildingCrawlResult, on both paths, leaving any non-base64 value unchanged (screenshotstays as it is: it is base64 text in local crawls too).
I'll open a PR.
OS
macOS 26.6 (platform-independent)
Python version
3.13
- Langage dominant
- Python
- Étoiles
- 87.2k
- Forks
- 9.1k
- Merge moyen
- 1 j 6 h
- PR mergées (30 j)
- 25
Préparer son environnement
- Fournit un Dockerfile ou un fichier Docker Compose
- Propose un modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackPeut-être pris @HhH666-ux l’a pris aujourd’hui. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 85/100
Les mainteneurs répondent en général sous 1 jour
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configPeut-être pris Une pull request liée à cette issue est ouverte ou déjà fusionnée. Ouverte🐞 Bug 📌 Root caused
Difficulté 2/5 1-3 heures Accessibilité débutants 70/100
unclecode/crawl4ai#2494 · 1 commentaire ·
Les mainteneurs répondent en général sous 1 jour
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresOuverte⚙ Done ✨ Enhancement
Difficulté 2/5 1-3 heures Accessibilité débutants 77/100
Les mainteneurs répondent en général sous 1 jour
-
⚙ Done 🐞 Bug 📌 Root caused
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
Les mainteneurs répondent en général sous 1 jour
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaOuverte⚙ Done 🐞 Bug
Difficulté 2/5 1-3 heures Accessibilité débutants 78/100
unclecode/crawl4ai#2469 · 1 commentaire ·
Les mainteneurs répondent en général sous 1 jour
Toutes les issues de unclecode/crawl4ai
Issues similaires
-
Difficulté 2/5 1-3 heures Accessibilité débutants 76/100
Les mainteneurs répondent en général sous 1 jour
-
documentation
Difficulté 2/5 1-3 heures Accessibilité débutants 85/100
Les mainteneurs répondent en général sous 1 jour
-
Install the skills as a Claude Code pluginPeut-être pris @ruban-24 l’a pris aujourd’hui. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 76/100
Les mainteneurs répondent en général sous 6 jours
-
Make the usage of time-zone aware UTC timestamps the norm across the pynxtools NOMAD ecosystemOuverte
Difficulté 2/5 1-3 heures Accessibilité débutants 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationOuvertedata
Difficulté 2/5 1-3 heures Accessibilité débutants 66/100