[Bug]: Crawl4aiDockerClient cannot send 0.9 declarative hooks, and returns result.pdf as base64 text

オープン
#2,483 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
70/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
docker, python
領域
api

調査の方向性

Start in the Crawl4aiDockerClient class, in its crawl() method, where hooks are currently wrapped as {"hooks": {"code": ..., "timeout": N}} and where CrawlResult(**r) is built on both /crawl and /crawl/stream. Check GET /hooks/info and deploy/docker/api.py for the 0.9 declarative format and the base64 pdf encoding. Done means: a list of {action, params} is sent as hooks.hooks, dict hooks still use the legacy path, the server's hooks warning is raised as a UserWarning, and result.pdf is decoded to raw PDF bytes on both paths.

索引モデルが issue の本文から書いたものです。

説明

⚙ Done 🐞 Bug
crawl4ai version

0.9.5 (also on develop)

Expected Behavior
  1. Crawl4aiDockerClient.crawl(..., hooks=...) can use the hooks the 0.9 server supports: declarative actions such as {"action": "block_resources", "params": {...}} (listed by GET /hooks/info).
  2. With CrawlerRunConfig(pdf=True), result.pdf holds the raw PDF bytes (b'%PDF-1.4 ...'), the same as AsyncWebCrawler.
Current Behavior

Hooks. The client only knows the pre-0.9 inline-code hooks. It always sends {"hooks": {"code": {...}, "timeout": N}}, which the 0.9 server never runs:

  • Hooks disabled (the default): RequestError: Server error 403: Inline hook code (hooks.code) was removed in 0.9.0 and cannot be enabled; it was not executed. ...
  • Hooks enabled (CRAWL4AI_HOOKS_ENABLED=true): the crawl returns success=True, the hook does not run, and the server's warning (results["hooks"]["warning"], or the X-Hooks-Warning header on /crawl/stream) is dropped by the client. The user is never told.

Passing the declarative format crashes in the client before any request:

AttributeError: 'list' object has no attribute 'values'

The crawl() docstring still shows Python function hooks as the example.

PDF. The server base64-encodes pdf for JSON (deploy/docker/api.py, b64encode(result_dict['pdf']), since 0.7.2), and the client passes that string straight into CrawlResult(**r). Pydantic turns the string into bytes without decoding, so result.pdf is base64 text (b'JVBERi0xLjQK...'). A file written from it is not a valid PDF, and no error is raised. Both /crawl and /crawl/stream are affected.

Is this reproducible?

Yes

Code snippets
from crawl4ai import CrawlerRunConfig
from crawl4ai.docker_client import Crawl4aiDockerClient

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    client._http_client.headers["Authorization"] = f"Bearer {token}"

    # 0.9 declarative hooks: crashes in the client
    await client.crawl(
        ["https://httpbin.org/headers"],
        hooks=[{"action": "set_headers", "params": {"headers": {"X-Test": "1"}}}],
    )
    # AttributeError: 'list' object has no attribute 'values'

    # Legacy function hooks on a server with hooks enabled: success, hook never runs, no warning
    async def before_goto(page, **kwargs):
        await page.set_extra_http_headers({"X-Test": "1"})
        return page

    result = await client.crawl(["https://httpbin.org/headers"], hooks={"before_goto": before_goto})
    print(result.success, "X-Test" in result.html)  # True False

    # PDF comes back as base64 text
    result = await client.crawl(["https://example.com"], crawler_config=CrawlerRunConfig(pdf=True))
    print(result.pdf[:10])  # b'JVBERi0xLj'  (local AsyncWebCrawler: b'%PDF-1.4\n%')
Proposed fix

In Crawl4aiDockerClient:

  • send a list of {"action", "params"} as {"hooks": {"hooks": [...], "timeout": hooks_timeout}};
  • keep sending a dict the legacy way, so servers older than 0.9 still run it;
  • raise the server's hooks warning as a UserWarning on both /crawl and /crawl/stream;
  • decode pdf with strict base64 before building CrawlResult, on both paths, leaving any non-base64 value unchanged (screenshot stays as it is: it is base64 text in local crawls too).

I'll open a PR.

OS

macOS 26.6 (platform-independent)

Python version

3.13

主要言語
Python
スター
87.2k
フォーク
9.1k
平均マージ
1日 6時間
マージ済み PR(30日)
25

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

unclecode/crawl4ai のほかの issue

unclecode/crawl4ai の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。