[Bug]: deep crawl configs from dump() are rejected by the server (logger: null) and DFS strategies dump empty

オープン
#2,450 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@Siddharth-DWT がすでに取り組んでいます。

2026年10月27日 から。

  • #2451 @Siddharth-DWT による — オープン

評価

難易度
3/5
見積もり時間
半日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
backend

調査の方向性

Start with to_serializable_dict() and the dump() path in CrawlerRunConfig, then look at DFSDeepCrawlStrategy.__init__ in the deep_crawling package, which only forwards *args, **kwargs to BFSDeepCrawlStrategy. Two fixes are needed: skip fields whose value serializes to None, and give DFS named parameters so it dumps max_depth and max_pages. Done when the snippet in the issue prints the expected params, loads with Provenance.UNTRUSTED without error, and a DFS round-trip keeps its parameters.

索引モデルが issue の本文から書いたものです。

説明

⚙️ In-progress 🐞 Bug
crawl4ai version

0.9.5 (also on develop)

Expected Behavior

A deep crawl config produced by CrawlerRunConfig.dump(), as Crawl4aiDockerClient sends it, is accepted by the server when CRAWL4AI_ALLOW_DEEP_CRAWL=true.

Current Behavior

Two problems in to_serializable_dict() stop it:

  1. logger: null. Objects that cannot be deserialized (a Logger, a callable) serialize to None, but the field is still written. Every deep crawl strategy dumps as {"max_depth": ..., "logger": null}, and the server rejects logger on deep crawl strategies from an untrusted request even when it is null:
    UntrustedConfigError: field 'logger' is not permitted on BFSDeepCrawlStrategy from an untrusted request
    
  2. DFSDeepCrawlStrategy dumps with no parameters. Its __init__ only forwards *args, **kwargs to BFSDeepCrawlStrategy, so there are no named parameters to read back: DFSDeepCrawlStrategy(max_depth=3, max_pages=7) dumps as {"type": "DFSDeepCrawlStrategy", "params": {}}, and loading it fails with TypeError: ... missing 1 required positional argument: 'max_depth'. The logger error currently hides this one.
Is this reproducible?

Yes

Inputs Causing the Bug

Any CrawlerRunConfig with a deep crawl strategy, loaded with Provenance.UNTRUSTED (the server path).

Steps to Reproduce

Run the snippet below; it reproduces the server's check in-process.

Code snippets
import json
import os

os.environ["CRAWL4AI_ALLOW_DEEP_CRAWL"] = "true"
from crawl4ai import CrawlerRunConfig
from crawl4ai.async_configs import Provenance, to_serializable_dict
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy, DFSDeepCrawlStrategy

cfg = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1))
print(cfg.dump()["params"]["deep_crawl_strategy"]["params"])  # {'max_depth': 1, 'logger': None}
CrawlerRunConfig.load(json.loads(json.dumps(cfg.dump())), provenance=Provenance.UNTRUSTED)
# UntrustedConfigError: field 'logger' is not permitted on BFSDeepCrawlStrategy from an untrusted request

print(to_serializable_dict(DFSDeepCrawlStrategy(max_depth=3, max_pages=7)))
# {'type': 'DFSDeepCrawlStrategy', 'params': {}}
OS

Windows 11 (platform-independent)

Python version

3.13

Browser

Not involved

Error logs & Screenshots (if applicable)

See above.

The logger part was found by @SohamKukreti in the review of #2425; the DFS part showed up once it was fixed. I'll open a PR.

主要言語
Python
スター
87.2k
フォーク
9.1k
平均マージ
1日 6時間
マージ済み PR(30日)
25

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

unclecode/crawl4ai のほかの issue

unclecode/crawl4ai の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。