[Bug]: deep crawl configs from dump() are rejected by the server (logger: null) and DFS strategies dump empty

Aperta
#2,450 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Siddharth-DWT ci sta già lavorando.

Dal 27/10/2026.

  • #2451 di @Siddharth-DWT — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
Mezza giornata
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start with to_serializable_dict() and the dump() path in CrawlerRunConfig, then look at DFSDeepCrawlStrategy.__init__ in the deep_crawling package, which only forwards *args, **kwargs to BFSDeepCrawlStrategy. Two fixes are needed: skip fields whose value serializes to None, and give DFS named parameters so it dumps max_depth and max_pages. Done when the snippet in the issue prints the expected params, loads with Provenance.UNTRUSTED without error, and a DFS round-trip keeps its parameters.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

⚙️ In-progress 🐞 Bug
crawl4ai version

0.9.5 (also on develop)

Expected Behavior

A deep crawl config produced by CrawlerRunConfig.dump(), as Crawl4aiDockerClient sends it, is accepted by the server when CRAWL4AI_ALLOW_DEEP_CRAWL=true.

Current Behavior

Two problems in to_serializable_dict() stop it:

  1. logger: null. Objects that cannot be deserialized (a Logger, a callable) serialize to None, but the field is still written. Every deep crawl strategy dumps as {"max_depth": ..., "logger": null}, and the server rejects logger on deep crawl strategies from an untrusted request even when it is null:
    UntrustedConfigError: field 'logger' is not permitted on BFSDeepCrawlStrategy from an untrusted request
    
  2. DFSDeepCrawlStrategy dumps with no parameters. Its __init__ only forwards *args, **kwargs to BFSDeepCrawlStrategy, so there are no named parameters to read back: DFSDeepCrawlStrategy(max_depth=3, max_pages=7) dumps as {"type": "DFSDeepCrawlStrategy", "params": {}}, and loading it fails with TypeError: ... missing 1 required positional argument: 'max_depth'. The logger error currently hides this one.
Is this reproducible?

Yes

Inputs Causing the Bug

Any CrawlerRunConfig with a deep crawl strategy, loaded with Provenance.UNTRUSTED (the server path).

Steps to Reproduce

Run the snippet below; it reproduces the server's check in-process.

Code snippets
import json
import os

os.environ["CRAWL4AI_ALLOW_DEEP_CRAWL"] = "true"
from crawl4ai import CrawlerRunConfig
from crawl4ai.async_configs import Provenance, to_serializable_dict
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy, DFSDeepCrawlStrategy

cfg = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1))
print(cfg.dump()["params"]["deep_crawl_strategy"]["params"])  # {'max_depth': 1, 'logger': None}
CrawlerRunConfig.load(json.loads(json.dumps(cfg.dump())), provenance=Provenance.UNTRUSTED)
# UntrustedConfigError: field 'logger' is not permitted on BFSDeepCrawlStrategy from an untrusted request

print(to_serializable_dict(DFSDeepCrawlStrategy(max_depth=3, max_pages=7)))
# {'type': 'DFSDeepCrawlStrategy', 'params': {}}
OS

Windows 11 (platform-independent)

Python version

3.13

Browser

Not involved

Error logs & Screenshots (if applicable)

See above.

The logger part was found by @SohamKukreti in the review of #2425; the DFS part showed up once it was fixed. I'll open a PR.

Lingua principale
Python
Stelle
87.2k
Fork
9.1k
Merge medio
1g 6h
PR unite (30g)
25

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.