[Bug]: deep crawl configs from dump() are rejected by the server (logger: null) and DFS strategies dump empty

Ouverte
#2,450 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@Siddharth-DWT y travaille déjà.

Depuis le 27/10/2026.

  • #2451 par @Siddharth-DWT — ouverte

Évaluation

Difficulté
3/5
Temps estimé
Une demi-journée
Accessibilité débutants
48/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
backend

Piste de recherche

Start with to_serializable_dict() and the dump() path in CrawlerRunConfig, then look at DFSDeepCrawlStrategy.__init__ in the deep_crawling package, which only forwards *args, **kwargs to BFSDeepCrawlStrategy. Two fixes are needed: skip fields whose value serializes to None, and give DFS named parameters so it dumps max_depth and max_pages. Done when the snippet in the issue prints the expected params, loads with Provenance.UNTRUSTED without error, and a DFS round-trip keeps its parameters.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

⚙️ In-progress 🐞 Bug
crawl4ai version

0.9.5 (also on develop)

Expected Behavior

A deep crawl config produced by CrawlerRunConfig.dump(), as Crawl4aiDockerClient sends it, is accepted by the server when CRAWL4AI_ALLOW_DEEP_CRAWL=true.

Current Behavior

Two problems in to_serializable_dict() stop it:

  1. logger: null. Objects that cannot be deserialized (a Logger, a callable) serialize to None, but the field is still written. Every deep crawl strategy dumps as {"max_depth": ..., "logger": null}, and the server rejects logger on deep crawl strategies from an untrusted request even when it is null:
    UntrustedConfigError: field 'logger' is not permitted on BFSDeepCrawlStrategy from an untrusted request
    
  2. DFSDeepCrawlStrategy dumps with no parameters. Its __init__ only forwards *args, **kwargs to BFSDeepCrawlStrategy, so there are no named parameters to read back: DFSDeepCrawlStrategy(max_depth=3, max_pages=7) dumps as {"type": "DFSDeepCrawlStrategy", "params": {}}, and loading it fails with TypeError: ... missing 1 required positional argument: 'max_depth'. The logger error currently hides this one.
Is this reproducible?

Yes

Inputs Causing the Bug

Any CrawlerRunConfig with a deep crawl strategy, loaded with Provenance.UNTRUSTED (the server path).

Steps to Reproduce

Run the snippet below; it reproduces the server's check in-process.

Code snippets
import json
import os

os.environ["CRAWL4AI_ALLOW_DEEP_CRAWL"] = "true"
from crawl4ai import CrawlerRunConfig
from crawl4ai.async_configs import Provenance, to_serializable_dict
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy, DFSDeepCrawlStrategy

cfg = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1))
print(cfg.dump()["params"]["deep_crawl_strategy"]["params"])  # {'max_depth': 1, 'logger': None}
CrawlerRunConfig.load(json.loads(json.dumps(cfg.dump())), provenance=Provenance.UNTRUSTED)
# UntrustedConfigError: field 'logger' is not permitted on BFSDeepCrawlStrategy from an untrusted request

print(to_serializable_dict(DFSDeepCrawlStrategy(max_depth=3, max_pages=7)))
# {'type': 'DFSDeepCrawlStrategy', 'params': {}}
OS

Windows 11 (platform-independent)

Python version

3.13

Browser

Not involved

Error logs & Screenshots (if applicable)

See above.

The logger part was found by @SohamKukreti in the review of #2425; the DFS part showed up once it was fixed. I'll open a PR.

Langage dominant
Python
Étoiles
87.2k
Forks
9.1k
Merge moyen
1 j 6 h
PR mergées (30 j)
25

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de unclecode/crawl4ai

Toutes les issues de unclecode/crawl4ai

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.