[Bug]: CrawlerRunConfig.dump()/load() drops deep-crawl filter and scorer arguments, so a DomainFilter allows every domain

Ouverte
#2,424 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@Siddharth-DWT y travaille déjà.

Depuis le 21/10/2026.

  • #2425 par @Siddharth-DWT — ouverte

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
35/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
backend

Piste de recherche

Start by tracing CrawlerRunConfig.dump() and load() through to_serializable_dict(), then inspect the deep-crawl filter and scorer classes named in the issue. The report includes a Python reproduction showing a DomainFilter loses its allowed domains and scorer arguments disappear or fail during loading. Done when round trips preserve each listed filter and scorer argument and the reproduction retains the domain restriction; the author says a fix and tests are already in progress.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

⚙️ In-progress 📌 Root caused
crawl4ai version

0.9.5 (also on main and develop)

Expected Behavior

CrawlerRunConfig.load(cfg.dump()) gives back a config that crawls the same way, as #1436 made true for URLPatternFilter.

Current Behavior

to_serializable_dict() rebuilds an object from attributes named after its constructor arguments. Several deep-crawl filters and scorers keep their arguments only in private slots (_allowed_domains, _keywords, ...), so the arguments are dropped:

Class Lost Result after load()
DomainFilter allowed_domains, blocked_domains allows every domain
ContentTypeFilter check_extension=False back to True
PathDepthScorer optimal_depth back to 3
FreshnessScorer current_year back to the default
KeywordRelevanceScorer keywords, case_sensitive TypeError on load
DomainAuthorityScorer domain_weights, default_weight TypeError on load
CompositeScorer scorers, normalize TypeError on load

Crawl4aiDockerClient sends crawler_config.dump() to the server, so a deep crawl run through the Docker client with a DomainFilter silently leaves the allowed domain.

Is this reproducible?

Yes

Inputs Causing the Bug

A CrawlerRunConfig whose deep_crawl_strategy uses DomainFilter or any of the scorers above.

Steps to Reproduce

Run the snippet below.

Code snippets
import asyncio
import json

from crawl4ai import CrawlerRunConfig
from crawl4ai.deep_crawling import (
    BestFirstCrawlingStrategy, BFSDeepCrawlStrategy, DomainFilter, FilterChain, KeywordRelevanceScorer,
)

cfg = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(
    max_depth=2, filter_chain=FilterChain([DomainFilter(allowed_domains=["docs.example.com"])])))
back = CrawlerRunConfig.load(json.loads(json.dumps(cfg.dump())))
url = "https://evil.example.net/page"
print(asyncio.run(cfg.deep_crawl_strategy.filter_chain.apply(url)))   # False
print(asyncio.run(back.deep_crawl_strategy.filter_chain.apply(url)))  # True, expected False

cfg2 = CrawlerRunConfig(deep_crawl_strategy=BestFirstCrawlingStrategy(
    max_depth=2, url_scorer=KeywordRelevanceScorer(keywords=["api", "guide"], weight=0.7)))
print(cfg2.dump()["params"]["deep_crawl_strategy"]["params"]["url_scorer"])
# {'type': 'KeywordRelevanceScorer', 'params': {'weight': 0.699999988079071}}  <- keywords missing
CrawlerRunConfig.load(json.loads(json.dumps(cfg2.dump())))
# TypeError: KeywordRelevanceScorer.__init__() missing 1 required positional argument: 'keywords'
OS

Windows 11 (the code path is platform-independent)

Python version

3.13

Browser

Not involved

Error logs & Screenshots (if applicable)

The TypeError above for KeywordRelevanceScorer, DomainAuthorityScorer and CompositeScorer; no error for the others.

I have a fix that follows the #1436 approach (read-only properties exposing the original arguments, so __slots__ and memory use don't change), with a round-trip test for every filter and scorer. I'll open a PR against develop referencing this issue.

Investigated with AI assistance; I ran the reproduction and checked the output myself.

Langage dominant
Python
Étoiles
87.2k
Forks
9.1k
Merge moyen
1 j 6 h
PR mergées (30 j)
25

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de unclecode/crawl4ai

Toutes les issues de unclecode/crawl4ai

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.