[Bug]: CrawlerRunConfig.dump()/load() drops deep-crawl filter and scorer arguments, so a DomainFilter allows every domain

Abierto
#2,424 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@Siddharth-DWT ya está trabajando en esto.

Desde el 21/10/2026.

  • #2425 de @Siddharth-DWT — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend

Línea de trabajo

Start by tracing CrawlerRunConfig.dump() and load() through to_serializable_dict(), then inspect the deep-crawl filter and scorer classes named in the issue. The report includes a Python reproduction showing a DomainFilter loses its allowed domains and scorer arguments disappear or fail during loading. Done when round trips preserve each listed filter and scorer argument and the reproduction retains the domain restriction; the author says a fix and tests are already in progress.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

⚙️ In-progress 📌 Root caused
crawl4ai version

0.9.5 (also on main and develop)

Expected Behavior

CrawlerRunConfig.load(cfg.dump()) gives back a config that crawls the same way, as #1436 made true for URLPatternFilter.

Current Behavior

to_serializable_dict() rebuilds an object from attributes named after its constructor arguments. Several deep-crawl filters and scorers keep their arguments only in private slots (_allowed_domains, _keywords, ...), so the arguments are dropped:

Class Lost Result after load()
DomainFilter allowed_domains, blocked_domains allows every domain
ContentTypeFilter check_extension=False back to True
PathDepthScorer optimal_depth back to 3
FreshnessScorer current_year back to the default
KeywordRelevanceScorer keywords, case_sensitive TypeError on load
DomainAuthorityScorer domain_weights, default_weight TypeError on load
CompositeScorer scorers, normalize TypeError on load

Crawl4aiDockerClient sends crawler_config.dump() to the server, so a deep crawl run through the Docker client with a DomainFilter silently leaves the allowed domain.

Is this reproducible?

Yes

Inputs Causing the Bug

A CrawlerRunConfig whose deep_crawl_strategy uses DomainFilter or any of the scorers above.

Steps to Reproduce

Run the snippet below.

Code snippets
import asyncio
import json

from crawl4ai import CrawlerRunConfig
from crawl4ai.deep_crawling import (
    BestFirstCrawlingStrategy, BFSDeepCrawlStrategy, DomainFilter, FilterChain, KeywordRelevanceScorer,
)

cfg = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(
    max_depth=2, filter_chain=FilterChain([DomainFilter(allowed_domains=["docs.example.com"])])))
back = CrawlerRunConfig.load(json.loads(json.dumps(cfg.dump())))
url = "https://evil.example.net/page"
print(asyncio.run(cfg.deep_crawl_strategy.filter_chain.apply(url)))   # False
print(asyncio.run(back.deep_crawl_strategy.filter_chain.apply(url)))  # True, expected False

cfg2 = CrawlerRunConfig(deep_crawl_strategy=BestFirstCrawlingStrategy(
    max_depth=2, url_scorer=KeywordRelevanceScorer(keywords=["api", "guide"], weight=0.7)))
print(cfg2.dump()["params"]["deep_crawl_strategy"]["params"]["url_scorer"])
# {'type': 'KeywordRelevanceScorer', 'params': {'weight': 0.699999988079071}}  <- keywords missing
CrawlerRunConfig.load(json.loads(json.dumps(cfg2.dump())))
# TypeError: KeywordRelevanceScorer.__init__() missing 1 required positional argument: 'keywords'
OS

Windows 11 (the code path is platform-independent)

Python version

3.13

Browser

Not involved

Error logs & Screenshots (if applicable)

The TypeError above for KeywordRelevanceScorer, DomainAuthorityScorer and CompositeScorer; no error for the others.

I have a fix that follows the #1436 approach (read-only properties exposing the original arguments, so __slots__ and memory use don't change), with a round-trip test for every filter and scorer. I'll open a PR against develop referencing this issue.

Investigated with AI assistance; I ran the reproduction and checked the output myself.

Lenguaje dominante
Python
Estrellas
87.2k
Forks
9.1k
Merge medio
1 d 6 h
PR fusionados (30 d)
25

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.