[Bug]: CrawlerRunConfig.dump()/load() drops deep-crawl filter and scorer arguments, so a DomainFilter allows every domain
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 35/100
Línea de trabajo
Start by tracing CrawlerRunConfig.dump() and load() through to_serializable_dict(), then inspect the deep-crawl filter and scorer classes named in the issue. The report includes a Python reproduction showing a DomainFilter loses its allowed domains and scorer arguments disappear or fail during loading. Done when round trips preserve each listed filter and scorer argument and the reproduction retains the domain restriction; the author says a fix and tests are already in progress.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
crawl4ai version
0.9.5 (also on main and develop)
Expected Behavior
CrawlerRunConfig.load(cfg.dump()) gives back a config that crawls the same way, as #1436 made true for URLPatternFilter.
Current Behavior
to_serializable_dict() rebuilds an object from attributes named after its constructor arguments. Several deep-crawl filters and scorers keep their arguments only in private slots (_allowed_domains, _keywords, ...), so the arguments are dropped:
| Class | Lost | Result after load() |
|---|---|---|
DomainFilter |
allowed_domains, blocked_domains |
allows every domain |
ContentTypeFilter |
check_extension=False |
back to True |
PathDepthScorer |
optimal_depth |
back to 3 |
FreshnessScorer |
current_year |
back to the default |
KeywordRelevanceScorer |
keywords, case_sensitive |
TypeError on load |
DomainAuthorityScorer |
domain_weights, default_weight |
TypeError on load |
CompositeScorer |
scorers, normalize |
TypeError on load |
Crawl4aiDockerClient sends crawler_config.dump() to the server, so a deep crawl run through the Docker client with a DomainFilter silently leaves the allowed domain.
Is this reproducible?
Yes
Inputs Causing the Bug
A CrawlerRunConfig whose deep_crawl_strategy uses DomainFilter or any of the scorers above.
Steps to Reproduce
Run the snippet below.
Code snippets
import asyncio
import json
from crawl4ai import CrawlerRunConfig
from crawl4ai.deep_crawling import (
BestFirstCrawlingStrategy, BFSDeepCrawlStrategy, DomainFilter, FilterChain, KeywordRelevanceScorer,
)
cfg = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(
max_depth=2, filter_chain=FilterChain([DomainFilter(allowed_domains=["docs.example.com"])])))
back = CrawlerRunConfig.load(json.loads(json.dumps(cfg.dump())))
url = "https://evil.example.net/page"
print(asyncio.run(cfg.deep_crawl_strategy.filter_chain.apply(url))) # False
print(asyncio.run(back.deep_crawl_strategy.filter_chain.apply(url))) # True, expected False
cfg2 = CrawlerRunConfig(deep_crawl_strategy=BestFirstCrawlingStrategy(
max_depth=2, url_scorer=KeywordRelevanceScorer(keywords=["api", "guide"], weight=0.7)))
print(cfg2.dump()["params"]["deep_crawl_strategy"]["params"]["url_scorer"])
# {'type': 'KeywordRelevanceScorer', 'params': {'weight': 0.699999988079071}} <- keywords missing
CrawlerRunConfig.load(json.loads(json.dumps(cfg2.dump())))
# TypeError: KeywordRelevanceScorer.__init__() missing 1 required positional argument: 'keywords'
OS
Windows 11 (the code path is platform-independent)
Python version
3.13
Browser
Not involved
Error logs & Screenshots (if applicable)
The TypeError above for KeywordRelevanceScorer, DomainAuthorityScorer and CompositeScorer; no error for the others.
I have a fix that follows the #1436 approach (read-only properties exposing the original arguments, so __slots__ and memory use don't change), with a round-trip test for every filter and scorer. I'll open a PR against develop referencing this issue.
Investigated with AI assistance; I ran the reproduction and checked the output myself.
- Lenguaje dominante
- Python
- Estrellas
- 87.2k
- Forks
- 9.1k
- Merge medio
- 1 d 6 h
- PR fusionados (30 d)
- 25
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackPosiblemente ocupada @HhH666-ux la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
unclecode/crawl4ai#2494 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAbierto⚙ Done ✨ Enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 1 día
-
⚙ Done 🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAbierto⚙ Done 🐞 Bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
unclecode/crawl4ai#2469 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de unclecode/crawl4ai
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 1 día
-
documentation
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
Install the skills as a Claude Code pluginPosiblemente ocupada @ruban-24 la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 6 días
-
Make the usage of time-zone aware UTC timestamps the norm across the pynxtools NOMAD ecosystemAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationAbiertodata
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100