[Bug]: crawler.base_config boolean values are silently ignored (regression from #1505)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start in api.py around CrawlerRunConfig.load() at line 675 and effective_config handling at lines 697-716; compare the normal and config-list paths, including the guard at line 715. Use the raw request dictionary to distinguish omitted fields from explicitly sent defaults. Done means base_config boolean and numeric defaults apply when omitted while explicit client values remain unchanged.
由索引模型根据 Issue 内容生成。
描述
crawl4ai version
0.9.2
Expected Behavior
config.yml sets a server-side default for every crawl:
# deploy/docker/config.yml:74-76
crawler:
base_config:
simulate_user: true
A POST /crawl request that does not send simulate_user should run with simulate_user=True, i.e. the server default applies.
A request that does send simulate_user should win over the server default — that is the intent of #1505.
Current Behavior
The server default never applies. simulate_user is False on every crawl.
api.py:715 reads "the client didn't send this field" as "the attribute is None or """:
current_value = getattr(crawler_config, key)
if current_value is None or current_value == "": # api.py:715
setattr(crawler_config, key, value)
CrawlerRunConfig.simulate_user defaults to False (async_configs.py:1650), and False is neither None nor "", so the guard never passes.
The same goes for every base_config key defaulting to a boolean or a number: magic, override_navigator, check_robots_txt, remove_overlay_elements, page_timeout.
The config-list path at api.py:707 (8995c1b, #1837) copies the guard.
So the stock image ships simulate_user: true (config.yml:74-76, utils.py:63)
but never injects the navigator_overrider script (browser_manager.py:1229-1235) or runs the mouse-move simulation (async_crawler_strategy.py:980-983).
a1950af (#1505) introduced this.
The setattr used to be unconditional and clobbered client-sent values, so reverting brings #1505 back.
After CrawlerRunConfig.load() (api.py:675) nothing tells "omitted" apart from "sent, equal to the default".
That information only exists in the raw request dict
Is this reproducible?
Yes
Inputs Causing the Bug
- URL(s): any, e.g. https://example.com
- Settings used: stock deploy/docker/config.yml, i.e. crawler.base_config.simulate_user: true
- Input data: {"urls": ["https://example.com"]} # no crawler_config key
Steps to Reproduce
1. Start the stock server image, config.yml untouched.
2. POST the body above to /crawl.
3. Read effective_config in handle_crawl_request (api.py:697-716).
simulate_user is False.
Code snippets
# The guard in isolation. No server or browser needed.
from crawl4ai import CrawlerRunConfig
cfg = CrawlerRunConfig() # client sent no crawler_config
value = getattr(cfg, "simulate_user") # False, the dataclass default
assert value is None or value == "" # api.py:715 -> fails, setattr skipped
OS
Linux (Docker image, python:3.12-slim-bookworm)
Python version
3.12
Browser
Chromium (Playwright, headless)
Browser version
No response
Error logs & Screenshots (if applicable)
No error. The server drops the value silently.
- 主要语言
- Python
- 星标
- 84.5k
- 派生
- 8.7k
- 平均合并
- 3 天 9 小时
- 30 天内合并 PR
- 17
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
unclecode/crawl4ai 的其他 Issue
-
[Bug]: Reusing BFSDeepCrawlStrategy leaks the previous crawl's max_pages budget into a fresh run未关闭
难度 2/5 1-3 小时 新手友好度 78/100
unclecode/crawl4ai#2309 · 2 条评论 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 84/100
unclecode/crawl4ai#2147 · 3 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
unclecode/crawl4ai#2123 · 1 条评论 ·
维护者通常 1 天内回复
-
🐞 Bug 🩺 Needs Triage
难度 4/5 3-5 天 新手友好度 55/100
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 35/100
维护者通常 1 天内回复
查看 unclecode/crawl4ai 的全部 Issue
相似的 Issue
-
namespace operations
难度 1/5 1 小时以内 新手友好度 82/100
EclipseFdn/open-vsx.org#13573 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
collective/icalendar#1854 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
rancher/rancher-ai-agent#412 ·
维护者通常 6 天内回复
-
难度 2/5 1-3 小时 新手友好度 84/100
TUDelftGeodesy/DePSI#134 ·
-
难度 2/5 1-3 小时 新手友好度 88/100
HenriquesLab/rxiv-maker#335 ·