[Bug]: Default user agent is a stale Chrome/116 string that contradicts the bundled Chromium
维护者通常 1 天内回复
评估
调研方向
Start in crawl4ai/async_configs.py, where BrowserConfig sets the default user_agent and, in __init__, calls self.headers.setdefault("sec-ch-ua", self.browser_hint). Then check the fallback in user_agent_generator.py and generate_client_hints. Done means an empty user_agent no longer produces an empty sec-ch-ua header, and the default User-Agent is consistent with the bundled Chromium. Maintainers should first settle whether the default is hard-coded or read from the launched browser.
由索引模型根据 Issue 内容生成。
描述
crawl4ai version
0.9.5 (Docker image unclecode/crawl4ai:0.9.5); the same BrowserConfig code is on develop.
Expected Behavior
The default BrowserConfig.user_agent matches the browser crawl4ai launches.
An empty user_agent (to keep the browser's own User-Agent) sends no forged client hints.
Current Behavior
- Stale, non-standard default.
BrowserConfig.user_agentdefaults to
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
(crawl4ai/async_configs.py, also thefallbackofuser_agent_generator.py).
It lacks the(KHTML, like Gecko)token Chrome sends, and announces Chrome 116, while the 0.9.5 image runs Playwright 1.64 with Chromium 156.0.8078.4.
The generatedsec-ch-uasays 116 too, whereas the other request headers (Accept-Encoding: … zstd,Priority: u=0, i) come from Chromium 156.
TheBrowserConfigdocstring documents yet another default, a well-formed macOS User-Agent.
Since Playwright is not pinned (playwright>=1.49.0), the bundled Chromium changes with each image build, so a hard-coded string keeps drifting. - Empty
sec-ch-uawith an empty User-Agent.BrowserConfig(user_agent="")keeps the browser's native User-Agent, but__init__still runsself.headers.setdefault("sec-ch-ua", self.browser_hint), andgenerate_client_hints("")returns'': every request then carries an emptysec-ch-uaheader. - A worker keeps the real User-Agent. With the default config, CreepJS reports in its
ServiceWorkerGlobalScopeworker the User-Agent… HeadlessChrome/156.0.0.0 …, while the page saysChrome/116.
Passing the same string as a launch argument (--user-agent=…inextra_args) makes the worker report it too.
(Observed on the 0.9.5 image with #2448 backported; the User-Agent code is the same without it.)
Related: #2472 / #2473 touch the same auto-added sec-ch-ua, but #2473 only leaves it out of dump(); __init__ is unchanged.
Is this reproducible?
Yes
Inputs Causing the Bug
- URL(s): https://httpbin.org/headers (headers sent), https://abrahamjuliot.github.io/creepjs/ (worker User-Agent)
- Settings: default BrowserConfig; Docker server /md endpoint
Steps to Reproduce
docker run -d --name c4ai --shm-size=1g -p 11235:11235 -e CRAWL4AI_API_TOKEN=test unclecode/crawl4ai:0.9.5
curl -s -X POST http://localhost:11235/md -H "Authorization: Bearer test" \
-H "Content-Type: application/json" -d '{"url": "https://httpbin.org/headers", "f": "raw"}'
# "User-Agent" and "Sec-Ch-Ua" announce Chrome 116; the bundled Chromium is 156:
docker exec c4ai sh -c 'grep -A3 "\"chromium\"" /usr/local/lib/python3.12/site-packages/playwright/driver/package/browsers.json'
Code snippets
from crawl4ai import BrowserConfig
print(BrowserConfig().user_agent)
# Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
print(BrowserConfig(user_agent="").headers)
# {'sec-ch-ua': ''}
Suggested fix
- Default to the launched browser's own User-Agent (with
HeadlessChromereplaced byChromein headless mode) rather than a hard-coded string; at least add(KHTML, like Gecko)and keep the version in step with the bundled Chromium. - Skip the
sec-ch-uadefault whenuser_agentis empty. - Also pass the User-Agent as
--user-agentat launch, so workers report the same one.
OS
Linux (Docker on WSL2)
Python version
3.12 (image)
Browser
Chromium bundled with Playwright 1.64 (the stock 0.9.5 image launches chrome-headless-shell, see #2382)
Browser version
156.0.8078.4
- 主要语言
- Python
- 星标
- 87.2k
- 派生
- 9.1k
- 平均合并
- 1 天 6 小时
- 30 天内合并 PR
- 25
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
unclecode/crawl4ai 的其他 Issue
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stack可能已有人在做 @HhH666-ux 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.config可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 70/100
unclecode/crawl4ai#2494 · 1 条评论 ·
维护者通常 1 天内回复
-
⚙ Done ✨ Enhancement
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 1 天内回复
-
⚙ Done 🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.media未关闭⚙ Done 🐞 Bug
难度 2/5 1-3 小时 新手友好度 78/100
unclecode/crawl4ai#2469 · 1 条评论 ·
维护者通常 1 天内回复
查看 unclecode/crawl4ai 的全部 Issue
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 78/100
-
Needs Triage
难度 2/5 1-3 小时 新手友好度 80/100
pandas-dev/pandas#72163 ·
维护者通常 2 天内回复
-
LightFM prepare_test_df ignores mapping values and silently mislabels evaluation pairs可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 2/5 1-3 小时 新手友好度 79/100
recommenders-team/recommenders#2480 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 85/100
521xueweihan/HelloGitHub#4385 ·
-
🐛 bug
难度 2/5 1-3 小时 新手友好度 72/100
pdm-project/pdm#4012 ·
维护者通常 1 天内回复