[Bug]: Default user agent is a stale Chrome/116 string that contradicts the bundled Chromium

未关闭
#2,493 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@rtmalikian 已经在做这个了。

开始于 2026年11月4日。

  • #2495 来自 @rtmalikian —— 未关闭

评估

难度
3/5
预计耗时
1-2 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
docker, python
领域
web-dev

调研方向

Start in crawl4ai/async_configs.py, where BrowserConfig sets the default user_agent and, in __init__, calls self.headers.setdefault("sec-ch-ua", self.browser_hint). Then check the fallback in user_agent_generator.py and generate_client_hints. Done means an empty user_agent no longer produces an empty sec-ch-ua header, and the default User-Agent is consistent with the bundled Chromium. Maintainers should first settle whether the default is hard-coded or read from the launched browser.

由索引模型根据 Issue 内容生成。

描述

⚙ Done 🐞 Bug 📌 Root caused
crawl4ai version

0.9.5 (Docker image unclecode/crawl4ai:0.9.5); the same BrowserConfig code is on develop.

Expected Behavior

The default BrowserConfig.user_agent matches the browser crawl4ai launches.
An empty user_agent (to keep the browser's own User-Agent) sends no forged client hints.

Current Behavior
  1. Stale, non-standard default. BrowserConfig.user_agent defaults to
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
    (crawl4ai/async_configs.py, also the fallback of user_agent_generator.py).
    It lacks the (KHTML, like Gecko) token Chrome sends, and announces Chrome 116, while the 0.9.5 image runs Playwright 1.64 with Chromium 156.0.8078.4.
    The generated sec-ch-ua says 116 too, whereas the other request headers (Accept-Encoding: … zstd, Priority: u=0, i) come from Chromium 156.
    The BrowserConfig docstring documents yet another default, a well-formed macOS User-Agent.
    Since Playwright is not pinned (playwright>=1.49.0), the bundled Chromium changes with each image build, so a hard-coded string keeps drifting.
  2. Empty sec-ch-ua with an empty User-Agent. BrowserConfig(user_agent="") keeps the browser's native User-Agent, but __init__ still runs self.headers.setdefault("sec-ch-ua", self.browser_hint), and generate_client_hints("") returns '': every request then carries an empty sec-ch-ua header.
  3. A worker keeps the real User-Agent. With the default config, CreepJS reports in its ServiceWorkerGlobalScope worker the User-Agent … HeadlessChrome/156.0.0.0 …, while the page says Chrome/116.
    Passing the same string as a launch argument (--user-agent=… in extra_args) makes the worker report it too.
    (Observed on the 0.9.5 image with #2448 backported; the User-Agent code is the same without it.)

Related: #2472 / #2473 touch the same auto-added sec-ch-ua, but #2473 only leaves it out of dump(); __init__ is unchanged.

Is this reproducible?

Yes

Inputs Causing the Bug
- URL(s): https://httpbin.org/headers (headers sent), https://abrahamjuliot.github.io/creepjs/ (worker User-Agent)
- Settings: default BrowserConfig; Docker server /md endpoint
Steps to Reproduce
docker run -d --name c4ai --shm-size=1g -p 11235:11235 -e CRAWL4AI_API_TOKEN=test unclecode/crawl4ai:0.9.5
curl -s -X POST http://localhost:11235/md -H "Authorization: Bearer test" \
  -H "Content-Type: application/json" -d '{"url": "https://httpbin.org/headers", "f": "raw"}'
# "User-Agent" and "Sec-Ch-Ua" announce Chrome 116; the bundled Chromium is 156:
docker exec c4ai sh -c 'grep -A3 "\"chromium\"" /usr/local/lib/python3.12/site-packages/playwright/driver/package/browsers.json'
Code snippets
from crawl4ai import BrowserConfig

print(BrowserConfig().user_agent)
# Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
print(BrowserConfig(user_agent="").headers)
# {'sec-ch-ua': ''}
Suggested fix
  • Default to the launched browser's own User-Agent (with HeadlessChrome replaced by Chrome in headless mode) rather than a hard-coded string; at least add (KHTML, like Gecko) and keep the version in step with the bundled Chromium.
  • Skip the sec-ch-ua default when user_agent is empty.
  • Also pass the User-Agent as --user-agent at launch, so workers report the same one.
OS

Linux (Docker on WSL2)

Python version

3.12 (image)

Browser

Chromium bundled with Playwright 1.64 (the stock 0.9.5 image launches chrome-headless-shell, see #2382)

Browser version

156.0.8078.4

主要语言
Python
星标
87.2k
派生
9.1k
平均合并
1 天 6 小时
30 天内合并 PR
25

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。