[Bug]: Default user agent is a stale Chrome/116 string that contradicts the bundled Chromium

オープン
#2,493 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@rtmalikian がすでに取り組んでいます。

2026年11月4日 から。

  • #2495 @rtmalikian による — オープン

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
docker, python
領域
web-dev

調査の方向性

Start in crawl4ai/async_configs.py, where BrowserConfig sets the default user_agent and, in __init__, calls self.headers.setdefault("sec-ch-ua", self.browser_hint). Then check the fallback in user_agent_generator.py and generate_client_hints. Done means an empty user_agent no longer produces an empty sec-ch-ua header, and the default User-Agent is consistent with the bundled Chromium. Maintainers should first settle whether the default is hard-coded or read from the launched browser.

索引モデルが issue の本文から書いたものです。

説明

⚙ Done 🐞 Bug 📌 Root caused
crawl4ai version

0.9.5 (Docker image unclecode/crawl4ai:0.9.5); the same BrowserConfig code is on develop.

Expected Behavior

The default BrowserConfig.user_agent matches the browser crawl4ai launches.
An empty user_agent (to keep the browser's own User-Agent) sends no forged client hints.

Current Behavior
  1. Stale, non-standard default. BrowserConfig.user_agent defaults to
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
    (crawl4ai/async_configs.py, also the fallback of user_agent_generator.py).
    It lacks the (KHTML, like Gecko) token Chrome sends, and announces Chrome 116, while the 0.9.5 image runs Playwright 1.64 with Chromium 156.0.8078.4.
    The generated sec-ch-ua says 116 too, whereas the other request headers (Accept-Encoding: … zstd, Priority: u=0, i) come from Chromium 156.
    The BrowserConfig docstring documents yet another default, a well-formed macOS User-Agent.
    Since Playwright is not pinned (playwright>=1.49.0), the bundled Chromium changes with each image build, so a hard-coded string keeps drifting.
  2. Empty sec-ch-ua with an empty User-Agent. BrowserConfig(user_agent="") keeps the browser's native User-Agent, but __init__ still runs self.headers.setdefault("sec-ch-ua", self.browser_hint), and generate_client_hints("") returns '': every request then carries an empty sec-ch-ua header.
  3. A worker keeps the real User-Agent. With the default config, CreepJS reports in its ServiceWorkerGlobalScope worker the User-Agent … HeadlessChrome/156.0.0.0 …, while the page says Chrome/116.
    Passing the same string as a launch argument (--user-agent=… in extra_args) makes the worker report it too.
    (Observed on the 0.9.5 image with #2448 backported; the User-Agent code is the same without it.)

Related: #2472 / #2473 touch the same auto-added sec-ch-ua, but #2473 only leaves it out of dump(); __init__ is unchanged.

Is this reproducible?

Yes

Inputs Causing the Bug
- URL(s): https://httpbin.org/headers (headers sent), https://abrahamjuliot.github.io/creepjs/ (worker User-Agent)
- Settings: default BrowserConfig; Docker server /md endpoint
Steps to Reproduce
docker run -d --name c4ai --shm-size=1g -p 11235:11235 -e CRAWL4AI_API_TOKEN=test unclecode/crawl4ai:0.9.5
curl -s -X POST http://localhost:11235/md -H "Authorization: Bearer test" \
  -H "Content-Type: application/json" -d '{"url": "https://httpbin.org/headers", "f": "raw"}'
# "User-Agent" and "Sec-Ch-Ua" announce Chrome 116; the bundled Chromium is 156:
docker exec c4ai sh -c 'grep -A3 "\"chromium\"" /usr/local/lib/python3.12/site-packages/playwright/driver/package/browsers.json'
Code snippets
from crawl4ai import BrowserConfig

print(BrowserConfig().user_agent)
# Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
print(BrowserConfig(user_agent="").headers)
# {'sec-ch-ua': ''}
Suggested fix
  • Default to the launched browser's own User-Agent (with HeadlessChrome replaced by Chrome in headless mode) rather than a hard-coded string; at least add (KHTML, like Gecko) and keep the version in step with the bundled Chromium.
  • Skip the sec-ch-ua default when user_agent is empty.
  • Also pass the User-Agent as --user-agent at launch, so workers report the same one.
OS

Linux (Docker on WSL2)

Python version

3.12 (image)

Browser

Chromium bundled with Playwright 1.64 (the stock 0.9.5 image launches chrome-headless-shell, see #2382)

Browser version

156.0.8078.4

主要言語
Python
スター
87.2k
フォーク
9.1k
平均マージ
1日 6時間
マージ済み PR(30日)
25

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

unclecode/crawl4ai のほかの issue

unclecode/crawl4ai の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。