[Bug]: Default user agent is a stale Chrome/116 string that contradicts the bundled Chromium

Aperta
#2,493 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@rtmalikian ci sta già lavorando.

Dal 4/11/2026.

  • #2495 di @rtmalikian — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
docker, python
Ambito
web-dev

Direzione di ricerca

Start in crawl4ai/async_configs.py, where BrowserConfig sets the default user_agent and, in __init__, calls self.headers.setdefault("sec-ch-ua", self.browser_hint). Then check the fallback in user_agent_generator.py and generate_client_hints. Done means an empty user_agent no longer produces an empty sec-ch-ua header, and the default User-Agent is consistent with the bundled Chromium. Maintainers should first settle whether the default is hard-coded or read from the launched browser.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

⚙ Done 🐞 Bug 📌 Root caused
crawl4ai version

0.9.5 (Docker image unclecode/crawl4ai:0.9.5); the same BrowserConfig code is on develop.

Expected Behavior

The default BrowserConfig.user_agent matches the browser crawl4ai launches.
An empty user_agent (to keep the browser's own User-Agent) sends no forged client hints.

Current Behavior
  1. Stale, non-standard default. BrowserConfig.user_agent defaults to
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
    (crawl4ai/async_configs.py, also the fallback of user_agent_generator.py).
    It lacks the (KHTML, like Gecko) token Chrome sends, and announces Chrome 116, while the 0.9.5 image runs Playwright 1.64 with Chromium 156.0.8078.4.
    The generated sec-ch-ua says 116 too, whereas the other request headers (Accept-Encoding: … zstd, Priority: u=0, i) come from Chromium 156.
    The BrowserConfig docstring documents yet another default, a well-formed macOS User-Agent.
    Since Playwright is not pinned (playwright>=1.49.0), the bundled Chromium changes with each image build, so a hard-coded string keeps drifting.
  2. Empty sec-ch-ua with an empty User-Agent. BrowserConfig(user_agent="") keeps the browser's native User-Agent, but __init__ still runs self.headers.setdefault("sec-ch-ua", self.browser_hint), and generate_client_hints("") returns '': every request then carries an empty sec-ch-ua header.
  3. A worker keeps the real User-Agent. With the default config, CreepJS reports in its ServiceWorkerGlobalScope worker the User-Agent … HeadlessChrome/156.0.0.0 …, while the page says Chrome/116.
    Passing the same string as a launch argument (--user-agent=… in extra_args) makes the worker report it too.
    (Observed on the 0.9.5 image with #2448 backported; the User-Agent code is the same without it.)

Related: #2472 / #2473 touch the same auto-added sec-ch-ua, but #2473 only leaves it out of dump(); __init__ is unchanged.

Is this reproducible?

Yes

Inputs Causing the Bug
- URL(s): https://httpbin.org/headers (headers sent), https://abrahamjuliot.github.io/creepjs/ (worker User-Agent)
- Settings: default BrowserConfig; Docker server /md endpoint
Steps to Reproduce
docker run -d --name c4ai --shm-size=1g -p 11235:11235 -e CRAWL4AI_API_TOKEN=test unclecode/crawl4ai:0.9.5
curl -s -X POST http://localhost:11235/md -H "Authorization: Bearer test" \
  -H "Content-Type: application/json" -d '{"url": "https://httpbin.org/headers", "f": "raw"}'
# "User-Agent" and "Sec-Ch-Ua" announce Chrome 116; the bundled Chromium is 156:
docker exec c4ai sh -c 'grep -A3 "\"chromium\"" /usr/local/lib/python3.12/site-packages/playwright/driver/package/browsers.json'
Code snippets
from crawl4ai import BrowserConfig

print(BrowserConfig().user_agent)
# Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
print(BrowserConfig(user_agent="").headers)
# {'sec-ch-ua': ''}
Suggested fix
  • Default to the launched browser's own User-Agent (with HeadlessChrome replaced by Chrome in headless mode) rather than a hard-coded string; at least add (KHTML, like Gecko) and keep the version in step with the bundled Chromium.
  • Skip the sec-ch-ua default when user_agent is empty.
  • Also pass the User-Agent as --user-agent at launch, so workers report the same one.
OS

Linux (Docker on WSL2)

Python version

3.12 (image)

Browser

Chromium bundled with Playwright 1.64 (the stock 0.9.5 image launches chrome-headless-shell, see #2382)

Browser version

156.0.8078.4

Lingua principale
Python
Stelle
87.2k
Fork
9.1k
Merge medio
1g 6h
PR unite (30g)
25

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.