[Bug]: Default user agent is a stale Chrome/116 string that contradicts the bundled Chromium
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 35/100
Direzione di ricerca
Start in crawl4ai/async_configs.py, where BrowserConfig sets the default user_agent and, in __init__, calls self.headers.setdefault("sec-ch-ua", self.browser_hint). Then check the fallback in user_agent_generator.py and generate_client_hints. Done means an empty user_agent no longer produces an empty sec-ch-ua header, and the default User-Agent is consistent with the bundled Chromium. Maintainers should first settle whether the default is hard-coded or read from the launched browser.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
crawl4ai version
0.9.5 (Docker image unclecode/crawl4ai:0.9.5); the same BrowserConfig code is on develop.
Expected Behavior
The default BrowserConfig.user_agent matches the browser crawl4ai launches.
An empty user_agent (to keep the browser's own User-Agent) sends no forged client hints.
Current Behavior
- Stale, non-standard default.
BrowserConfig.user_agentdefaults to
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
(crawl4ai/async_configs.py, also thefallbackofuser_agent_generator.py).
It lacks the(KHTML, like Gecko)token Chrome sends, and announces Chrome 116, while the 0.9.5 image runs Playwright 1.64 with Chromium 156.0.8078.4.
The generatedsec-ch-uasays 116 too, whereas the other request headers (Accept-Encoding: … zstd,Priority: u=0, i) come from Chromium 156.
TheBrowserConfigdocstring documents yet another default, a well-formed macOS User-Agent.
Since Playwright is not pinned (playwright>=1.49.0), the bundled Chromium changes with each image build, so a hard-coded string keeps drifting. - Empty
sec-ch-uawith an empty User-Agent.BrowserConfig(user_agent="")keeps the browser's native User-Agent, but__init__still runsself.headers.setdefault("sec-ch-ua", self.browser_hint), andgenerate_client_hints("")returns'': every request then carries an emptysec-ch-uaheader. - A worker keeps the real User-Agent. With the default config, CreepJS reports in its
ServiceWorkerGlobalScopeworker the User-Agent… HeadlessChrome/156.0.0.0 …, while the page saysChrome/116.
Passing the same string as a launch argument (--user-agent=…inextra_args) makes the worker report it too.
(Observed on the 0.9.5 image with #2448 backported; the User-Agent code is the same without it.)
Related: #2472 / #2473 touch the same auto-added sec-ch-ua, but #2473 only leaves it out of dump(); __init__ is unchanged.
Is this reproducible?
Yes
Inputs Causing the Bug
- URL(s): https://httpbin.org/headers (headers sent), https://abrahamjuliot.github.io/creepjs/ (worker User-Agent)
- Settings: default BrowserConfig; Docker server /md endpoint
Steps to Reproduce
docker run -d --name c4ai --shm-size=1g -p 11235:11235 -e CRAWL4AI_API_TOKEN=test unclecode/crawl4ai:0.9.5
curl -s -X POST http://localhost:11235/md -H "Authorization: Bearer test" \
-H "Content-Type: application/json" -d '{"url": "https://httpbin.org/headers", "f": "raw"}'
# "User-Agent" and "Sec-Ch-Ua" announce Chrome 116; the bundled Chromium is 156:
docker exec c4ai sh -c 'grep -A3 "\"chromium\"" /usr/local/lib/python3.12/site-packages/playwright/driver/package/browsers.json'
Code snippets
from crawl4ai import BrowserConfig
print(BrowserConfig().user_agent)
# Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
print(BrowserConfig(user_agent="").headers)
# {'sec-ch-ua': ''}
Suggested fix
- Default to the launched browser's own User-Agent (with
HeadlessChromereplaced byChromein headless mode) rather than a hard-coded string; at least add(KHTML, like Gecko)and keep the version in step with the bundled Chromium. - Skip the
sec-ch-uadefault whenuser_agentis empty. - Also pass the User-Agent as
--user-agentat launch, so workers report the same one.
OS
Linux (Docker on WSL2)
Python version
3.12 (image)
Browser
Chromium bundled with Playwright 1.64 (the stock 0.9.5 image launches chrome-headless-shell, see #2382)
Browser version
156.0.8078.4
- Lingua principale
- Python
- Stelle
- 87.2k
- Fork
- 9.1k
- Merge medio
- 1g 6h
- PR unite (30g)
- 25
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackForse già presa @HhH666-ux l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
I maintainer di solito rispondono entro 1 giorno
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configForse già presa Una pull request collegata a questa issue è aperta o già unita. Aperta🐞 Bug 📌 Root caused
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
unclecode/crawl4ai#2494 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAperta⚙ Done ✨ Enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
I maintainer di solito rispondono entro 1 giorno
-
⚙ Done 🐞 Bug 📌 Root caused
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAperta⚙ Done 🐞 Bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
unclecode/crawl4ai#2469 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di unclecode/crawl4ai
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 73/100
Netcracker/qubership-ai-packages#266 ·
I maintainer di solito rispondono entro 1 giorno
-
PROCESSED_DATA_FOLDER: strip whitespace and treat empty as unsetForse già presa @TorecLuik l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
NL-BioImaging/BIOMERO.importer#63 ·
I maintainer di solito rispondono entro 1 giorno
-
MLX training-complete chat/merge suggestions omit --backend mlxForse già presa @vaputa l’ha presa oggi. Apertaneeds-triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
MakazhanAlpamys/Soup#2913 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
area:security effort:S priority:P3 status:needs-triage type:chore
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
randomparity/kdive#3657 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
aws/mcp-proxy-for-aws#476 ·
I maintainer di solito rispondono entro 1 giorno