Locale-dependent text IO: tasklist probe and browser launch diagnostics crash under UTF-8 mode; Common Crawl URL cache fails on non-ASCII URLs
维护者通常 1 天内回复
评估
调研方向
Start by reading browser_profiler.py, browser_manager.py, and async_url_seeder.py, focusing on the subprocess output decoding and the Common Crawl JSONL cache reads and writes described in the issue. Add tests for non-ASCII output and URLs under UTF-8 mode, then run the relevant tests. Done means all three paths handle the specified encoding cases without crashes or corrupted cache data.
由索引模型根据 Issue 内容生成。
描述
Summary
Three independent spots decode or write text using the platform default codec. On Windows with a non-UTF-8 ANSI codepage (cp1252/cp936) — and, more broadly, under UTF-8 mode, which becomes the default in Python 3.15 (PEP 686) — they raise UnicodeDecodeError / UnicodeEncodeError or silently corrupt data.
1. BrowserProfiler._is_browser_running (browser_profiler.py)
process = subprocess.run(["tasklist", "/FI", f"PID eq {pid}"],
capture_output=True, text=True)
return str(pid) in process.stdout
tasklist writes ANSI/OEM-encoded text (e.g. GBK on CJK Windows). Under UTF-8 mode the strict decode fails (observed live on a cp936 machine: 'utf-8' codec can't decode byte 0xcf in position 4). UnicodeDecodeError is a ValueError, so it escapes the except (ProcessLookupError, PermissionError, OSError) tuple. Every caller — get_builtin_browser_info(), the launch-time liveness check, and the kill loop — then treats a live builtin browser as dead, so kill/cleanup logic silently degrades and browser processes leak.
2. Chromium stdout/stderr diagnostics (browser_manager.py)
Five bytes.decode() calls decode Chromium's launch diagnostics with the strict default codec. Chromium emits ANSI/OEM-encoded messages on Windows (localized error text is common), so diagnosing an early exit can itself raise UnicodeDecodeError — converting a diagnosable launch failure into an opaque Failed to start browser: ....
3. Common Crawl URL cache (async_url_seeder.py)
_from_cc streams Common Crawl index results into a .jsonl cache and reads it back via aiofiles.open(path, "w" / "r") with no encoding. Real-world CC URLs contain raw non-ASCII characters; under cp1252/cp936 the write raises UnicodeEncodeError mid-generator (it is not caught by the httpx exception handlers), aborting the whole seeding run. A cache written under one locale also mojibakes when read back under another.
Suggested fix
Pin encoding="utf-8" at these spots (plus errors="replace" where console/subprocess output is only searched, not parsed). ASCII-only content is unaffected in all three cases. Patch and tests incoming as a PR.
- 主要语言
- Python
- 星标
- 87.2k
- 派生
- 9.1k
- 平均合并
- 1 天 6 小时
- 30 天内合并 PR
- 25
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
unclecode/crawl4ai 的其他 Issue
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stack可能已有人在做 @HhH666-ux 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.config可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 70/100
unclecode/crawl4ai#2494 · 1 条评论 ·
维护者通常 1 天内回复
-
⚙ Done ✨ Enhancement
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 1 天内回复
-
⚙ Done 🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.media未关闭⚙ Done 🐞 Bug
难度 2/5 1-3 小时 新手友好度 78/100
unclecode/crawl4ai#2469 · 1 条评论 ·
维护者通常 1 天内回复
查看 unclecode/crawl4ai 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
documentation
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
Install the skills as a Claude Code plugin可能已有人在做 @ruban-24 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 6 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
data
难度 2/5 1-3 小时 新手友好度 66/100