Locale-dependent text IO: tasklist probe and browser launch diagnostics crash under UTF-8 mode; Common Crawl URL cache fails on non-ASCII URLs

未关闭
#2,439 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@sun04222 已经在做这个了。

开始于 2026年10月24日。

  • #2440 来自 @sun04222 —— 未关闭

评估

难度
3/5
预计耗时
1-2 天
新手友好度
68/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
backend

调研方向

Start by reading browser_profiler.py, browser_manager.py, and async_url_seeder.py, focusing on the subprocess output decoding and the Common Crawl JSONL cache reads and writes described in the issue. Add tests for non-ASCII output and URLs under UTF-8 mode, then run the relevant tests. Done means all three paths handle the specified encoding cases without crashes or corrupted cache data.

由索引模型根据 Issue 内容生成。

描述

⚙️ In-progress ⚙️ Under Review 🐞 Bug

Summary

Three independent spots decode or write text using the platform default codec. On Windows with a non-UTF-8 ANSI codepage (cp1252/cp936) — and, more broadly, under UTF-8 mode, which becomes the default in Python 3.15 (PEP 686) — they raise UnicodeDecodeError / UnicodeEncodeError or silently corrupt data.

1. BrowserProfiler._is_browser_running (browser_profiler.py)
process = subprocess.run(["tasklist", "/FI", f"PID eq {pid}"],
                         capture_output=True, text=True)
return str(pid) in process.stdout

tasklist writes ANSI/OEM-encoded text (e.g. GBK on CJK Windows). Under UTF-8 mode the strict decode fails (observed live on a cp936 machine: 'utf-8' codec can't decode byte 0xcf in position 4). UnicodeDecodeError is a ValueError, so it escapes the except (ProcessLookupError, PermissionError, OSError) tuple. Every caller — get_builtin_browser_info(), the launch-time liveness check, and the kill loop — then treats a live builtin browser as dead, so kill/cleanup logic silently degrades and browser processes leak.

2. Chromium stdout/stderr diagnostics (browser_manager.py)

Five bytes.decode() calls decode Chromium's launch diagnostics with the strict default codec. Chromium emits ANSI/OEM-encoded messages on Windows (localized error text is common), so diagnosing an early exit can itself raise UnicodeDecodeError — converting a diagnosable launch failure into an opaque Failed to start browser: ....

3. Common Crawl URL cache (async_url_seeder.py)

_from_cc streams Common Crawl index results into a .jsonl cache and reads it back via aiofiles.open(path, "w" / "r") with no encoding. Real-world CC URLs contain raw non-ASCII characters; under cp1252/cp936 the write raises UnicodeEncodeError mid-generator (it is not caught by the httpx exception handlers), aborting the whole seeding run. A cache written under one locale also mojibakes when read back under another.

Suggested fix

Pin encoding="utf-8" at these spots (plus errors="replace" where console/subprocess output is only searched, not parsed). ASCII-only content is unaffected in all three cases. Patch and tests incoming as a PR.

主要语言
Python
星标
87.2k
派生
9.1k
平均合并
1 天 6 小时
30 天内合并 PR
25

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。