Locale-dependent text IO: tasklist probe and browser launch diagnostics crash under UTF-8 mode; Common Crawl URL cache fails on non-ASCII URLs
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 68/100
Línea de trabajo
Start by reading browser_profiler.py, browser_manager.py, and async_url_seeder.py, focusing on the subprocess output decoding and the Common Crawl JSONL cache reads and writes described in the issue. Add tests for non-ASCII output and URLs under UTF-8 mode, then run the relevant tests. Done means all three paths handle the specified encoding cases without crashes or corrupted cache data.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Summary
Three independent spots decode or write text using the platform default codec. On Windows with a non-UTF-8 ANSI codepage (cp1252/cp936) — and, more broadly, under UTF-8 mode, which becomes the default in Python 3.15 (PEP 686) — they raise UnicodeDecodeError / UnicodeEncodeError or silently corrupt data.
1. BrowserProfiler._is_browser_running (browser_profiler.py)
process = subprocess.run(["tasklist", "/FI", f"PID eq {pid}"],
capture_output=True, text=True)
return str(pid) in process.stdout
tasklist writes ANSI/OEM-encoded text (e.g. GBK on CJK Windows). Under UTF-8 mode the strict decode fails (observed live on a cp936 machine: 'utf-8' codec can't decode byte 0xcf in position 4). UnicodeDecodeError is a ValueError, so it escapes the except (ProcessLookupError, PermissionError, OSError) tuple. Every caller — get_builtin_browser_info(), the launch-time liveness check, and the kill loop — then treats a live builtin browser as dead, so kill/cleanup logic silently degrades and browser processes leak.
2. Chromium stdout/stderr diagnostics (browser_manager.py)
Five bytes.decode() calls decode Chromium's launch diagnostics with the strict default codec. Chromium emits ANSI/OEM-encoded messages on Windows (localized error text is common), so diagnosing an early exit can itself raise UnicodeDecodeError — converting a diagnosable launch failure into an opaque Failed to start browser: ....
3. Common Crawl URL cache (async_url_seeder.py)
_from_cc streams Common Crawl index results into a .jsonl cache and reads it back via aiofiles.open(path, "w" / "r") with no encoding. Real-world CC URLs contain raw non-ASCII characters; under cp1252/cp936 the write raises UnicodeEncodeError mid-generator (it is not caught by the httpx exception handlers), aborting the whole seeding run. A cache written under one locale also mojibakes when read back under another.
Suggested fix
Pin encoding="utf-8" at these spots (plus errors="replace" where console/subprocess output is only searched, not parsed). ASCII-only content is unaffected in all three cases. Patch and tests incoming as a PR.
- Lenguaje dominante
- Python
- Estrellas
- 87.2k
- Forks
- 9.1k
- Merge medio
- 1 d 6 h
- PR fusionados (30 d)
- 25
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackPosiblemente ocupada @HhH666-ux la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
unclecode/crawl4ai#2494 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAbierto⚙ Done ✨ Enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 1 día
-
⚙ Done 🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAbierto⚙ Done 🐞 Bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
unclecode/crawl4ai#2469 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de unclecode/crawl4ai
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 1 día
-
documentation
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
Install the skills as a Claude Code pluginPosiblemente ocupada @ruban-24 la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 6 días
-
Make the usage of time-zone aware UTC timestamps the norm across the pynxtools NOMAD ecosystemAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationAbiertodata
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100