Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Windows: Python pipe encoding corrupts Unicode shell output despite UTF-8 PowerShell prefix

Aperta Adatta ai principianti
  • Qualcosa non va?
#54,836 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
84/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
powershell, python, rust
Ambito
cli, internationalization

Direzione di ricerca

Inizia dalla costruzione del comando PowerShell e dai relativi test in codex-rs/shell-command/src/powershell.rs. Riproduci l’errore dell’output cinese di stdout/stderr reindirizzato, quindi aggiungi una copertura per l’output UTF-8 e per la conservazione di un’impostazione cp936:strict esplicita. Esegui cargo test -p codex-shell-command powershell::tests --lib; la modifica è completa quando l’output Unicode reindirizzato viene preservato senza sovrascrivere le codifiche Python esplicite.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug CLI tool-calls windows-os

Problem

Python's redirected standard streams on Windows CP936 can corrupt Chinese shell-tool output even in PowerShell 7 and even when the source file was explicitly read as UTF-8. If an agent uses the corrupted shell output to update an automation prompt, the corrupted text becomes persistent. This is a reproducible shell-output boundary issue, not evidence of a scheduler serialization defect.

Environment

  • Codex CLI bundled with the affected desktop installation: 0.162.0-alpha.17.2
  • Windows, ANSI code page 936; Python 3.12.12 with redirected stdout encoding gbk
  • Reproduced with PowerShell 7.6.5 and Windows PowerShell 5.1
  • Proposed fix tested against official main ce0323f2f2d3b25e82806cd555237da3db4e1897

Minimal reproduction

Run the following through the Codex shell tool / redirected process output, rather than a Python interactive console:

Remove-Item Env:PYTHONIOENCODING -ErrorAction SilentlyContinue
$env:PYTHONUTF8 = '0'
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
python -c 'import json; print(json.dumps(chr(0x7ee7)+chr(0x7eed),ensure_ascii=False))'

Expected: JSON containing the original Chinese characters. Actual: replacement characters. The PowerShell console prefix is insufficient because Python uses the ANSI code page for redirected pipes independently of Console.OutputEncoding.

Changing only the child-process encoding fixes it:

$env:PYTHONIOENCODING = 'utf-8'
python -c 'import json; print(json.dumps(chr(0x7ee7)+chr(0x7eed),ensure_ascii=False))'

Other caller-side workarounds are python -X utf8 or ASCII-escaped JSON. Explicit file decoding alone does not solve the output boundary.

Why it persisted in an automation

The file was decoded correctly in Python, printed using json.dumps(..., ensure_ascii=False) to a GBK pipe, and received as UTF-8 with replacement characters. JSON punctuation survived, so parsing still succeeded. An agent then reused the damaged string in an automation update. An unchecked missing section marker in that agent's read/modify/write code additionally retained corrupted old content instead of replacing it. Those agent-side errors amplified the shell encoding problem; this report does not blame the scheduler for independently damaging valid input.

Concrete fix and regression tests

Proposed commit with tests / diff against upstream.

In codex-rs/shell-command/src/powershell.rs, extend the existing UTF-8 prefix with:

if (-not $env:PYTHONIOENCODING) { $env:PYTHONIOENCODING='utf-8' }

This preserves nonempty explicit encodings and changes Python's standard streams, not default file decoding. Added Windows tests verify Chinese stdout/stderr and preservation of an explicit cp936:strict setting. Existing prefix idempotence coverage remains.

Validation on Windows:

  • cargo test -p codex-shell-command powershell::tests --lib: 11 passed.
  • cargo test -p codex-shell-command --lib: 175 passed.
  • Independent real-child-process matrix for PowerShell 7 and 5.1: original-prefix corruption reproduced, fixed stdout/stderr preserved, explicit encoding preserved (6 checks).
  • rustfmt --edition 2024 --check and git diff --check: passed.

Local mitigation sets shell_environment_policy.set.PYTHONIOENCODING = "utf-8". An already-running process may not reload the setting, so explicit UTF-8 is still used for current commands. The damaged task prompt was rewritten from clean text through a structured Unicode argument and checked after saving.

Related #23044 concerns PowerShell 5.1 Get-Content's file-decoding default; this is a different boundary and reproduces in PowerShell 7. #31219 describes handoff corruption without this minimal pipe reproduction.

The affected user requested an upstream PR. I read the current contribution policy; GitHub rejected upstream PR creation with CreatePullRequest permission denial. This issue follows the policy's supported route for a root-cause report and concrete fix proposal. No real automation prompts, private logs, credentials, or personal paths are included.

Lingua principale
Rust
Stelle
130k
Fork
20.5k
Merge medio
1m
PR unite (30g)
997

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di openai/codex

Tutte le issue di openai/codex

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.