Windows: Python pipe encoding corrupts Unicode shell output despite UTF-8 PowerShell prefix

オープン 初心者向け
  • 何か問題がありますか？
#54,836 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
2/5
見積もり時間
1〜3時間
初心者へのやさしさ
84/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
powershell, python, rust
領域
cli, internationalization

調査の方向性

まず codex-rs/shell-command/src/powershell.rs にある PowerShell コマンドの構築とそのテストを確認します。redirect された中国語の stdout/stderr の障害を再現し、UTF-8 出力と明示的な cp936:strict 設定の保持をカバーするテストを追加します。cargo test -p codex-shell-command powershell::tests --lib を実行します。redirect された Unicode 出力が、明示的な Python のエンコーディングを上書きせずに保持されるようになれば、変更は完了です。

索引モデルが issue の本文から書いたものです。

説明

bug CLI tool-calls windows-os

Problem

Python's redirected standard streams on Windows CP936 can corrupt Chinese shell-tool output even in PowerShell 7 and even when the source file was explicitly read as UTF-8. If an agent uses the corrupted shell output to update an automation prompt, the corrupted text becomes persistent. This is a reproducible shell-output boundary issue, not evidence of a scheduler serialization defect.

Environment

  • Codex CLI bundled with the affected desktop installation: 0.162.0-alpha.17.2
  • Windows, ANSI code page 936; Python 3.12.12 with redirected stdout encoding gbk
  • Reproduced with PowerShell 7.6.5 and Windows PowerShell 5.1
  • Proposed fix tested against official main ce0323f2f2d3b25e82806cd555237da3db4e1897

Minimal reproduction

Run the following through the Codex shell tool / redirected process output, rather than a Python interactive console:

Remove-Item Env:PYTHONIOENCODING -ErrorAction SilentlyContinue
$env:PYTHONUTF8 = '0'
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
python -c 'import json; print(json.dumps(chr(0x7ee7)+chr(0x7eed),ensure_ascii=False))'

Expected: JSON containing the original Chinese characters. Actual: replacement characters. The PowerShell console prefix is insufficient because Python uses the ANSI code page for redirected pipes independently of Console.OutputEncoding.

Changing only the child-process encoding fixes it:

$env:PYTHONIOENCODING = 'utf-8'
python -c 'import json; print(json.dumps(chr(0x7ee7)+chr(0x7eed),ensure_ascii=False))'

Other caller-side workarounds are python -X utf8 or ASCII-escaped JSON. Explicit file decoding alone does not solve the output boundary.

Why it persisted in an automation

The file was decoded correctly in Python, printed using json.dumps(..., ensure_ascii=False) to a GBK pipe, and received as UTF-8 with replacement characters. JSON punctuation survived, so parsing still succeeded. An agent then reused the damaged string in an automation update. An unchecked missing section marker in that agent's read/modify/write code additionally retained corrupted old content instead of replacing it. Those agent-side errors amplified the shell encoding problem; this report does not blame the scheduler for independently damaging valid input.

Concrete fix and regression tests

Proposed commit with tests / diff against upstream.

In codex-rs/shell-command/src/powershell.rs, extend the existing UTF-8 prefix with:

if (-not $env:PYTHONIOENCODING) { $env:PYTHONIOENCODING='utf-8' }

This preserves nonempty explicit encodings and changes Python's standard streams, not default file decoding. Added Windows tests verify Chinese stdout/stderr and preservation of an explicit cp936:strict setting. Existing prefix idempotence coverage remains.

Validation on Windows:

  • cargo test -p codex-shell-command powershell::tests --lib: 11 passed.
  • cargo test -p codex-shell-command --lib: 175 passed.
  • Independent real-child-process matrix for PowerShell 7 and 5.1: original-prefix corruption reproduced, fixed stdout/stderr preserved, explicit encoding preserved (6 checks).
  • rustfmt --edition 2024 --check and git diff --check: passed.

Local mitigation sets shell_environment_policy.set.PYTHONIOENCODING = "utf-8". An already-running process may not reload the setting, so explicit UTF-8 is still used for current commands. The damaged task prompt was rewritten from clean text through a structured Unicode argument and checked after saving.

Related #23044 concerns PowerShell 5.1 Get-Content's file-decoding default; this is a different boundary and reproduces in PowerShell 7. #31219 describes handoff corruption without this minimal pipe reproduction.

The affected user requested an upstream PR. I read the current contribution policy; GitHub rejected upstream PR creation with CreatePullRequest permission denial. This issue follows the policy's supported route for a root-cause report and concrete fix proposal. No real automation prompts, private logs, credentials, or personal paths are included.

主要言語
Rust
スター
133k
フォーク
21.1k
平均マージ
1分
マージ済み PR（30日）
997

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

openai/codex のほかの issue

openai/codex の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。