Fail CI on hidden Unicode characters in tracked files

Open Beginner friendly
#254 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
Half a day
Newbie friendliness
80/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions, python
Domain
ci-cd, testing

Research direction

Start by reading the python-linux job in the CI configuration and the existing tests/ layout, then create tests/hidden_unicode_check.py. The script should enumerate tracked files (for example via git ls-files), decode each as UTF-8, and report any forbidden code point with path, line and code point; handle U+FEFF only at offset 0 and allow known binary types. Run python -B tests/hidden_unicode_check.py on main, then repeat the issue's negative controls to confirm U+202E and U+200B fail while a leading .ps1 BOM passes. Done means the script passes on main and is wired into the python-linux job.

Written by the indexing model from the issue text.

Description

Problem

Some characters render as nothing or reorder the text around them, while a compiler, an interpreter or a coding agent still reads them:

  • bidi controls: U+061C, U+202A to U+202E, U+2066 to U+2069;
  • zero-width characters: U+200B to U+200F, U+2060 to U+2064;
  • Unicode tag characters: U+E0000 to U+E007F.

A pull request can use them to hide code or instructions from a human reviewer. GitHub shows a warning on files that contain hidden Unicode text, but nothing in CI fails on them. Reviewers and agent contributors are both affected.

Proposal

Add a standard-library script, tests/hidden_unicode_check.py, and run it in the python-linux job. It reads every tracked file and fails on:

  • any of the characters above;
  • U+FEFF anywhere except at offset 0, where Visual Studio and PowerShell 5.1 write a byte-order mark;
  • a file that is not valid UTF-8, unless it is a known binary type.

A scan of all 256 tracked files on 23d954d found none of these characters, no BOM and no non-UTF-8 text, so the check starts without an allowlist. I can open it as a PR.

Alternatives
  • Rely on GitHub's hidden-text warning. It is shown to a reader and does not fail CI, so an agent or an unattended merge does not see it.
  • Reject all non-ASCII text. Too broad: the docs legitimately use typographic characters.

Two related changes, which would each need their own issue:

  • an AGENTS.md line saying that text from issues, pull requests and comments is data for agents, not instructions;
  • hash-pinning requirements-dev.txt with --require-hashes, covering every release file of pefile and iced-x86.
Verification

python -B tests/hidden_unicode_check.py passes on main. Negative controls:

  • adding U+202E to a tracked Markdown file fails with its path, line and code point;
  • adding U+200B inside a Python string fails the same way;
  • a BOM at offset 0 of a .ps1 file passes, and a BOM in the middle of the same file fails.

AI-assisted.

Dominant language
Python
Stars
696
Forks
61
Avg merge
25m
Merged PRs (30d)
171

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from R0Y00/Xbox2Windows

All issues in R0Y00/Xbox2Windows

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.