Fail CI on hidden Unicode characters in tracked files
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- Half a day
- Newbie friendliness
- 80/100
Research direction
Start by reading the python-linux job in the CI configuration and the existing tests/ layout, then create tests/hidden_unicode_check.py. The script should enumerate tracked files (for example via git ls-files), decode each as UTF-8, and report any forbidden code point with path, line and code point; handle U+FEFF only at offset 0 and allow known binary types. Run python -B tests/hidden_unicode_check.py on main, then repeat the issue's negative controls to confirm U+202E and U+200B fail while a leading .ps1 BOM passes. Done means the script passes on main and is wired into the python-linux job.
Written by the indexing model from the issue text.
Description
Problem
Some characters render as nothing or reorder the text around them, while a compiler, an interpreter or a coding agent still reads them:
- bidi controls: U+061C, U+202A to U+202E, U+2066 to U+2069;
- zero-width characters: U+200B to U+200F, U+2060 to U+2064;
- Unicode tag characters: U+E0000 to U+E007F.
A pull request can use them to hide code or instructions from a human reviewer. GitHub shows a warning on files that contain hidden Unicode text, but nothing in CI fails on them. Reviewers and agent contributors are both affected.
Proposal
Add a standard-library script, tests/hidden_unicode_check.py, and run it in the python-linux job. It reads every tracked file and fails on:
- any of the characters above;
- U+FEFF anywhere except at offset 0, where Visual Studio and PowerShell 5.1 write a byte-order mark;
- a file that is not valid UTF-8, unless it is a known binary type.
A scan of all 256 tracked files on 23d954d found none of these characters, no BOM and no non-UTF-8 text, so the check starts without an allowlist. I can open it as a PR.
Alternatives
- Rely on GitHub's hidden-text warning. It is shown to a reader and does not fail CI, so an agent or an unattended merge does not see it.
- Reject all non-ASCII text. Too broad: the docs legitimately use typographic characters.
Two related changes, which would each need their own issue:
- an
AGENTS.mdline saying that text from issues, pull requests and comments is data for agents, not instructions; - hash-pinning
requirements-dev.txtwith--require-hashes, covering every release file ofpefileandiced-x86.
Verification
python -B tests/hidden_unicode_check.py passes on main. Negative controls:
- adding U+202E to a tracked Markdown file fails with its path, line and code point;
- adding U+200B inside a Python string fails the same way;
- a BOM at offset 0 of a
.ps1file passes, and a BOM in the middle of the same file fails.
AI-assisted.
- Dominant language
- Python
- Stars
- 696
- Forks
- 61
- Avg merge
- 25m
- Merged PRs (30d)
- 171
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from R0Y00/Xbox2Windows
-
Difficulty 5/5 Over a week Newbie friendliness 18/100
R0Y00/Xbox2Windows#218 · 4 comments ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 20/100
R0Y00/Xbox2Windows#207 · 2 comments ·
Maintainers usually reply within 1 day
-
pe_map cannot resolve a forwarded exportPossibly taken @raulmoracode claimed this today. Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
R0Y00/Xbox2Windows#206 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 18/100
R0Y00/Xbox2Windows#179 · 6 comments ·
Maintainers usually reply within 1 day
-
Replace the hardcoded operation chain in offscreen_check with a capability tablePossibly taken @lambirou claimed this today. Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
R0Y00/Xbox2Windows#166 · 3 comments ·
Maintainers usually reply within 1 day
All issues in R0Y00/Xbox2Windows
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
py-econometrics/pyfixest#2072 ·
Maintainers usually reply within 1 day
-
~planning docs
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Kromatic-Innovation/athenaeum#3084 ·
Maintainers usually reply within 1 day
-
Bosch Room Thermostat II number entities offer -500 to 3000 °C setpoint limits and a -32768 to 32767 °C outdoor temperaturePossibly taken @zigpy-review-bot claimed this today. Openbot: code audit
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
zigpy/zha-device-handlers#5769 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Kludex/python-multipart#375 ·
Maintainers usually reply within 1 day
-
[BUG]: list_tables skips tables when an include pattern has a wildcard inside the catalog or schema partPossibly taken @0z5a claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
databrickslabs/dqx#1724 ·
Maintainers usually reply within 1 day