BUG UnicodeReplacementConverter writes characters outside the BMP as invalid five-digit \u escapes
Maintainers usually reply within 2 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Read UnicodeReplacementConverter and compare its output with JsonStringConverter, which the issue says already emits the expected surrogate pair. Reproduce with the provided Python snippet using an emoji and confirm BMP characters retain their current output. Done when non-BMP characters are represented as valid UTF-16 surrogate pairs and the reproduction decodes to the original text.
Written by the indexing model from the issue text.
Description
Describe the bug
UnicodeReplacementConverter formats each character as \u{ord(ch):04x}. :04x is a minimum width, so a character outside the Basic Multilingual Plane (emoji, many CJK extension characters, math alphanumerics) gets five or six hex digits. A \u escape holds exactly four in JSON, JavaScript, Java and Python, so the output decodes to a different character followed by a stray digit.
To Reproduce
import asyncio, json
from pyrit.converter import UnicodeReplacementConverter
out = asyncio.run(UnicodeReplacementConverter().convert_async(prompt="hi 😀")).output_text
print(out) # hi ὠ0
print(json.loads(f'"{out}"')) # hi ὠ0
Expected behavior
😀 is written as its UTF-16 surrogate pair, 😀, which is what JsonStringConverter already emits for the same character. BMP characters keep their current output.
Version
main at 60c3386a, Python 3.11, Windows.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 944
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 309
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
microsoft/PyRIT#3095 · 3 comments ·
Maintainers usually reply within 2 days
-
BUG Configuration keeps runtime-status errors after polling recoversPossibly taken @joeyvictorino claimed this 3 days ago. OpenBug: triage GUI help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
microsoft/PyRIT#2868 · 4 comments · 1 assignee ·
Maintainers usually reply within 2 days
-
BUG: MemoryInterface.get_request_from_response fails with IndexError for non-zero starting sequence or sequence gapsPossibly taken @RohithPariki claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
Maintainers usually reply within 2 days
-
Conversation editor never revokes blob URLs of removed attachmentsPossibly taken @ManoharPaturi claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
Maintainers usually reply within 2 days
-
Scenario progress double-counts an attack result after its timestamp is bumpedPossibly taken @ManoharPaturi claimed this today. Open
Difficulty 3/5 Half a day Newbie friendliness 25/100
Maintainers usually reply within 2 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
task
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
danparshall/claude_researcher#87 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
datajoint/datajoint-python#1586 ·
Maintainers usually reply within 1 day
-
1D intercept of length 1 is accepted as `fix_params` in population GLM with more than 1 unitPossibly taken @dhvani7014 claimed this today. Openbug good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
flatironinstitute/nemos#642 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
uniswap-python/uniswap-python#487 ·