Anthropic and Copilot SSE streams still decode UTF-8 per transport chunk (U+FFFD in CJK output)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- backend-api-design
Research direction
Read the Anthropic stream loop in crates/jcode-provider-anthropic-runtime/src/lib.rs around line 2349 and the Copilot loop in crates/jcode-provider-copilot-runtime/src/lib.rs around line 730. Compare their handling with the OpenAI/OpenRouter streams and sse_stream_integrity_tests.rs; verify split multibyte CJK characters decode intact in both paths, then run the relevant tests.
Written by the indexing model from the issue text.
Description
Two provider streams still decode each transport chunk independently with String::from_utf8_lossy. When a chunk boundary splits a multi-byte character, both halves turn into U+FFFD. #609/#613 (OpenAI/OpenRouter) and #1944 (OpenCode Go Responses) fixed the same pattern elsewhere, but these two paths were not covered.
At 3c448c32328e39a58e3b7a87915c5479bf46c6ab:
- https://github.com/1jehuang/jcode/blob/3c448c32328e39a58e3b7a87915c5479bf46c6ab/crates/jcode-provider-anthropic-runtime/src/lib.rs#L2349-L2350
let chunk_str = String::from_utf8_lossy(&chunk); buffer.push_str(&chunk_str); - https://github.com/1jehuang/jcode/blob/3c448c32328e39a58e3b7a87915c5479bf46c6ab/crates/jcode-provider-copilot-runtime/src/lib.rs#L730
buffer.push_str(&String::from_utf8_lossy(&chunk));
Observed (runtime, not just source inspection)
On v0.99.1 (7206efd0c, Windows, Anthropic provider, claude-opus-5-5), Japanese assistant output intermittently contains ��� in place of exactly one character. The bytes are already persisted in the session journal:
履歴を���認するのだ (expected: 履歴を確認するのだ)
こう���るのだ (expected: こうするのだ)
Each occurrence is one 3-byte UTF-8 character replaced by three U+FFFD. That matches a chunk boundary falling inside the character. It happens a few times across long sessions, which fits network-dependent chunking.
(The Copilot path is from source inspection only; I have only observed the Anthropic one.)
Suggested fix
Reuse the existing jcode_core::util::Utf8StreamDecoder in both loops, as the OpenAI/OpenRouter streams already do:
let mut utf8 = jcode_core::util::Utf8StreamDecoder::new();
// ...
buffer.push_str(&utf8.decode(&chunk));
Optionally flush any carried bytes at stream end, and add the same split-at-every-offset CJK test used in sse_stream_integrity_tests.rs.
- Dominant language
- Rust
- Stars
- 19.9k
- Forks
- 2.3k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 12
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from 1jehuang/jcode
-
autonomous: no triage: needs-decision
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
1jehuang/jcode#2079 · 1 comment ·
Maintainers usually reply within 1 day
-
autonomous: no triage: needs-decision
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
1jehuang/jcode#2070 · 1 comment ·
Maintainers usually reply within 1 day
-
bug triage: fixed-pending-release
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
1jehuang/jcode#2019 · 3 comments ·
Maintainers usually reply within 1 day
-
Add GPT-6.1 Terra to the OpenAI model libraryPossibly taken @Aafimalek claimed this 11 days ago. Opentriage: needs-decision
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
1jehuang/jcode#1997 · 2 comments ·
Maintainers usually reply within 1 day
-
bug needs-info triage: unclear
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
1jehuang/jcode#1823 · 2 comments ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
cloudflare/workers-rs#1130 ·
Maintainers usually reply within 11 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
vfarcic/dot-agent-deck#2160 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
RustCrypto/utils#1597 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day