Digests of parsed JSON with ensure_ascii=False raise UnicodeEncodeError for a lone-surrogate escape in the input

未关闭
  • 哪里不对？
#720 0 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@antonwolfy 已经在做这个了。

开始于 2026年11月13日。

评估

这个 Issue 还没有评估数据。

描述

Several digests dump parsed data with ensure_ascii=False and encode with .encode("utf-8"): item_digest and assignment_digest (dokimi/dokimi/eval/leakgate.py:162,175), graph_digest (dokimi/dokimi/eval_coverage.py:71), expectations_digest (dokimi/dokimi/repos/conformance.py:108), response_cache_key (dokimi/dokimi/caching.py:377) and _content_key (dokimi/dokimi/harnesses/harbor_export.py:100, resolved.encode()). A JSON or YAML escape such as "\udcff" in the input file loads as a lone surrogate and the encode raises. Plain file text is not a trigger: it is decoded as strict UTF-8.

Repro, each with json.loads('"n\\udcff"') as the value:

leakgate.item_digest({"grading": {"expected": S}}) -> UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 14: surrogates not allowed
eval_coverage.graph_digest({"nodes": [{"id": "a", "text": S, "path_class": "x"}]}) -> ... position 65
conformance.expectations_digest({"a": S}) -> ... position 7
caching.response_cache_key(litellm_model="m", messages=[{"role": "user", "content": S}]) -> ... position 41
harbor_export._content_key(path, "n\udcff") -> ... position 1

Fix options: .encode("utf-8", "surrogatepass") keeps every digest that encodes today unchanged and separates different surrogates; or refuse a lone surrogate in the loader with a message. Not errors="replace", which makes two inputs collide. Whether _content_key is reached depends on _task_toml writing a raw surrogate, which was not traced.

Found while fixing #675. The fix's PR is #721.

主要语言
Python
星标
15
派生
11
平均合并
1 天 9 小时
30 天内合并 PR
464

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

intel/dokimi 的其他 Issue

查看 intel/dokimi 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。