RemoteFortressReader: GetCreatureRaws sends CP437 (not UTF-8) strings, which breaks protobuf decoding
メンテナーはふだん 2 日以内に返信
評価
調査の方向性
Read plugins/remotefortressreader/remotefortressreader.cpp and check the creature-raw string fields, starting with the caste description assignment identified in the issue. Convert game-sourced strings to UTF-8 consistently, then validate GetCreatureRaws replies with strict UTF-8 decoding. A linked pull request is already open, so check its changes and status before taking this on.
索引モデルが issue の本文から書いたものです。
説明
Disclaimer: this issue was drafted by GitHub Copilot (an AI assistant) under the direction of user @dwmuller. The test results below come from real runs against a live game, but the source-code analysis is AI-generated, so please verify it.
Summary
RemoteFortressReader.GetCreatureRaws returns some strings in the game's CP437 encoding instead of UTF-8. Protobuf string fields must be UTF-8, so strict clients (Rust prost, and likely others) reject the whole reply. A single bad string makes the 23 MB reply undecodable.
The 50.08-r2 changelog says "RemoteFortressReader: ensured names are transmitted in UTF-8 instead of CP437", but DF2UTF is applied selectively and was not applied here.
Environment
- Dwarf Fortress 53.16
- DFHack 53.16-r3
- Remote client: Rust
dfhack-remote0.12.0 (prost) - Fortress with many generated creatures (raw indices ~2026-2085 are affected)
Observed
Decoding fails with:
failed to decode Protobuf message: CasteRaw.description: CreatureRaw.caste:
CreatureRawList.creature_raws: invalid string value: data is not UTF-8 encoded.
I walked the raw reply bytes without UTF-8 validation and found 74 non-UTF-8 strings. They fall into three fields:
| Field | Count | Example (raw bytes) |
|---|---|---|
CreatureRaw.name (field 3) |
19 | Being of r\x86sh |
CreatureRaw.caste[].caste_name (path 1.10.3) |
31 | Being of r\x86sh |
CreatureRaw.caste[].description (path 1.10.15) |
24 | ...created by the dwarven goddess R\x86sh and is of a part with deformity... |
0x86 is "å" in CP437, which fits these generated names ("Råsh").
Paging GetPartialCreatureRaws and bisecting shows that exactly 14 raws in the 2026-2085 range fail. Indices before 2026 decode fine.
Cause (from reading the source)
In plugins/remotefortressreader/remotefortressreader.cpp, the caste loop has:
send_caste->set_description(orig_caste->description);
There is no DF2UTF(...). The CreatureRaw/CasteRaw name fields (name, caste_name, baby/child names) show the same symptom in the data and should be checked as well. I only confirmed the unconverted set_description call in the source.
Not affected in this fortress
GetUnitListandGetUnitListInside(305 units) have no invalid UTF-8.GetMaterialList,GetGrowthList,GetItemListandGetPlantRawshave no invalid UTF-8.
I looked at these specifically because they also carry caste strings. They're fine here, but it's worth auditing every string that reaches a protobuf message.
Suggested fix
Wrap all game-sourced strings in the creature-raw path with DF2UTF, and audit the other set_* string calls in the plugin for the same omission.
Reproduction
Any client works. Call GetCreatureRaws and validate the reply strictly as UTF-8. I wrote a small test harness that does this. It's in a project of mine, and everything is read-only:
Run it against a live game (DFHack remote on port 5000):
cargo test --test live_game_tests -- --ignored --nocapture --test-threads=1
creature_rawsandcreature_raws_pageduse the typed client, and the paged one bisects to the failing indices.raw_creature_raws_utf8bypasses prost, reads the protobuf wire format directly, and lists every non-UTF-8 field by path with example bytes.- The other
raw_*tests do the same for the unit and material/item lists.
- 主要言語
- C++
- スター
- 2.1k
- フォーク
- 509
- 平均マージ
- 2日 13時間
- マージ済み PR（30日）
- 28
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
DFHack/dfhack のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 2 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
DFHack/dfhack#5853 · コメント 1 件 ·
メンテナーはふだん 2 日以内に返信
-
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
メンテナーはふだん 2 日以内に返信
-
Floor-mosaic plannerオープン
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
メンテナーはふだん 2 日以内に返信
-
up stair weirdnessオープン
難易度 3/5 1〜2日 初心者へのやさしさ 35/100
メンテナーはふだん 2 日以内に返信
似ている issue
-
[Docs] CONTRIBUTING still says FreeType 2.14.3 and TabLabel.h says ImGui 1.92.9b after the bumps対応中かも @mgradwohl が今日担当しました。 オープンagent:WSL documentation LOW
難易度 1/5 1〜3時間 初心者へのやさしさ 83/100
メンテナーはふだん 1 日以内に返信
-
cudf-polars feature request
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
メンテナーはふだん 2 日以内に返信
-
(S1 - Need confirmation)
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
CleverRaven/Cataclysm-DDA#89212 ·
メンテナーはふだん 1 日以内に返信
-
Status: Needs confirmation Status: Needs triage
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
メンテナーはふだん 1 日以内に返信
-
a byte order mark at the start of a migration makes dbmate miss the up block対応中かも @Dem1241 が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
メンテナーはふだん 2 日以内に返信