HermitCrab analysis: finish each stratum as it completes instead of keeping every state until synthesis (per-word memory)

オープン
  • 何か問題がありますか？
#639 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
リファクタリング
明瞭さ
おおむね明確
活発さ
活発
技術スタック
csharp
領域
backend, performance

調査の方向性

Read AnalysisLanguageRule.cs and Morpher.cs to trace how stratum results reach synthesis, then inspect AnalysisStratumRule.cs for state merging and MaxAlternatives behavior. Measure peak working set on a heavy word such as Mbugwe mindeshi after #593 and #595. Done means reducing how many full states remain live while preserving uncapped analyses; verify capped-word exceptions, result enumeration order, and tracing behavior.

索引モデルが issue の本文から書いたものです。

説明

Hermit Crab performance

HermitCrab analysis keeps every word of every stratum alive until the whole analysis ends, so per-word memory grows linearly with the number of distinct states reached. AnalysisLanguageRule.Apply adds each stratum's outputs to one results set, Morpher.ParseWord copies that into a queue, and only then does synthesis start. The only bound is MaxAlternatives, which counts alternatives, not bytes.

Recorded for later, in the same family as #593 and #595 (which shrink each word; this one is about how many are live at once). C# numbers for this mechanism do not exist yet; the numbers below are PanGloss ones. The C# structure is the same: breadth-first stratum search, a deduplicating output set, equivalents folded into a canonical.

Idea (the PanGloss approach, owner-approved there): finish each stratum's results as the stratum completes (hand its canonicals to synthesis, drop them once the next stratum has consumed them), keep a compact exact visited key per state instead of the whole word, and keep the full search tree only when tracing (Try a Word).

Trade-off: uncapped words give identical analyses. A capped word's partial result changes in PanGloss because work is spent in a different order. In C# the cap throws MaxAlternativesExceededException rather than returning partials, so the outcome is expected to stay "throws", but where it throws and the enumeration order of results would change; that needs checking. The compact key must be exact: a hash-only visited set can lose a parse on collision and is not proposed.

Reference implementation: sillsdev/PanGloss#249 and #250 cover the sharing and alternative-storage parts; the per-stratum finishing is the planned next step there.

Evidence (PanGloss, Rust)

Word mindeshi, Mbugwe grammar, peak resident set, one thread. Linear growth with work, about 14 MB per million steps before any fix. With shape and syntactic feature-structure sharing plus trail-stored alternatives (#249, #250): 50M cap 199 to 59 MB, 200M cap 1,152 to 267 MB, 1G cap 5,074 to 774 MB. Remaining memory is one word per canonical state, which is what this issue targets. At a 200M cap the search held 379,394 words: 94,010 canonical, 285,384 folded alternatives.

Where C# retains states (origin/master)
  • AnalysisLanguageRule.cs:26, :44 results accumulates every stratum's output; :24-25 input and temp sets.
  • AnalysisStratumRule.cs:150 per-stratum output HashSet; :140-144, :177-178 merge-key cache wordCache of canonicals.
  • Morpher.cs:186 new ConcurrentQueue<Word>(_analysisRule.Apply(input)) materialises all analyses; synthesis starts at :200.
  • Morpher.cs:97 MaxAlternatives (throws, AnalysisStratumRule.cs:156-159); no byte accounting.
  • AnalysisStratumRule.cs:142 merging (and so alternatives) is skipped while tracing, so a traced parse already holds every state; see #565.
  • A fix touches Morpher.Synthesize* (:345-420), which also has a parallel path using the queue count.

Next: recorded for later, not scheduled. Measure C# peak working set on a heavy word (for example Mbugwe mindeshi) after #593 and #595, to see how much remains.

主要言語
C#
スター
29
フォーク
16
平均マージ
4日 11時間
マージ済み PR（30日）
14

環境構築

  • Dockerfile・Docker Compose ファイルなし
  • プルリクエストのテンプレートあり
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

sillsdev/machine のほかの issue

sillsdev/machine の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。