HermitCrab analysis: finish each stratum as it completes instead of keeping every state until synthesis (per-word memory)

Aperta
  • Qualcosa non va?
#639 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
backend, performance

Direzione di ricerca

Read AnalysisLanguageRule.cs and Morpher.cs to trace how stratum results reach synthesis, then inspect AnalysisStratumRule.cs for state merging and MaxAlternatives behavior. Measure peak working set on a heavy word such as Mbugwe mindeshi after #593 and #595. Done means reducing how many full states remain live while preserving uncapped analyses; verify capped-word exceptions, result enumeration order, and tracing behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hermit Crab performance

HermitCrab analysis keeps every word of every stratum alive until the whole analysis ends, so per-word memory grows linearly with the number of distinct states reached. AnalysisLanguageRule.Apply adds each stratum's outputs to one results set, Morpher.ParseWord copies that into a queue, and only then does synthesis start. The only bound is MaxAlternatives, which counts alternatives, not bytes.

Recorded for later, in the same family as #593 and #595 (which shrink each word; this one is about how many are live at once). C# numbers for this mechanism do not exist yet; the numbers below are PanGloss ones. The C# structure is the same: breadth-first stratum search, a deduplicating output set, equivalents folded into a canonical.

Idea (the PanGloss approach, owner-approved there): finish each stratum's results as the stratum completes (hand its canonicals to synthesis, drop them once the next stratum has consumed them), keep a compact exact visited key per state instead of the whole word, and keep the full search tree only when tracing (Try a Word).

Trade-off: uncapped words give identical analyses. A capped word's partial result changes in PanGloss because work is spent in a different order. In C# the cap throws MaxAlternativesExceededException rather than returning partials, so the outcome is expected to stay "throws", but where it throws and the enumeration order of results would change; that needs checking. The compact key must be exact: a hash-only visited set can lose a parse on collision and is not proposed.

Reference implementation: sillsdev/PanGloss#249 and #250 cover the sharing and alternative-storage parts; the per-stratum finishing is the planned next step there.

Evidence (PanGloss, Rust)

Word mindeshi, Mbugwe grammar, peak resident set, one thread. Linear growth with work, about 14 MB per million steps before any fix. With shape and syntactic feature-structure sharing plus trail-stored alternatives (#249, #250): 50M cap 199 to 59 MB, 200M cap 1,152 to 267 MB, 1G cap 5,074 to 774 MB. Remaining memory is one word per canonical state, which is what this issue targets. At a 200M cap the search held 379,394 words: 94,010 canonical, 285,384 folded alternatives.

Where C# retains states (origin/master)
  • AnalysisLanguageRule.cs:26, :44 results accumulates every stratum's output; :24-25 input and temp sets.
  • AnalysisStratumRule.cs:150 per-stratum output HashSet; :140-144, :177-178 merge-key cache wordCache of canonicals.
  • Morpher.cs:186 new ConcurrentQueue<Word>(_analysisRule.Apply(input)) materialises all analyses; synthesis starts at :200.
  • Morpher.cs:97 MaxAlternatives (throws, AnalysisStratumRule.cs:156-159); no byte accounting.
  • AnalysisStratumRule.cs:142 merging (and so alternatives) is skipped while tracing, so a traced parse already holds every state; see #565.
  • A fix touches Morpher.Synthesize* (:345-420), which also has a parallel path using the queue count.

Next: recorded for later, not scheduled. Measure C# peak working set on a heavy word (for example Mbugwe mindeshi) after #593 and #595, to see how much remains.

Lingua principale
C#
Stelle
29
Fork
16
Merge medio
4g 11h
PR unite (30g)
14

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di sillsdev/machine

Tutte le issue di sillsdev/machine

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.