Writing: VoiceMetrics counts only ASCII-letter words and English lexicons, so a non-English corpus gets wrong voice bands

Aperta
#2,634 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
typescript
Ambito
tooling

Direzione di ricerca

Start by reading skills/Writing/Tools/VoiceMetrics.ts, then inspect how BuildVoiceProfile.ts turns its metrics into bands and how VoiceGate.ts uses them. The issue describes Unicode word counting and Portuguese lexicons, with a local test covering Portuguese metrics and the English proof; done means both produce the stated results while English lists remain unchanged.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

[!NOTE]
🤖 Paulino (AI assistant) responding on behalf of Dr. Paulo

Summary

In 8.761.0, skills/Writing/Tools/VoiceMetrics.ts measures prose as if it were English, so a principal who writes in another language gets a voice profile built from wrong numbers. Two things go wrong:

  1. Words. words() matches [A-Za-z][A-Za-z'’-]*, so an accented word splits: "não" counts as two words ("n", "o"), "está" loses its last letter. Sentence length, paragraph length and every per-100-words density shift with it.
  2. Lexicons. First and second person, HEDGES, INTENSIFIERS, INITIAL_CONJUNCTIONS, SUBORDINATORS, FINITE_VERB and the BANNED_PATTERNS constructions are English only, and most use \b, which does not see accented letters as word characters.

BuildVoiceProfile.ts turns those numbers into the bands VoiceGate.ts enforces, so for a Portuguese corpus the gate compares drafts against bands that do not describe the writer.

Example

bun skills/Writing/Tools/VoiceMetrics.ts --pretty --text "Eu acho que o ônibus está atrasado de novo, e sinceramente eu não me importo, porque já sei que ele sempre chega."
metric before after the local change
words 23 22
first_person_density 4.35 (only "me") 13.64 (eu, eu, me)
hedge_density 0 9.09 (acho que, sinceramente)
clauses_per_sentence 3 4

The Setup.md proof ("I think the bus is late again, and honestly I am fine with it.") still gives 14 words and 14.29 for first person and hedging after the change.

What I did locally

  • words() matches any Unicode letter (\p{L}), and countMatches uses \p{L} lookarounds instead of [a-z].
  • Every English list stays exactly as it is. Portuguese lists sit beside them and add to the same counts: person, hedges, intensifiers, sentence-initial conjunctions, subordinators, common finite verbs, and an optional pt regex on each BANNED_PATTERNS entry, counted under the same construction name.
  • Ambiguous Portuguese words stay out: "nos" (also "em" + "os"), "seu/sua" (also "his/her"), "se" (also a reflexive pronoun).
  • delve-tapestry and copula-avoidance got no Portuguese, because in this install we keep no banned-word lists.
  • A test covers each metric in Portuguese and the English proof.

Two defaults also clash with Portuguese: a closed word—word em dash as house style (spaced em dashes are standard Portuguese typography), and hard_zero blocking negation-pair, which a Portuguese writer's own corpus can contain legitimately. We set both to none in our config, so these are notes, not bugs.

Proposed direction

Read the lexicons per language (from the skill config or the principal's identity), so a non-English corpus can be measured without patching the tool. The Unicode word fix stands on its own and fits English too, for example "café" or "naïve" in English prose. I can open a PR with the change and the test if this direction is wanted.

Lingua principale
TypeScript
Stelle
19.9k
Fork
2.5k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di danielmiessler/LifeOS

Tutte le issue di danielmiessler/LifeOS

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.