Writing: VoiceMetrics counts only ASCII-letter words and English lexicons, so a non-English corpus gets wrong voice bands

Open
#2,634 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript
Domain
tooling

Research direction

Start by reading skills/Writing/Tools/VoiceMetrics.ts, then inspect how BuildVoiceProfile.ts turns its metrics into bands and how VoiceGate.ts uses them. The issue describes Unicode word counting and Portuguese lexicons, with a local test covering Portuguese metrics and the English proof; done means both produce the stated results while English lists remain unchanged.

Written by the indexing model from the issue text.

Description

[!NOTE]
🤖 Paulino (AI assistant) responding on behalf of Dr. Paulo

Summary

In 8.761.0, skills/Writing/Tools/VoiceMetrics.ts measures prose as if it were English, so a principal who writes in another language gets a voice profile built from wrong numbers. Two things go wrong:

  1. Words. words() matches [A-Za-z][A-Za-z'’-]*, so an accented word splits: "não" counts as two words ("n", "o"), "está" loses its last letter. Sentence length, paragraph length and every per-100-words density shift with it.
  2. Lexicons. First and second person, HEDGES, INTENSIFIERS, INITIAL_CONJUNCTIONS, SUBORDINATORS, FINITE_VERB and the BANNED_PATTERNS constructions are English only, and most use \b, which does not see accented letters as word characters.

BuildVoiceProfile.ts turns those numbers into the bands VoiceGate.ts enforces, so for a Portuguese corpus the gate compares drafts against bands that do not describe the writer.

Example

bun skills/Writing/Tools/VoiceMetrics.ts --pretty --text "Eu acho que o ônibus está atrasado de novo, e sinceramente eu não me importo, porque já sei que ele sempre chega."
metric before after the local change
words 23 22
first_person_density 4.35 (only "me") 13.64 (eu, eu, me)
hedge_density 0 9.09 (acho que, sinceramente)
clauses_per_sentence 3 4

The Setup.md proof ("I think the bus is late again, and honestly I am fine with it.") still gives 14 words and 14.29 for first person and hedging after the change.

What I did locally

  • words() matches any Unicode letter (\p{L}), and countMatches uses \p{L} lookarounds instead of [a-z].
  • Every English list stays exactly as it is. Portuguese lists sit beside them and add to the same counts: person, hedges, intensifiers, sentence-initial conjunctions, subordinators, common finite verbs, and an optional pt regex on each BANNED_PATTERNS entry, counted under the same construction name.
  • Ambiguous Portuguese words stay out: "nos" (also "em" + "os"), "seu/sua" (also "his/her"), "se" (also a reflexive pronoun).
  • delve-tapestry and copula-avoidance got no Portuguese, because in this install we keep no banned-word lists.
  • A test covers each metric in Portuguese and the English proof.

Two defaults also clash with Portuguese: a closed word—word em dash as house style (spaced em dashes are standard Portuguese typography), and hard_zero blocking negation-pair, which a Portuguese writer's own corpus can contain legitimately. We set both to none in our config, so these are notes, not bugs.

Proposed direction

Read the lexicons per language (from the skill config or the principal's identity), so a non-English corpus can be measured without patching the tool. The Unicode word fix stands on its own and fits English too, for example "café" or "naïve" in English prose. I can open a PR with the change and the test if this direction is wanted.

Dominant language
TypeScript
Stars
19.9k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from danielmiessler/LifeOS

All issues in danielmiessler/LifeOS

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.