Writing: VoiceMetrics counts only ASCII-letter words and English lexicons, so a non-English corpus gets wrong voice bands

Abierto
#2,634 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 5 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
typescript
Área
tooling

Línea de trabajo

Start by reading skills/Writing/Tools/VoiceMetrics.ts, then inspect how BuildVoiceProfile.ts turns its metrics into bands and how VoiceGate.ts uses them. The issue describes Unicode word counting and Portuguese lexicons, with a local test covering Portuguese metrics and the English proof; done means both produce the stated results while English lists remain unchanged.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

[!NOTE]
🤖 Paulino (AI assistant) responding on behalf of Dr. Paulo

Summary

In 8.761.0, skills/Writing/Tools/VoiceMetrics.ts measures prose as if it were English, so a principal who writes in another language gets a voice profile built from wrong numbers. Two things go wrong:

  1. Words. words() matches [A-Za-z][A-Za-z'’-]*, so an accented word splits: "não" counts as two words ("n", "o"), "está" loses its last letter. Sentence length, paragraph length and every per-100-words density shift with it.
  2. Lexicons. First and second person, HEDGES, INTENSIFIERS, INITIAL_CONJUNCTIONS, SUBORDINATORS, FINITE_VERB and the BANNED_PATTERNS constructions are English only, and most use \b, which does not see accented letters as word characters.

BuildVoiceProfile.ts turns those numbers into the bands VoiceGate.ts enforces, so for a Portuguese corpus the gate compares drafts against bands that do not describe the writer.

Example

bun skills/Writing/Tools/VoiceMetrics.ts --pretty --text "Eu acho que o ônibus está atrasado de novo, e sinceramente eu não me importo, porque já sei que ele sempre chega."
metric before after the local change
words 23 22
first_person_density 4.35 (only "me") 13.64 (eu, eu, me)
hedge_density 0 9.09 (acho que, sinceramente)
clauses_per_sentence 3 4

The Setup.md proof ("I think the bus is late again, and honestly I am fine with it.") still gives 14 words and 14.29 for first person and hedging after the change.

What I did locally

  • words() matches any Unicode letter (\p{L}), and countMatches uses \p{L} lookarounds instead of [a-z].
  • Every English list stays exactly as it is. Portuguese lists sit beside them and add to the same counts: person, hedges, intensifiers, sentence-initial conjunctions, subordinators, common finite verbs, and an optional pt regex on each BANNED_PATTERNS entry, counted under the same construction name.
  • Ambiguous Portuguese words stay out: "nos" (also "em" + "os"), "seu/sua" (also "his/her"), "se" (also a reflexive pronoun).
  • delve-tapestry and copula-avoidance got no Portuguese, because in this install we keep no banned-word lists.
  • A test covers each metric in Portuguese and the English proof.

Two defaults also clash with Portuguese: a closed word—word em dash as house style (spaced em dashes are standard Portuguese typography), and hard_zero blocking negation-pair, which a Portuguese writer's own corpus can contain legitimately. We set both to none in our config, so these are notes, not bugs.

Proposed direction

Read the lexicons per language (from the skill config or the principal's identity), so a non-English corpus can be measured without patching the tool. The Unicode word fix stands on its own and fits English too, for example "café" or "naïve" in English prose. I can open a PR with the change and the test if this direction is wanted.

Lenguaje dominante
TypeScript
Estrellas
19.9k
Forks
2.5k
Métricas de merge de PR
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de danielmiessler/LifeOS

Todos los issues de danielmiessler/LifeOS

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.