Multilingual typing: committed words are learned in the user history of the wrong language

Cerrado
  • ¿Algo no está bien?
#763 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
62/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
android, kotlin
Área
internationalization, mobile

Línea de trabajo

Start in DictionaryFacilitatorImpl, where addToUserHistory() reads currentlyPreferredDictionaryGroup before adjustConfidences() and then calls addWordToUserHistory with that group. Check unlearnFromUserHistory() as well, since it only touches the preferred group's history. Done when a German word typed after English words is stored in the de_DE history, and unlearnFromUserHistory() clears the word from every group that holds it. The reporter's groupForLearning idea is a starting point to verify, not a finished patch.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug
With multilingual typing, DictionaryFacilitatorImpl.addToUserHistory() always adds the word to the user history of currentlyPreferredDictionaryGroup. It even reads that group before adjustConfidences() runs for the word:

val preferredGroup = currentlyPreferredDictionaryGroup   // read here
...
adjustConfidences(suggestion, wasAutoCapitalized)        // confidence changes afterwards
...
// add to history for preferred dictionary group, to avoid mixing languages in history
addWordToUserHistory(preferredGroup, ngramContextForCurrentWord, currentWord, ...)

So after a few English words, a German word, typed or picked from the suggestions, goes into the English user history. There:

  • the English main dictionary doesn't know it (mainFreq = -1), so it is stored as an invalid word and needs several uses before it's suggested;
  • it is scored with the English locale weight;
  • the bigram (previous word → German word) is stored in the English history too.

This is the opposite of what the comment intends ("to avoid mixing languages in history"). It also means that correcting a wrongly swiped word (e.g. German "hier" that came out as English "hitter") barely helps next time.

unlearnFromUserHistory() has the same issue: it only removes the word from the preferred group's history.

Log from our build (German main, English (US) secondary, with a temporary debug line in addToUserHistory). After "the weather is good and I think that", the user types "hier":

learn 'that' -> en_US, preferred en_US, confidences [de_DE=0, en_US=8]
learn 'hier' -> de_DE, preferred en_US, confidences [de_DE=1, en_US=2]

With the current LeanType code, "hier" goes to en_US, because that group is preferred. With our fix it goes to de_DE.

Fix that works for us
Pick the history by the word's language: the preferred group if the word belongs to it (or to no language), otherwise the group whose main/apps/personal dictionary contains it:

private fun groupForLearning(word: String): DictionaryGroup {
    val preferred = currentlyPreferredDictionaryGroup
    if (dictionaryGroups.size == 1) return preferred
    fun belongsTo(group: DictionaryGroup) = isValidWord(word, DICTIONARY_TYPES_EXCLUDING_HISTORY, group)
        || isValidWord(word.lowercase(group.locale), DICTIONARY_TYPES_EXCLUDING_HISTORY, group)
    if (belongsTo(preferred)) return preferred
    return dictionaryGroups.filter { it !== preferred && belongsTo(it) }.maxByOrNull { it.confidence } ?: preferred
}

Use it in addToUserHistory() (per word, after adjustConfidences()), and let unlearnFromUserHistory() remove the word from the history of all groups. I can send a PR once we've tested it a bit longer.

To Reproduce

  1. Languages & Layouts → German → Multilingual typing → add English (US). Main dictionaries for both, personalized suggestions on.
  2. Type a few English words, then a German word that is not English (e.g. "hier").
  3. The word is added to the English user history (visible with a debug log, or via the dictionary dump in the debug settings).

Expected behavior
Words are learned in the user history of their own language.

App version & Flavor
Found in GlassType, my fork, now based on LeanType 4.3.1 (0e673e1). The code is unchanged in current main.

Device Info

  • Model: Samsung Galaxy Z Fold 3 (SM-F926B)
  • OS version: Android 15
Lenguaje dominante
Kotlin
Estrellas
1k
Forks
59
Merge medio
14 h 9 min
PR fusionados (30 d)
56

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de LeanBitLab/LeanType

Todos los issues de LeanBitLab/LeanType

Issues similares

Más issues de Kotlin

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.