Multilingual typing: committed words are learned in the user history of the wrong language

Chiusa
  • Qualcosa non va?
#763 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
62/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
android, kotlin
Ambito
internationalization, mobile

Direzione di ricerca

Start in DictionaryFacilitatorImpl, where addToUserHistory() reads currentlyPreferredDictionaryGroup before adjustConfidences() and then calls addWordToUserHistory with that group. Check unlearnFromUserHistory() as well, since it only touches the preferred group's history. Done when a German word typed after English words is stored in the de_DE history, and unlearnFromUserHistory() clears the word from every group that holds it. The reporter's groupForLearning idea is a starting point to verify, not a finished patch.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug
With multilingual typing, DictionaryFacilitatorImpl.addToUserHistory() always adds the word to the user history of currentlyPreferredDictionaryGroup. It even reads that group before adjustConfidences() runs for the word:

val preferredGroup = currentlyPreferredDictionaryGroup   // read here
...
adjustConfidences(suggestion, wasAutoCapitalized)        // confidence changes afterwards
...
// add to history for preferred dictionary group, to avoid mixing languages in history
addWordToUserHistory(preferredGroup, ngramContextForCurrentWord, currentWord, ...)

So after a few English words, a German word, typed or picked from the suggestions, goes into the English user history. There:

  • the English main dictionary doesn't know it (mainFreq = -1), so it is stored as an invalid word and needs several uses before it's suggested;
  • it is scored with the English locale weight;
  • the bigram (previous word → German word) is stored in the English history too.

This is the opposite of what the comment intends ("to avoid mixing languages in history"). It also means that correcting a wrongly swiped word (e.g. German "hier" that came out as English "hitter") barely helps next time.

unlearnFromUserHistory() has the same issue: it only removes the word from the preferred group's history.

Log from our build (German main, English (US) secondary, with a temporary debug line in addToUserHistory). After "the weather is good and I think that", the user types "hier":

learn 'that' -> en_US, preferred en_US, confidences [de_DE=0, en_US=8]
learn 'hier' -> de_DE, preferred en_US, confidences [de_DE=1, en_US=2]

With the current LeanType code, "hier" goes to en_US, because that group is preferred. With our fix it goes to de_DE.

Fix that works for us
Pick the history by the word's language: the preferred group if the word belongs to it (or to no language), otherwise the group whose main/apps/personal dictionary contains it:

private fun groupForLearning(word: String): DictionaryGroup {
    val preferred = currentlyPreferredDictionaryGroup
    if (dictionaryGroups.size == 1) return preferred
    fun belongsTo(group: DictionaryGroup) = isValidWord(word, DICTIONARY_TYPES_EXCLUDING_HISTORY, group)
        || isValidWord(word.lowercase(group.locale), DICTIONARY_TYPES_EXCLUDING_HISTORY, group)
    if (belongsTo(preferred)) return preferred
    return dictionaryGroups.filter { it !== preferred && belongsTo(it) }.maxByOrNull { it.confidence } ?: preferred
}

Use it in addToUserHistory() (per word, after adjustConfidences()), and let unlearnFromUserHistory() remove the word from the history of all groups. I can send a PR once we've tested it a bit longer.

To Reproduce

  1. Languages & Layouts → German → Multilingual typing → add English (US). Main dictionaries for both, personalized suggestions on.
  2. Type a few English words, then a German word that is not English (e.g. "hier").
  3. The word is added to the English user history (visible with a debug log, or via the dictionary dump in the debug settings).

Expected behavior
Words are learned in the user history of their own language.

App version & Flavor
Found in GlassType, my fork, now based on LeanType 4.3.1 (0e673e1). The code is unchanged in current main.

Device Info

  • Model: Samsung Galaxy Z Fold 3 (SM-F926B)
  • OS version: Android 15
Lingua principale
Kotlin
Stelle
1k
Fork
59
Merge medio
14h 9m
PR unite (30g)
56

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di LeanBitLab/LeanType

Tutte le issue di LeanBitLab/LeanType

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.