Multilingual typing: committed words are learned in the user history of the wrong language

Offen
  • Stimmt etwas nicht?
#763 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
62/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
android, kotlin
Bereich
internationalization, mobile

Rechercherichtung

Start in DictionaryFacilitatorImpl, where addToUserHistory() reads currentlyPreferredDictionaryGroup before adjustConfidences() and then calls addWordToUserHistory with that group. Check unlearnFromUserHistory() as well, since it only touches the preferred group's history. Done when a German word typed after English words is stored in the de_DE history, and unlearnFromUserHistory() clears the word from every group that holds it. The reporter's groupForLearning idea is a starting point to verify, not a finished patch.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Describe the bug
With multilingual typing, DictionaryFacilitatorImpl.addToUserHistory() always adds the word to the user history of currentlyPreferredDictionaryGroup. It even reads that group before adjustConfidences() runs for the word:

val preferredGroup = currentlyPreferredDictionaryGroup   // read here
...
adjustConfidences(suggestion, wasAutoCapitalized)        // confidence changes afterwards
...
// add to history for preferred dictionary group, to avoid mixing languages in history
addWordToUserHistory(preferredGroup, ngramContextForCurrentWord, currentWord, ...)

So after a few English words, a German word, typed or picked from the suggestions, goes into the English user history. There:

  • the English main dictionary doesn't know it (mainFreq = -1), so it is stored as an invalid word and needs several uses before it's suggested;
  • it is scored with the English locale weight;
  • the bigram (previous word → German word) is stored in the English history too.

This is the opposite of what the comment intends ("to avoid mixing languages in history"). It also means that correcting a wrongly swiped word (e.g. German "hier" that came out as English "hitter") barely helps next time.

unlearnFromUserHistory() has the same issue: it only removes the word from the preferred group's history.

Log from our build (German main, English (US) secondary, with a temporary debug line in addToUserHistory). After "the weather is good and I think that", the user types "hier":

learn 'that' -> en_US, preferred en_US, confidences [de_DE=0, en_US=8]
learn 'hier' -> de_DE, preferred en_US, confidences [de_DE=1, en_US=2]

With the current LeanType code, "hier" goes to en_US, because that group is preferred. With our fix it goes to de_DE.

Fix that works for us
Pick the history by the word's language: the preferred group if the word belongs to it (or to no language), otherwise the group whose main/apps/personal dictionary contains it:

private fun groupForLearning(word: String): DictionaryGroup {
    val preferred = currentlyPreferredDictionaryGroup
    if (dictionaryGroups.size == 1) return preferred
    fun belongsTo(group: DictionaryGroup) = isValidWord(word, DICTIONARY_TYPES_EXCLUDING_HISTORY, group)
        || isValidWord(word.lowercase(group.locale), DICTIONARY_TYPES_EXCLUDING_HISTORY, group)
    if (belongsTo(preferred)) return preferred
    return dictionaryGroups.filter { it !== preferred && belongsTo(it) }.maxByOrNull { it.confidence } ?: preferred
}

Use it in addToUserHistory() (per word, after adjustConfidences()), and let unlearnFromUserHistory() remove the word from the history of all groups. I can send a PR once we've tested it a bit longer.

To Reproduce

  1. Languages & Layouts → German → Multilingual typing → add English (US). Main dictionaries for both, personalized suggestions on.
  2. Type a few English words, then a German word that is not English (e.g. "hier").
  3. The word is added to the English user history (visible with a debug log, or via the dictionary dump in the debug settings).

Expected behavior
Words are learned in the user history of their own language.

App version & Flavor
Found in GlassType, my fork, now based on LeanType 4.3.1 (0e673e1). The code is unchanged in current main.

Device Info

  • Model: Samsung Galaxy Z Fold 3 (SM-F926B)
  • OS version: Android 15
Vorherrschende Sprache
Kotlin
Sterne
1k
Forks
59
Ø Merge
14 Std. 9 Min.
Gemergte PRs (30 T.)
56

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus LeanBitLab/LeanType

Alle Issues in LeanBitLab/LeanType

Ähnliche Issues

Weitere Issues zu Kotlin

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.