Multilingual typing: committed words are learned in the user history of the wrong language

Aberta
  • Algo errado?
#763 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
62/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
android, kotlin
Domínio
internationalization, mobile

Direção de pesquisa

Start in DictionaryFacilitatorImpl, where addToUserHistory() reads currentlyPreferredDictionaryGroup before adjustConfidences() and then calls addWordToUserHistory with that group. Check unlearnFromUserHistory() as well, since it only touches the preferred group's history. Done when a German word typed after English words is stored in the de_DE history, and unlearnFromUserHistory() clears the word from every group that holds it. The reporter's groupForLearning idea is a starting point to verify, not a finished patch.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Describe the bug
With multilingual typing, DictionaryFacilitatorImpl.addToUserHistory() always adds the word to the user history of currentlyPreferredDictionaryGroup. It even reads that group before adjustConfidences() runs for the word:

val preferredGroup = currentlyPreferredDictionaryGroup   // read here
...
adjustConfidences(suggestion, wasAutoCapitalized)        // confidence changes afterwards
...
// add to history for preferred dictionary group, to avoid mixing languages in history
addWordToUserHistory(preferredGroup, ngramContextForCurrentWord, currentWord, ...)

So after a few English words, a German word, typed or picked from the suggestions, goes into the English user history. There:

  • the English main dictionary doesn't know it (mainFreq = -1), so it is stored as an invalid word and needs several uses before it's suggested;
  • it is scored with the English locale weight;
  • the bigram (previous word → German word) is stored in the English history too.

This is the opposite of what the comment intends ("to avoid mixing languages in history"). It also means that correcting a wrongly swiped word (e.g. German "hier" that came out as English "hitter") barely helps next time.

unlearnFromUserHistory() has the same issue: it only removes the word from the preferred group's history.

Log from our build (German main, English (US) secondary, with a temporary debug line in addToUserHistory). After "the weather is good and I think that", the user types "hier":

learn 'that' -> en_US, preferred en_US, confidences [de_DE=0, en_US=8]
learn 'hier' -> de_DE, preferred en_US, confidences [de_DE=1, en_US=2]

With the current LeanType code, "hier" goes to en_US, because that group is preferred. With our fix it goes to de_DE.

Fix that works for us
Pick the history by the word's language: the preferred group if the word belongs to it (or to no language), otherwise the group whose main/apps/personal dictionary contains it:

private fun groupForLearning(word: String): DictionaryGroup {
    val preferred = currentlyPreferredDictionaryGroup
    if (dictionaryGroups.size == 1) return preferred
    fun belongsTo(group: DictionaryGroup) = isValidWord(word, DICTIONARY_TYPES_EXCLUDING_HISTORY, group)
        || isValidWord(word.lowercase(group.locale), DICTIONARY_TYPES_EXCLUDING_HISTORY, group)
    if (belongsTo(preferred)) return preferred
    return dictionaryGroups.filter { it !== preferred && belongsTo(it) }.maxByOrNull { it.confidence } ?: preferred
}

Use it in addToUserHistory() (per word, after adjustConfidences()), and let unlearnFromUserHistory() remove the word from the history of all groups. I can send a PR once we've tested it a bit longer.

To Reproduce

  1. Languages & Layouts → German → Multilingual typing → add English (US). Main dictionaries for both, personalized suggestions on.
  2. Type a few English words, then a German word that is not English (e.g. "hier").
  3. The word is added to the English user history (visible with a debug log, or via the dictionary dump in the debug settings).

Expected behavior
Words are learned in the user history of their own language.

App version & Flavor
Found in GlassType, my fork, now based on LeanType 4.3.1 (0e673e1). The code is unchanged in current main.

Device Info

  • Model: Samsung Galaxy Z Fold 3 (SM-F926B)
  • OS version: Android 15
Linguagem predominante
Kotlin
Estrelas
1k
Forks
59
Merge médio
14h 9min
PRs com merge (30d)
56

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de LeanBitLab/LeanType

Todas as issues de LeanBitLab/LeanType

Issues semelhantes

Mais issues de Kotlin

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.