Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

[Android] LlmGenerationConfig default temperature (0.8) overrides LlmModule temperature in the config generate overload

Ouverte
  • Quelque chose ne va pas ?
#23,263 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
55/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
android, cpp, kotlin
Domaine
ai, mobile

Piste de recherche

Start with LlmGenerationConfig.kt and LlmModule.kt, especially the builder default and generate overloads, then trace the temperature handling in jni_layer_llama.cpp. Run the ReproAB.kt program to confirm the differing A-D behavior. Done means the chosen temperature precedence is consistent across overloads and is reflected in docs/source/llm/run-on-android.md.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

🐛 Describe the bug
Summary

On Android, temperature can be set on the LlmModule constructor and on LlmGenerationConfig. The native layer uses a per-call temperature when it is >= 0 and otherwise falls back to the constructor's (#16728). The simple generate(...) overloads pass -1.0f, so they use the constructor's value. LlmGenerationConfig.Builder defaults temperature to 0.8f, so generate(prompt, config, callback) replaces the constructor's value unless the config sets temperature explicitly.

  • Expected: LlmModule(..., temperature = 0.0f) decodes greedily whichever generate overload is used.
  • Actual: through the config overload it samples at 0.8 unless .temperature(...) is set on the config.
To reproduce

Environment: executorch-android 1.4.0 from Maven Central (the latest published), with fbjni 0.7.0 and nativeloader 0.10.5; the test program also uses kotlin-stdlib 2.3.20. Models (both attention only, so #23262 cannot affect them):

The module is built for greedy decoding, then each way of calling generate runs twice from resetContext():

val module = LlmModule(LlmModule.MODEL_TYPE_TEXT, modelPath, tokenizerPath, 0.0f)   // greedy intended
module.load()
fun config() = LlmGenerationConfig.create().seqLen(160).echo(false)

// A: config, temperature not set
module.generate(prompt, config().build(), callback)
// B: config, temperature set to 0
module.generate(prompt, config().temperature(0.0f).build(), callback)
// C: config, temperature set to -1 ("use the module's")
module.generate(prompt, config().temperature(-1.0f).build(), callback)
// D: simple overload
module.generate(prompt, 160, callback, false)

Were the two runs identical?

Call Qwen3, POCO X8 Pro Max Gemma3, POCO X8 Pro Max Qwen3, Android emulator
A: config, temperature not set no no no
B: config, temperature(0.0f) yes yes yes
C: config, temperature(-1.0f) yes yes yes
D: simple overload yes yes yes

Devices: Xiaomi POCO X8 Pro Max (MT6991, Android 16) and an Android 16 emulator (android-36 google_apis arm64-v8a image) on an Apple silicon Mac. On the phone the Qwen3 A/B ran twice with the same result, and an earlier version comparing only A and D gave the same result in two more runs. Example from A (Qwen3, phone):

run 1: In the quiet town of Cape Cod, where the sea whispered secrets to the
run 2: In the quiet coastal town of Elmhurst, there stood a lighthouse that h

Within each B, C and D pair, the two runs are identical, and the first 70 characters are also the same across B, C and D on each device. Runs under A start with the same first word and diverge shortly after, because the first token is chosen during prefill with temperature 0 (text_prefiller.cpp#L32, called at text_llm_runner.cpp#L181) and config.temperature only applies to the decode loop (text_llm_runner.cpp#L240-L245). That makes the problem easy to mistake for small numerical noise.

Code path

Permalinks at 2f78245. The cited lines are the same at v1.5.1, and javap on the 1.4.0 AAR shows the same Kotlin behaviour (builder default 0.8f, config overload passes the config's temperature, simple overloads pass -1.0f).

run-on-android.md#L173-L197 shows a config with .temperature(0.8f) and says the defaults match the C++ GenerationConfig, but does not say that a config temperature replaces the one given to the constructor.

Suggested fix

Either:

  1. Default LlmGenerationConfig.Builder.temperature to -1.0f ("use the module's"), matching the simple overloads. Variant C above shows this restores the constructor's temperature. It changes behaviour for anyone relying on the 0.8 default while passing a different constructor value, so it is worth a release note.
  2. Keep 0.8 and document the precedence on the LlmModule constructor, on LlmGenerationConfig.temperature and in run-on-android.md.

#17637 (consolidating the LlmModule overloads) may be a natural place to settle this. Scope: the generic text path (MODEL_TYPE_TEXT to TextLLMRunner); I have not checked the multimodal or vendor runners.

Full program

ReproAB.kt, verbatim. Built against the AAR's classes.jar and run with app_process, so no app is needed. Commands as run, from a directory holding ReproAB.kt, deps/ (the AAR classes.jar files and nativeloader-0.10.5.jar) and kotlinc/:

kotlinc ReproAB.kt \
  -cp deps/et/classes.jar:deps/fbjni/classes.jar:deps/nativeloader-0.10.5.jar:$ANDROID_HOME/platforms/android-36/android.jar \
  -jvm-target 11 -d reproab-classes.jar
d8 --release --min-api 26 --lib $ANDROID_HOME/platforms/android-36/android.jar --output ab \
  reproab-classes.jar kotlinc/lib/kotlin-stdlib.jar deps/et/classes.jar deps/fbjni/classes.jar deps/nativeloader-0.10.5.jar
# Push ab/classes.dex as /data/local/tmp/et-temp-repro/reproab.dex, plus the model files, plus lib/ with
# libexecutorch.so (AAR jni/arm64-v8a) and libfbjni.so, libc++_shared.so (fbjni AAR).
adb shell "cd /data/local/tmp/et-temp-repro && LD_LIBRARY_PATH=/data/local/tmp/et-temp-repro/lib \
  CLASSPATH=/data/local/tmp/et-temp-repro/reproab.dex app_process -Djava.library.path=/data/local/tmp/et-temp-repro/lib \
  /data/local/tmp/et-temp-repro ReproABKt \
  /data/local/tmp/et-temp-repro/qwen3.pte /data/local/tmp/et-temp-repro/qwen3.tokenizer.json qwen3"   # gemma: its files, "gemma"

// Issue-2 A/B: which way of calling generate() samples when LlmModule was built with temperature 0?
import org.pytorch.executorch.extension.llm.LlmCallback
import org.pytorch.executorch.extension.llm.LlmGenerationConfig
import org.pytorch.executorch.extension.llm.LlmModule
import kotlin.system.exitProcess

fun collect(block: (LlmCallback) -> Unit): String {
    val sb = StringBuilder()
    block(object : LlmCallback { override fun onResult(result: String) { sb.append(result) } })
    return sb.toString()
}

fun main(args: Array<String>) {
    val (modelPath, tokenizerPath, family) = Triple(args[0], args[1], args[2])
    val q = "Tell me a short story about a lighthouse keeper."
    val prompt = when (family) {
        "gemma" -> "<bos><start_of_turn>user\n$q<end_of_turn>\n<start_of_turn>model\n"
        else -> "<|im_start|>user\n$q<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
    }
    val module = LlmModule(LlmModule.MODEL_TYPE_TEXT, modelPath, tokenizerPath, 0.0f)   // greedy intended
    module.load()
    val seqLen = 160
    val variants = listOf<Pair<String, (LlmCallback) -> Unit>>(
        "A config, temperature not set" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).build(), cb) },
        "B config, temperature(0.0f)" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).temperature(0.0f).build(), cb) },
        "C config, temperature(-1.0f)" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).temperature(-1.0f).build(), cb) },
        "D simple overload" to { cb -> module.generate(prompt, seqLen, cb, false) },
    )
    val out = StringBuilder()
    for ((name, gen) in variants) {
        module.resetContext(); val r1 = collect(gen)
        module.resetContext(); val r2 = collect(gen)
        out.append("RESULT $family | $name | identical: ${r1 == r2}\n")
        out.append("   run 1: ${r1.take(70).replace("\n", "\\n")}\n")
        out.append("   run 2: ${r2.take(70).replace("\n", "\\n")}\n")
    }
    print(out)
    System.out.flush()
    exitProcess(0)
}
Versions

executorch-android 1.4.0 (Maven Central), fbjni 0.7.0, nativeloader 0.10.5, kotlin-stdlib 2.3.20 (test program). Source checked at main 2f7824593d0d14f9d5d73faa540b681b6b2b507c and v1.5.1. Devices: Xiaomi POCO X8 Pro Max (MT6991, Android 16); Android 16 emulator (android-36 google_apis arm64-v8a) on an Apple silicon Mac.

Langage dominant
Python
Étoiles
5k
Forks
1.2k
Merge moyen
2 j 7 h
PR mergées (30 j)
521

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de pytorch/executorch

Toutes les issues de pytorch/executorch

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.