Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Android] LlmGenerationConfig default temperature (0.8) overrides LlmModule temperature in the config generate overload

Cerrado
  • ¿Algo no está bien?
#23,263 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
android, cpp, kotlin
Área
ai, mobile

Línea de trabajo

Start with LlmGenerationConfig.kt and LlmModule.kt, especially the builder default and generate overloads, then trace the temperature handling in jni_layer_llama.cpp. Run the ReproAB.kt program to confirm the differing A-D behavior. Done means the chosen temperature precedence is consistent across overloads and is reflected in docs/source/llm/run-on-android.md.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

🐛 Describe the bug
Summary

On Android, temperature can be set on the LlmModule constructor and on LlmGenerationConfig. The native layer uses a per-call temperature when it is >= 0 and otherwise falls back to the constructor's (#16728). The simple generate(...) overloads pass -1.0f, so they use the constructor's value. LlmGenerationConfig.Builder defaults temperature to 0.8f, so generate(prompt, config, callback) replaces the constructor's value unless the config sets temperature explicitly.

  • Expected: LlmModule(..., temperature = 0.0f) decodes greedily whichever generate overload is used.
  • Actual: through the config overload it samples at 0.8 unless .temperature(...) is set on the config.
To reproduce

Environment: executorch-android 1.4.0 from Maven Central (the latest published), with fbjni 0.7.0 and nativeloader 0.10.5; the test program also uses kotlin-stdlib 2.3.20. Models (both attention only, so #23262 cannot affect them):

The module is built for greedy decoding, then each way of calling generate runs twice from resetContext():

val module = LlmModule(LlmModule.MODEL_TYPE_TEXT, modelPath, tokenizerPath, 0.0f)   // greedy intended
module.load()
fun config() = LlmGenerationConfig.create().seqLen(160).echo(false)

// A: config, temperature not set
module.generate(prompt, config().build(), callback)
// B: config, temperature set to 0
module.generate(prompt, config().temperature(0.0f).build(), callback)
// C: config, temperature set to -1 ("use the module's")
module.generate(prompt, config().temperature(-1.0f).build(), callback)
// D: simple overload
module.generate(prompt, 160, callback, false)

Were the two runs identical?

Call Qwen3, POCO X8 Pro Max Gemma3, POCO X8 Pro Max Qwen3, Android emulator
A: config, temperature not set no no no
B: config, temperature(0.0f) yes yes yes
C: config, temperature(-1.0f) yes yes yes
D: simple overload yes yes yes

Devices: Xiaomi POCO X8 Pro Max (MT6991, Android 16) and an Android 16 emulator (android-36 google_apis arm64-v8a image) on an Apple silicon Mac. On the phone the Qwen3 A/B ran twice with the same result, and an earlier version comparing only A and D gave the same result in two more runs. Example from A (Qwen3, phone):

run 1: In the quiet town of Cape Cod, where the sea whispered secrets to the
run 2: In the quiet coastal town of Elmhurst, there stood a lighthouse that h

Within each B, C and D pair, the two runs are identical, and the first 70 characters are also the same across B, C and D on each device. Runs under A start with the same first word and diverge shortly after, because the first token is chosen during prefill with temperature 0 (text_prefiller.cpp#L32, called at text_llm_runner.cpp#L181) and config.temperature only applies to the decode loop (text_llm_runner.cpp#L240-L245). That makes the problem easy to mistake for small numerical noise.

Code path

Permalinks at 2f78245. The cited lines are the same at v1.5.1, and javap on the 1.4.0 AAR shows the same Kotlin behaviour (builder default 0.8f, config overload passes the config's temperature, simple overloads pass -1.0f).

run-on-android.md#L173-L197 shows a config with .temperature(0.8f) and says the defaults match the C++ GenerationConfig, but does not say that a config temperature replaces the one given to the constructor.

Suggested fix

Either:

  1. Default LlmGenerationConfig.Builder.temperature to -1.0f ("use the module's"), matching the simple overloads. Variant C above shows this restores the constructor's temperature. It changes behaviour for anyone relying on the 0.8 default while passing a different constructor value, so it is worth a release note.
  2. Keep 0.8 and document the precedence on the LlmModule constructor, on LlmGenerationConfig.temperature and in run-on-android.md.

#17637 (consolidating the LlmModule overloads) may be a natural place to settle this. Scope: the generic text path (MODEL_TYPE_TEXT to TextLLMRunner); I have not checked the multimodal or vendor runners.

Full program

ReproAB.kt, verbatim. Built against the AAR's classes.jar and run with app_process, so no app is needed. Commands as run, from a directory holding ReproAB.kt, deps/ (the AAR classes.jar files and nativeloader-0.10.5.jar) and kotlinc/:

kotlinc ReproAB.kt \
  -cp deps/et/classes.jar:deps/fbjni/classes.jar:deps/nativeloader-0.10.5.jar:$ANDROID_HOME/platforms/android-36/android.jar \
  -jvm-target 11 -d reproab-classes.jar
d8 --release --min-api 26 --lib $ANDROID_HOME/platforms/android-36/android.jar --output ab \
  reproab-classes.jar kotlinc/lib/kotlin-stdlib.jar deps/et/classes.jar deps/fbjni/classes.jar deps/nativeloader-0.10.5.jar
# Push ab/classes.dex as /data/local/tmp/et-temp-repro/reproab.dex, plus the model files, plus lib/ with
# libexecutorch.so (AAR jni/arm64-v8a) and libfbjni.so, libc++_shared.so (fbjni AAR).
adb shell "cd /data/local/tmp/et-temp-repro && LD_LIBRARY_PATH=/data/local/tmp/et-temp-repro/lib \
  CLASSPATH=/data/local/tmp/et-temp-repro/reproab.dex app_process -Djava.library.path=/data/local/tmp/et-temp-repro/lib \
  /data/local/tmp/et-temp-repro ReproABKt \
  /data/local/tmp/et-temp-repro/qwen3.pte /data/local/tmp/et-temp-repro/qwen3.tokenizer.json qwen3"   # gemma: its files, "gemma"

// Issue-2 A/B: which way of calling generate() samples when LlmModule was built with temperature 0?
import org.pytorch.executorch.extension.llm.LlmCallback
import org.pytorch.executorch.extension.llm.LlmGenerationConfig
import org.pytorch.executorch.extension.llm.LlmModule
import kotlin.system.exitProcess

fun collect(block: (LlmCallback) -> Unit): String {
    val sb = StringBuilder()
    block(object : LlmCallback { override fun onResult(result: String) { sb.append(result) } })
    return sb.toString()
}

fun main(args: Array<String>) {
    val (modelPath, tokenizerPath, family) = Triple(args[0], args[1], args[2])
    val q = "Tell me a short story about a lighthouse keeper."
    val prompt = when (family) {
        "gemma" -> "<bos><start_of_turn>user\n$q<end_of_turn>\n<start_of_turn>model\n"
        else -> "<|im_start|>user\n$q<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
    }
    val module = LlmModule(LlmModule.MODEL_TYPE_TEXT, modelPath, tokenizerPath, 0.0f)   // greedy intended
    module.load()
    val seqLen = 160
    val variants = listOf<Pair<String, (LlmCallback) -> Unit>>(
        "A config, temperature not set" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).build(), cb) },
        "B config, temperature(0.0f)" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).temperature(0.0f).build(), cb) },
        "C config, temperature(-1.0f)" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).temperature(-1.0f).build(), cb) },
        "D simple overload" to { cb -> module.generate(prompt, seqLen, cb, false) },
    )
    val out = StringBuilder()
    for ((name, gen) in variants) {
        module.resetContext(); val r1 = collect(gen)
        module.resetContext(); val r2 = collect(gen)
        out.append("RESULT $family | $name | identical: ${r1 == r2}\n")
        out.append("   run 1: ${r1.take(70).replace("\n", "\\n")}\n")
        out.append("   run 2: ${r2.take(70).replace("\n", "\\n")}\n")
    }
    print(out)
    System.out.flush()
    exitProcess(0)
}
Versions

executorch-android 1.4.0 (Maven Central), fbjni 0.7.0, nativeloader 0.10.5, kotlin-stdlib 2.3.20 (test program). Source checked at main 2f7824593d0d14f9d5d73faa540b681b6b2b507c and v1.5.1. Devices: Xiaomi POCO X8 Pro Max (MT6991, Android 16); Android 16 emulator (android-36 google_apis arm64-v8a) on an Apple silicon Mac.

Lenguaje dominante
Python
Estrellas
5.1k
Forks
1.2k
Merge medio
2 d 7 h
PR fusionados (30 d)
691

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pytorch/executorch

Todos los issues de pytorch/executorch

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.