Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Android] LlmGenerationConfig default temperature (0.8) overrides LlmModule temperature in the config generate overload

Đã đóng
  • Có gì không ổn?
#23,263 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
55/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
android, cpp, kotlin
Lĩnh vực
ai, mobile

Hướng nghiên cứu

Start with LlmGenerationConfig.kt and LlmModule.kt, especially the builder default and generate overloads, then trace the temperature handling in jni_layer_llama.cpp. Run the ReproAB.kt program to confirm the differing A-D behavior. Done means the chosen temperature precedence is consistent across overloads and is reflected in docs/source/llm/run-on-android.md.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

🐛 Describe the bug
Summary

On Android, temperature can be set on the LlmModule constructor and on LlmGenerationConfig. The native layer uses a per-call temperature when it is >= 0 and otherwise falls back to the constructor's (#16728). The simple generate(...) overloads pass -1.0f, so they use the constructor's value. LlmGenerationConfig.Builder defaults temperature to 0.8f, so generate(prompt, config, callback) replaces the constructor's value unless the config sets temperature explicitly.

  • Expected: LlmModule(..., temperature = 0.0f) decodes greedily whichever generate overload is used.
  • Actual: through the config overload it samples at 0.8 unless .temperature(...) is set on the config.
To reproduce

Environment: executorch-android 1.4.0 from Maven Central (the latest published), with fbjni 0.7.0 and nativeloader 0.10.5; the test program also uses kotlin-stdlib 2.3.20. Models (both attention only, so #23262 cannot affect them):

The module is built for greedy decoding, then each way of calling generate runs twice from resetContext():

val module = LlmModule(LlmModule.MODEL_TYPE_TEXT, modelPath, tokenizerPath, 0.0f)   // greedy intended
module.load()
fun config() = LlmGenerationConfig.create().seqLen(160).echo(false)

// A: config, temperature not set
module.generate(prompt, config().build(), callback)
// B: config, temperature set to 0
module.generate(prompt, config().temperature(0.0f).build(), callback)
// C: config, temperature set to -1 ("use the module's")
module.generate(prompt, config().temperature(-1.0f).build(), callback)
// D: simple overload
module.generate(prompt, 160, callback, false)

Were the two runs identical?

Call Qwen3, POCO X8 Pro Max Gemma3, POCO X8 Pro Max Qwen3, Android emulator
A: config, temperature not set no no no
B: config, temperature(0.0f) yes yes yes
C: config, temperature(-1.0f) yes yes yes
D: simple overload yes yes yes

Devices: Xiaomi POCO X8 Pro Max (MT6991, Android 16) and an Android 16 emulator (android-36 google_apis arm64-v8a image) on an Apple silicon Mac. On the phone the Qwen3 A/B ran twice with the same result, and an earlier version comparing only A and D gave the same result in two more runs. Example from A (Qwen3, phone):

run 1: In the quiet town of Cape Cod, where the sea whispered secrets to the
run 2: In the quiet coastal town of Elmhurst, there stood a lighthouse that h

Within each B, C and D pair, the two runs are identical, and the first 70 characters are also the same across B, C and D on each device. Runs under A start with the same first word and diverge shortly after, because the first token is chosen during prefill with temperature 0 (text_prefiller.cpp#L32, called at text_llm_runner.cpp#L181) and config.temperature only applies to the decode loop (text_llm_runner.cpp#L240-L245). That makes the problem easy to mistake for small numerical noise.

Code path

Permalinks at 2f78245. The cited lines are the same at v1.5.1, and javap on the 1.4.0 AAR shows the same Kotlin behaviour (builder default 0.8f, config overload passes the config's temperature, simple overloads pass -1.0f).

run-on-android.md#L173-L197 shows a config with .temperature(0.8f) and says the defaults match the C++ GenerationConfig, but does not say that a config temperature replaces the one given to the constructor.

Suggested fix

Either:

  1. Default LlmGenerationConfig.Builder.temperature to -1.0f ("use the module's"), matching the simple overloads. Variant C above shows this restores the constructor's temperature. It changes behaviour for anyone relying on the 0.8 default while passing a different constructor value, so it is worth a release note.
  2. Keep 0.8 and document the precedence on the LlmModule constructor, on LlmGenerationConfig.temperature and in run-on-android.md.

#17637 (consolidating the LlmModule overloads) may be a natural place to settle this. Scope: the generic text path (MODEL_TYPE_TEXT to TextLLMRunner); I have not checked the multimodal or vendor runners.

Full program

ReproAB.kt, verbatim. Built against the AAR's classes.jar and run with app_process, so no app is needed. Commands as run, from a directory holding ReproAB.kt, deps/ (the AAR classes.jar files and nativeloader-0.10.5.jar) and kotlinc/:

kotlinc ReproAB.kt \
  -cp deps/et/classes.jar:deps/fbjni/classes.jar:deps/nativeloader-0.10.5.jar:$ANDROID_HOME/platforms/android-36/android.jar \
  -jvm-target 11 -d reproab-classes.jar
d8 --release --min-api 26 --lib $ANDROID_HOME/platforms/android-36/android.jar --output ab \
  reproab-classes.jar kotlinc/lib/kotlin-stdlib.jar deps/et/classes.jar deps/fbjni/classes.jar deps/nativeloader-0.10.5.jar
# Push ab/classes.dex as /data/local/tmp/et-temp-repro/reproab.dex, plus the model files, plus lib/ with
# libexecutorch.so (AAR jni/arm64-v8a) and libfbjni.so, libc++_shared.so (fbjni AAR).
adb shell "cd /data/local/tmp/et-temp-repro && LD_LIBRARY_PATH=/data/local/tmp/et-temp-repro/lib \
  CLASSPATH=/data/local/tmp/et-temp-repro/reproab.dex app_process -Djava.library.path=/data/local/tmp/et-temp-repro/lib \
  /data/local/tmp/et-temp-repro ReproABKt \
  /data/local/tmp/et-temp-repro/qwen3.pte /data/local/tmp/et-temp-repro/qwen3.tokenizer.json qwen3"   # gemma: its files, "gemma"

// Issue-2 A/B: which way of calling generate() samples when LlmModule was built with temperature 0?
import org.pytorch.executorch.extension.llm.LlmCallback
import org.pytorch.executorch.extension.llm.LlmGenerationConfig
import org.pytorch.executorch.extension.llm.LlmModule
import kotlin.system.exitProcess

fun collect(block: (LlmCallback) -> Unit): String {
    val sb = StringBuilder()
    block(object : LlmCallback { override fun onResult(result: String) { sb.append(result) } })
    return sb.toString()
}

fun main(args: Array<String>) {
    val (modelPath, tokenizerPath, family) = Triple(args[0], args[1], args[2])
    val q = "Tell me a short story about a lighthouse keeper."
    val prompt = when (family) {
        "gemma" -> "<bos><start_of_turn>user\n$q<end_of_turn>\n<start_of_turn>model\n"
        else -> "<|im_start|>user\n$q<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
    }
    val module = LlmModule(LlmModule.MODEL_TYPE_TEXT, modelPath, tokenizerPath, 0.0f)   // greedy intended
    module.load()
    val seqLen = 160
    val variants = listOf<Pair<String, (LlmCallback) -> Unit>>(
        "A config, temperature not set" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).build(), cb) },
        "B config, temperature(0.0f)" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).temperature(0.0f).build(), cb) },
        "C config, temperature(-1.0f)" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).temperature(-1.0f).build(), cb) },
        "D simple overload" to { cb -> module.generate(prompt, seqLen, cb, false) },
    )
    val out = StringBuilder()
    for ((name, gen) in variants) {
        module.resetContext(); val r1 = collect(gen)
        module.resetContext(); val r2 = collect(gen)
        out.append("RESULT $family | $name | identical: ${r1 == r2}\n")
        out.append("   run 1: ${r1.take(70).replace("\n", "\\n")}\n")
        out.append("   run 2: ${r2.take(70).replace("\n", "\\n")}\n")
    }
    print(out)
    System.out.flush()
    exitProcess(0)
}
Versions

executorch-android 1.4.0 (Maven Central), fbjni 0.7.0, nativeloader 0.10.5, kotlin-stdlib 2.3.20 (test program). Source checked at main 2f7824593d0d14f9d5d73faa540b681b6b2b507c and v1.5.1. Devices: Xiaomi POCO X8 Pro Max (MT6991, Android 16); Android 16 emulator (android-36 google_apis arm64-v8a) on an Apple silicon Mac.

Ngôn ngữ chính
Python
Star
5k
Fork
1.2k
Merge trung bình
2 ngày 5 giờ
Pull request đã merge (30 ngày)
533

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của pytorch/executorch

Tất cả issue của pytorch/executorch

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.