Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Retire kllama-cli's legacy dense-FP32 LlamaRuntime path; route Llama/Mistral GGUF through the DSL like Qwen already does

Aperta
  • Qualcosa non va?
#354 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
kotlin
Ambito
backend, performance

Direzione di ricerca

Start at llm-runtime/kllama's Main.kt architecture branch and trace the Qwen DSL route alongside the Llama/Mistral GGUF route. Read KLlamaJava.loadGGUF and inspect LlamaNetworkLoader.fromWeights and the Llama2DotCWeightLoader lane to establish what can be reused and what still depends on the legacy runtime. Audit non-CLI callers of LlamaIngestion and LlamaRuntime; done means GGUF uses the DSL and any retained legacy path is scoped to .bin if needed.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

`llm-runtime/kllama`'s `Main.kt` (kllama-cli) branches on GGUF architecture: Qwen GGUFs go through the modern DSL path (`DecoderGgufWeightLoader` → `QwenNetworkLoader` → `OptimizedLLMRuntime` DIRECT — packed/MAPPED weights, kernel-dispatch-routed matmuls), but Llama/Mistral GGUFs still fall to the legacy `LlamaRuntime` path via `LlamaIngestion`/`LlamaLoadConfig` — dense FP32, no `KernelDispatch`/`WeightForm` involvement at all. See the comment at the branch point:

```kotlin
// --- Llama / SafeTensors / BIN: legacy LlamaRuntime path
// (dense FP32; the eager kernels assume dense data).
// Qwen GGUF stays on the DSL branch above.
```

This is the exact machinery the #338 engine-adoption arc (PR #353) migrated everything else off of — `LlamaRuntime` predates `WeightForm`/MAPPED residency/the row-major kernel packs entirely, and (being dense-FP32-only) can't benefit from any of it.

Impact (measured)

Loading `Llama-3.2-1B-Instruct-Q4_K_M.gguf` (762 MB) via kllama-cli's legacy branch: still running 16-token generation after 5+ minutes at 190%+ CPU, never completing. The exact same file through the DSL path (`KLlamaJava.loadGGUF`, used by `skainet-cli` and by SKaiNET-EdgeTranslator's SkaiNet engine integration) loads in 1.9s and generates 16 tokens in 661ms (~24 tok/s) — coherent output.

This is a correctness-adjacent trap, not just a perf gap: anyone hitting kllama-cli with a plain Llama/Mistral GGUF (the common case — Qwen is the exception, not the rule) gets a runtime that appears to hang, with the working DSL path sitting right there in the same module used one branch away.

Ask

  • Route the Llama/Mistral GGUF branch in kllama-cli's `Main.kt` through the same DSL path Qwen already uses (`DecoderGgufWeightLoader` → `LlamaNetworkLoader.fromWeights` → `OptimizedLLMRuntime` DIRECT — exactly what `KLlamaJava.loadGGUF` already does).
  • Once nothing exercises it, delete `LlamaRuntime`, `LlamaRuntimeInterface`, `LlamaIngestion`/`LlamaLoadConfig`'s eager-dense path, and any other FP32-only/no-DSL runtime left over from before the engine-loader migration. (`LlamaRuntime` also still serves the llama2.c `.bin` lane via `Llama2DotCWeightLoader` — confirm that lane's fate first; either port it onto the DSL too or keep a narrowly-scoped legacy runtime for `.bin` only, not GGUF.)
  • Audit other non-CLI callers of `LlamaIngestion`/`LlamaRuntime` for the same trap (kllama's Android/native test targets, samples) while this is being cleaned up.

Context

Found while wiring SKaiNET-EdgeTranslator's SkaiNet engine option (PR #353's downstream consumer) — `KLlamaJava.loadGGUF` (the facade EdgeTranslator calls) was already correctly on the DSL path, but was separately missing the `KernelPacks.install()`/`FfmRowMajorKernelPack.install()` calls skainet-cli has (fixed alongside this investigation). kllama-cli's own Main.kt, investigated as a comparison baseline, is what surfaced this separate, deeper issue.

🤖 Generated with Claude Code

Lingua principale
Kotlin
Stelle
8
Fork
3
Merge medio
19h 45m
PR unite (30g)
38

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET-transformers

Tutte le issue di SKaiNET-developers/SKaiNET-transformers

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.