Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Ternary decode on 4× Cortex-A78C: row-thread pool only 40–60 % busy, 1.8 tok/s (BitNet 2B4T) — dispatch-bound, not bandwidth-bound

Aperta
  • Qualcosa non va?
#1,357 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@michalharakal ci sta già lavorando.

Dal 17/10/2026.

  • #1361 di @michalharakal — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c, java, kotlin, linux
Ambito
backend, machine-learning, performance

Direzione di ricerca

Start by reading skainet_row_threads.c and the native GEMV entry points NativeTernaryLmheadKernel.lmheadStage1 and NativeTernaryF32GemvKernel.gemvPackedStorage. Profile decode to distinguish row partition and barrier costs from FFM downcall overhead and JVM topK time. Compare throughput and worker utilization after a targeted change; the issue reports 1.8 tok/s and 40–60% worker utilization as the baseline.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

compute-backend research skill:native

Problem

During BitNet b1.58 2B4T decode on a linux-arm64 JVM with the native kernels loaded, the 4 worker cores are only 40–60 % busy, and decode is 1.8 tok/s. The implied weight traffic is about 1.4 GB/s against roughly 24 GB/s of measured DRAM read bandwidth, so decode is neither bandwidth-bound nor compute-saturated. That points at dispatch and row-thread overhead.

For comparison, on the same cores llama.cpp b9868 decodes the same model (TQ2_0, f16 tied head) at 17.3 tok/s.

Setup

  • Arduino Ventuno Q (QCS8275), docker --cpuset-cpus=0-3 (4× Cortex-A78C up to 2.36 GHz), Temurin 21.0.12.1.
  • SKaiNET 0.61.0, native kernels cross-built for linux-arm64 (see https://github.com/SKaiNET-developers/SKaiNET/issues/1356).
  • skainet-cli -m ggml-model-i2_s.gguf -s 64 -k 0 "The capital of France is", with the two-stage lm_head enabled (BITNET_PLANES, top-200 rescoring).

Measurements

Per-core utilisation from mpstat, sampled every 8 s during decode:

t=49s decode cpu0=56% cpu1=43% cpu2=74% cpu3=61%
t=57s decode cpu0=46% cpu1=39% cpu2=43% cpu3=60%
t=65s decode cpu0=40% cpu1=40% cpu2=44% cpu3=58%
t=73s decode cpu0=42% cpu1=31% cpu2=62% cpu3=40%

The first SKaiNET frame on the main thread, across 12 jcmd Thread.print samples during decode:

Frame Samples
NativeTernaryLmheadKernel.lmheadStage1 4
NativeTernaryF32GemvKernel.gemvPackedStorage 4
BitNetTwoStageDecode.topK 2
ResidualAdd.onForward 1
JvmVectorKernels.binaryFloat 1

Rebuilding the NeoGPU kernel and its adapter with -mcpu=cortex-a78c instead of the armv8-a pin made no difference (1.82 tok/s).

Questions / suggestions

  • Row partitioning and barrier cost in skainet_row_threads.c for the many small 2560-wide GEMVs per token (q/k/v/o, gate/up/down × 30 layers). Is one job per GEMV too fine-grained? Could batching q/k/v and gate/up into one pool job help?
  • Do the FFM downcalls themselves add measurable overhead? 7 of the 10 deep samples were inside DowncallStub. A per-token batched entry point might help.
  • topK over the 128k-vocab stage-1 scores runs on the JVM main thread while the pool idles. Could it be parallelised or moved native?
Lingua principale
Kotlin
Stelle
52
Fork
15
Merge medio
23h 34m
PR unite (30g)
45

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET

Tutte le issue di SKaiNET-developers/SKaiNET

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.