Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Ternary decode on 4× Cortex-A78C: row-thread pool only 40–60 % busy, 1.8 tok/s (BitNet 2B4T) — dispatch-bound, not bandwidth-bound

Open
  • Is something wrong?
#1,357 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c, java, kotlin, linux
Domain
backend, machine-learning, performance

Research direction

Start by reading skainet_row_threads.c and the native GEMV entry points NativeTernaryLmheadKernel.lmheadStage1 and NativeTernaryF32GemvKernel.gemvPackedStorage. Profile decode to distinguish row partition and barrier costs from FFM downcall overhead and JVM topK time. Compare throughput and worker utilization after a targeted change; the issue reports 1.8 tok/s and 40–60% worker utilization as the baseline.

Written by the indexing model from the issue text.

Description

compute-backend research skill:native

Problem

During BitNet b1.58 2B4T decode on a linux-arm64 JVM with the native kernels loaded, the 4 worker cores are only 40–60 % busy, and decode is 1.8 tok/s. The implied weight traffic is about 1.4 GB/s against roughly 24 GB/s of measured DRAM read bandwidth, so decode is neither bandwidth-bound nor compute-saturated. That points at dispatch and row-thread overhead.

For comparison, on the same cores llama.cpp b9868 decodes the same model (TQ2_0, f16 tied head) at 17.3 tok/s.

Setup

  • Arduino Ventuno Q (QCS8275), docker --cpuset-cpus=0-3 (4× Cortex-A78C up to 2.36 GHz), Temurin 21.0.12.1.
  • SKaiNET 0.61.0, native kernels cross-built for linux-arm64 (see https://github.com/SKaiNET-developers/SKaiNET/issues/1356).
  • skainet-cli -m ggml-model-i2_s.gguf -s 64 -k 0 "The capital of France is", with the two-stage lm_head enabled (BITNET_PLANES, top-200 rescoring).

Measurements

Per-core utilisation from mpstat, sampled every 8 s during decode:

t=49s decode cpu0=56% cpu1=43% cpu2=74% cpu3=61%
t=57s decode cpu0=46% cpu1=39% cpu2=43% cpu3=60%
t=65s decode cpu0=40% cpu1=40% cpu2=44% cpu3=58%
t=73s decode cpu0=42% cpu1=31% cpu2=62% cpu3=40%

The first SKaiNET frame on the main thread, across 12 jcmd Thread.print samples during decode:

Frame Samples
NativeTernaryLmheadKernel.lmheadStage1 4
NativeTernaryF32GemvKernel.gemvPackedStorage 4
BitNetTwoStageDecode.topK 2
ResidualAdd.onForward 1
JvmVectorKernels.binaryFloat 1

Rebuilding the NeoGPU kernel and its adapter with -mcpu=cortex-a78c instead of the armv8-a pin made no difference (1.82 tok/s).

Questions / suggestions

  • Row partitioning and barrier cost in skainet_row_threads.c for the many small 2560-wide GEMVs per token (q/k/v/o, gate/up/down × 30 layers). Is one job per GEMV too fine-grained? Could batching q/k/v and gate/up into one pool job help?
  • Do the FFM downcalls themselves add measurable overhead? 7 of the 10 deep samples were inside DowncallStub. A per-token batched entry point might help.
  • topK over the 128k-vocab stage-1 scores runs on the JVM main thread while the pool idles. Could it be parallelised or moved native?
Dominant language
Kotlin
Stars
52
Forks
15
Avg merge
1d 10h
Merged PRs (30d)
59

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SKaiNET-developers/SKaiNET

All issues in SKaiNET-developers/SKaiNET

Similar issues

More Kotlin issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.