ggml-org/whisper.cpp

NPU support in whisper.cpp

Open

#1,557 建立於 2023年11月27日

在 GitHub 查看
 (18 留言) (12 反應) (0 負責人)C++ (49,693 star) (5,535 fork)batch import
good first issueperformanceresearch🔬

描述

Christmas is coming soon, and I want to take some time to research something interesting, such as edge low-power inference. Although current whisper.cpp can run on Raspberry Pi, the inference performance cannot achieve real-time transcription. Fortunately, there are now some development boards that use processors with NPUs, which can be used to achieve real-time transcription of large models. My primary goal is to first support RK3566 and RK3588.

Roadmap:

  • MatMul offloading
  • Conv-Gelu offloading
  • LayerNorm offloading ...

Reference:

https://github.com/rockchip-linux/rknpu2

貢獻者指南

技術棧
cpp
領域
machine learningperformance
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
5
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
over 1 week
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
active
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
C++ programmingNeural network inference basicsNPU architecture knowledgeRK3566/RK3588 familiarity
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
20
研究方向
The issue proposes adding NPU support for RK3566 and RK3588, with a roadmap for offloading MatMul, Conv Gelu, and LayerNorm. Research should focus on integrating with the Rockchip RKNN toolkit (rknpu2) and modifying the whisper.cpp backend to delegate these operations to the NPU. Investigate existing ggml backend implementations for similar hardware and adapt the build system. Review the linked discussion on Raspberry Pi for performance baselines.