Build Failure When Enabling KleidiAI on ARMv9 in llama-cpp-python ≥ 3.10.0

Open
#2,095 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
c, cmake, cpp, python
Domain
build-system

Research direction

The error occurs during CMake install when it cannot find libkleidiai.so. Start by examining the CMake configuration in the ggml/src directory, particularly the kleidiai_download-build scripts. Compare the build process between versions 3.9.0 and 3.10.0 to see what changed in handling the KleidiAI library on ARMv9. Check the CMake install step that references libkleidiai.so and verify the library path. Done looks like a successful build with the given CMAKE_ARGS on ARMv9.

Written by the indexing model from the issue text.

Description

When I try to install the latest version of llama-cpp-python and enable KleidiAI on an ARMv9 CPU, I use the following command:

CMAKE_ARGS="-DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv9-a+i8mm+dotprod -DGGML_CPU_KLEIDIAI=ON" pip3 install llama-cpp-python

However, during the wheel-building process, it fails with the following error:

      [93/94] : && /usr/bin/cmake -E rm -f vendor/llama.cpp/common/libcommon.a && /usr/bin/aarch64-linux-gnu-ar qc vendor/llama.cpp/common/libcommon.a  vendor/llama.cpp/common/CMakeFiles/build_info.dir/build-info.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/arg.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/chat-parser.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/chat.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/common.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/console.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/json-partial.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/json-schema-to-grammar.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/llguidance.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/log.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/ngram-cache.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/regex-partial.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/sampling.cpp.o vendor/llama.cpp/common/CMakeFiles/common.dir/speculative.cpp.o && /usr/bin/aarch64-linux-gnu-ranlib vendor/llama.cpp/common/libcommon.a && :
      [94/94] : && /usr/bin/aarch64-linux-gnu-g++ -O3 -DNDEBUG  vendor/llama.cpp/tools/mtmd/CMakeFiles/llama-mtmd-cli.dir/mtmd-cli.cpp.o -o vendor/llama.cpp/tools/mtmd/llama-mtmd-cli  -Wl,-rpath,/tmp/tmpuri70waa/build/vendor/llama.cpp/tools/mtmd:/tmp/tmpuri70waa/build/bin:  vendor/llama.cpp/common/libcommon.a  vendor/llama.cpp/tools/mtmd/libmtmd.so  bin/libllama.so  bin/libggml.so  bin/libggml-cpu.so  bin/libggml-base.so && :
      *** Installing project into wheel...
      -- Install configuration: "Release"
      -- Installing: /tmp/tmpuri70waa/wheel/platlib/lib/libggml-cpu.so
      CMake Error at /tmp/tmpuri70waa/build/_deps/kleidiai_download-build/cmake_install.cmake:52 (file):
        file INSTALL cannot find "/tmp/tmpuri70waa/build/bin/libkleidiai.so": No
        such file or directory.
      Call Stack (most recent call first):
        /tmp/tmpuri70waa/build/vendor/llama.cpp/ggml/src/cmake_install.cmake:65 (include)
        /tmp/tmpuri70waa/build/vendor/llama.cpp/ggml/cmake_install.cmake:47 (include)
        /tmp/tmpuri70waa/build/vendor/llama.cpp/cmake_install.cmake:47 (include)
        /tmp/tmpuri70waa/build/cmake_install.cmake:47 (include)
      
      
      
      *** CMake install failed
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for llama-cpp-python
Failed to build llama-cpp-python
error: failed-wheel-build-for-install

× Failed to build installable wheels for some pyproject.toml based projects
╰─> llama-cpp-python

These CMake flags (-DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv9-a+i8mm+dotprod -DGGML_CPU_KLEIDIAI=ON) have already been tested on llama.cpp, and the build succeeds there.

The installation works for llama-cpp-python==3.9.0, but starting from 3.10.0, the build fails with the error above.

Dominant language
Python
Stars
10.7k
Forks
1.5k
Avg merge
1h 40m
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from abetlen/llama-cpp-python

All issues in abetlen/llama-cpp-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.