Add support for experimental wheel variants (i.e., wheelnext)

Open
#2,092 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python

Research direction

The issue is about modifying the wheel building and publishing process. Start by examining the project's build scripts, likely in setup.py or pyproject.toml, and CI/CD workflows. Research the WheelNext specification and how projects like PyTorch implement variant metadata. The goal is to produce backend-specific wheels (CUDA, ROCm, Metal) with correct metadata, ensuring CPU wheels remain as a fallback. Testing will involve building wheels locally and verifying the metadata.

Written by the indexing model from the issue text.

Description

Is your feature request related to a problem? Please describe.
Today, installing llama-cpp-python on machines with different GPU backends (CUDA, ROCm, Metal, etc.) requires separate package names, custom extra indexes, or installer-level logic to select the correct wheel. This creates friction for downstream tooling (CLIs, orchestrators, and packaging systems) that want to provide a “just works” experience, especially when users don’t know which backend they need. Even a simple developer-driven install might require picking precisely the correct wheel.

Describe the solution you'd like
Add support for WheelNext-compatible experimental wheel variants when building and publishing wheels.

This would allow llama-cpp-python to produce a single package version that provides multiple backend-aware binary wheels, each annotated with variant metadata (e.g., GPU type, CUDA version, ROCm version).

Installers that understand the WheelNext spec (now used experimentally by PyTorch, uv, and others) can automatically select the correct backend wheel based on the system’s hardware/software configuration without a need for custom index URLs, separate packages, or manual backend flags.

Key pieces:

  • Generate wheels with variant metadata following the experimental WheelNext (wheel variants) conventions.
  • Publish per-backend wheels using the standardized naming + metadata fields.
  • Ensure that CPU-only wheels remain available as fallback.

This would significantly simplify installation for all users and remove backend-selection logic from downstream tools. Wheel variants are fully backward-compatible so existing workflows won't be disrupted.

Describe alternatives you've considered

  • Separate package names per backend (e.g., llama-cpp-python-cuda): fragments packaging and forces manual selection.
  • Extras for backend variants (pip install llama-cpp-python[cuda]): still requires external detection and doesn’t integrate with hardware-aware installer selection.
  • Custom index URLs for backend wheels: brittle and requires orchestration logic outside Python packaging.
  • CLI-backed installation routing (what many downstream projects do currently): it’s reinventing the wheel and provides an inconsistent experience for end users.

All of these solutions put the burden on downstream tooling rather than on standardized wheel metadata.

Additional context

Dominant language
Python
Stars
10.7k
Forks
1.5k
Avg merge
1h 40m
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from abetlen/llama-cpp-python

All issues in abetlen/llama-cpp-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.