kernels benchmark pulls from the old model repos and says every gpu is cpu

Open Beginner friendly
  • Is something wrong?
#914 1 comment 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@drbh is already working on this.

Since Oct 26, 2026.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend, cli

Research direction

Read cli/benchmark.py and inspect collect_machine_info() and the snapshot_download call. Check how _backend() represents CUDA and ROCm, then review the benchmark download options for selecting the kernel repo and limiting files to benchmarks/. Done means benchmarks load scripts from the requested kernel revision and machine info reports the detected GPU and backend.

Written by the indexing model from the issue text.

Description

bug
Description

was running kernels benchmark on an a100 a v100 and an mi100 and ran into two things. both are still there on main 78e14e6 and in 0.17.2

first one is it grabs the benchmark scripts from the old model repo. the snapshot_download call in cli/benchmark.py doesnt pass repo_type so it ends up in models--kernels-community--... and any version that only exists on the kernel repo just 404s. v1 still works but it runs the old copy of the scripts, relu v1 is e804dc4 on the model repo and b5ed5f6 on the kernel repo. so once the model repos go away (#757) every benchmark will break. it also downloads every build variant just to read benchmarks/, for relu that was 73 variants and 32 MB

second one is collect_machine_info() does _backend() in {"cuda", "rocm"} but _backend() gives back a CUDA(version=...) or ROCm(...) object so it never matches and always lands in the cpu branch. every run on a linux gpu saves "gpu": "N/A", "backend": "CPU" and thats the same payload --upload sends. the ROCm label from #902 never gets reached either

Steps to reproduce
$ kernels benchmark kernels-community/paged-attention@v2 --warmup 2 --iterations 5 --output pa.json
Downloading kernels-community/paged-attention@v2...
...
huggingface_hub.errors.RevisionNotFoundError: 404 Client Error.
Revision Not Found for url: https://huggingface.co/api/models/kernels-community/paged-attention/revision/v2.
Invalid rev id: v2

this next one is main on the a100. 0.17.2 gave the same machineInfo on the v100 and the mi100 too

$ kernels benchmark kernels-community/relu@v1 --warmup 2 --iterations 5 --output relu.json
...
  GPU      N/A
  PyTorch  2.14.1+cu130
...
$ python -c "import json; print(json.load(open('relu.json'))['machineInfo'])"
{'gpu': 'N/A', 'backend': 'CPU', 'pytorchVersion': '2.14.1+cu130', 'os': 'Linux 5.14.0-570.37.1.el9_6.x86_64', 'cpu': 'x86_64'}
$ python -c "from kernels.backends import _backend; print(_backend(), _backend() in {'cuda', 'rocm'})"
CUDA(version=<Version('13.0')>) False
Expected behavior

scripts come from the kernel repo at that version, only benchmarks/ gets downloaded, and the result says what it actually ran on like NVIDIA A100-PCIE-40GB and CUDA 13.0

Environment
  • kernels version: main 78e14e6 built from source, also 0.17.2 from pypi
  • Python version: 3.12.14
  • PyTorch version: 2.14.1+cu130, 2.14.1+cu126, 2.14.1+rocm7.2
  • CUDA version: 13.0 and 12.6, ROCm 7.2
  • GPU model: A100-PCIE-40GB, V100-PCIE-32GB, MI100
  • OS: AlmaLinux 9.6
Additional context

looks like a two line fix. I patched main on the a100 and relu@v1 and paged-attention@v2 both run and verify, they show NVIDIA A100-PCIE-40GB and CUDA 13.0, and the cache is 56 KB and 194 KB

-        backend_name = _backend()
+        backend_name = _backend().name

-        repo_path = Path(str(_get_hf_api().snapshot_download(repo_id=repo_id, revision=revision)))
+        repo_path = Path(
+            str(
+                _get_hf_api().snapshot_download(
+                    repo_id=repo_id, repo_type="kernel", revision=revision, allow_patterns=["benchmarks/**"]
+                )
+            )
+        )

happy to open a PR with a test if that sounds ok

Dominant language
Python
Stars
781
Forks
141
Avg merge
1d 10h
Merged PRs (30d)
57

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/kernels

All issues in huggingface/kernels

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.