First-time loading of large kernels (e.g. `deep-gemm`) hits repeated 429 rate limits and takes very long

Start with kernels/_versions.py, especially _get_available_versions and list_repo_refs, then trace the recursive repository tree listing through get_kernel. Reproduce with the provided Qwen3 FP8 cold-cache torchrun example. Done means first-time distributed kernel loading completes in a reasonable time without repeated 429 backoffs.

Description

The kernel is loaded indirectly via transformers (finegrained FP8 → DeepGEMM integration) on the first forward pass, not by calling get_kernel directly. During version resolution (kernels/_versions.py → list_repo_refs) and the recursive repo tree listing, requests get 429-rate-limited. This is a multi-rank distributed run, so every rank hits the Hub API independently on startup and together we blow the 3000 api-requests / 5-min limit, stalling the first training step for many minutes (~120–207s backoff per retry).

HTTP Error 429 thrown while requesting GET https://huggingface.co/api/models/kernels-community/deep-gemm/tree/9590415046fa95a187af7ea03391d4782047170d?expand=false&recursive=true&limit=1000&cursor=ZXlKbWFXeGxYMjVoYldVaU9pSmlkV2xzWkM5MGIzSmphREl4TWkxamRURXpNaTFoWVhKamFEWTBMV3hwYm5WNEwybHVZMngxWkdVdlpHVmxjRjluWlcxdEwybHRjR3h6TDNOdE1UQXdYMkp0YTE5aWJtdGZiVzR1WTNWb0lpd2lkSEpsWlY5dmFXUWlPaUk1TURZM05HWXdNbU16TVRJNFlUWTFObU13WkRZek1tRTFPRGs0TjJKbVpHRmpPR1JqTmpNeUluMD06MjUwMDA%3D
Rate limited. Waiting 207.0s before retry [Retry 1/5].

[rank4]: Traceback (most recent call last):
[rank4]:   File "/workspace/flux_finetune_lovescape.py", line 734, in <module>
[rank4]:     main()
[rank4]:   File "/workspace/flux_finetune_lovescape.py", line 575, in main
[rank4]:     negative_ctx  = text_encoder([negative_prompt] * BATCH_SIZE).to(device)
[rank4]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
[rank4]:     return func(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/flux2/text_encoder.py", line 411, in forward
[rank4]:     output = self.model(
[rank4]:              ^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/utils/generic.py", line 907, in wrapper
[rank4]:     output = func(self, *args, **kwargs)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 492, in forward
[rank4]:     outputs: BaseModelOutputWithPast = self.model(
[rank4]:                                        ^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/utils/generic.py", line 1036, in wrapper
[rank4]:     output = func(self, *args, **kwargs)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/utils/output_capturing.py", line 252, in wrapper
[rank4]:     outputs = func(self, *args, **kwargs)
[rank4]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 424, in forward
[rank4]:     hidden_states = decoder_layer(
[rank4]:                     ^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/modeling_layers.py", line 93, in __call__
[rank4]:     return super().__call__(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1884, in _call_impl
[rank4]:     return inner()
[rank4]:            ^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1832, in inner
[rank4]:     result = forward_call(*args, **kwargs)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 318, in forward
[rank4]:     hidden_states, _ = self.self_attn(
[rank4]:                        ^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1884, in _call_impl
[rank4]:     return inner()
[rank4]:            ^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1832, in inner
[rank4]:     result = forward_call(*args, **kwargs)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 263, in forward
[rank4]:     query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
[rank4]:                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/finegrained_fp8.py", line 308, in forward
[rank4]:     return fp8_linear(
[rank4]:            ^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/finegrained_fp8.py", line 248, in fp8_linear
[rank4]:     return deepgemm_fp8_fp4_linear(
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/deepgemm.py", line 459, in deepgemm_fp8_fp4_linear
[rank4]:     deepgemm = load_deepgemm_kernel(requires_sm100=weight.dtype == torch.int8)
[rank4]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/deepgemm.py", line 169, in load_deepgemm_kernel
[rank4]:     return _load_deepgemm_kernel(requires_sm100=requires_sm100)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/deepgemm.py", line 106, in _load_deepgemm_kernel
[rank4]:     kernel = lazy_load_kernel("deep-gemm")
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/hub_kernels.py", line 409, in lazy_load_kernel
[rank4]:     kernel = get_kernel(repo_id, revision=revision, version=version, allow_all_kernels=ALLOW_ALL_KERNELS)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/hub_kernels.py", line 469, in get_kernel
[rank4]:     return get_kernel_hub(kernel_name, revision=revision, version=version, user_agent=user_agent)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/kernels/utils.py", line 319, in get_kernel
[rank4]:     revision = select_revision_or_version(repo_id, revision=revision, version=version)
[rank4]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/kernels/_versions.py", line 85, in select_revision_or_version
[rank4]:     return resolve_version_spec_as_ref(repo_id, version).target_commit
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/kernels/_versions.py", line 49, in resolve_version_spec_as_ref
[rank4]:     versions = _get_available_versions(repo_id)
[rank4]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/kernels/_versions.py", line 12, in _get_available_versions
[rank4]:     for branch in HfApi().list_repo_refs(repo_id).branches:
[rank4]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
[rank4]:     return fn(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 4061, in list_repo_refs
[rank4]:     hf_raise_for_status(response)
[rank4]:   File "/venv/main/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 868, in hf_raise_for_status
[rank4]:     raise _format(HfHubHTTPError, message, response) from e
[rank4]: huggingface_hub.errors.HfHubHTTPError: (Request ID: Root=1-6a2d24c0-614bb9854003adb23b4e8fc3;0d7e38c0-8542-47df-9bd3-96fdae3f917a)

[rank4]: 429 Too Many Requests: you have reached your 'api' rate limit.
[rank4]: Retry after 122 seconds (0/3000 requests remaining in current 300s window).
[rank4]: Url: https://huggingface.co/api/models/kernels-community/deep-gemm/refs.
[rank4]: We had to rate limit you, you hit the quota of 3000 api requests per 5 minutes period. Upgrade to higher plan to get higher limits (current: team). See https://huggingface.co/docs/hub/rate-limits

Steps to reproduce

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Distributed run (torchrun, multiple ranks), cold HF cache.
model_spec = "Qwen/Qwen3-8B-FP8"   # finegrained FP8 checkpoint
model = AutoModelForCausalLM.from_pretrained(model_spec, torch_dtype=None, device_map="cuda")
tok = AutoTokenizer.from_pretrained(model_spec)

inputs = tok("hello", return_tensors="pt").to(model.device)
with torch.no_grad():
    model(**inputs)   # first forward -> FP8 path lazily pulls kernels-community/deep-gemm -> 429

Expected behavior

Loading the kernel on a cold cache completes in a reasonable time without repeated 429 rate limits stalling for ~200s each

Environment

• kernels version: 0.12.3
• Python version: 3.12.13
• PyTorch version: 2.13.0.dev20260611+cu130
• CUDA version: 13.0
• GPU model: NVIDIA H200
• OS: Linux-6.8.0-90-generic-x86_64-with-glibc2.39

Additional context

No response