First-time loading of large kernels (e.g. `deep-gemm`) hits repeated 429 rate limits and takes very long

Bắt đầu với kernels/_versions.py, đặc biệt là _get_available_versions và list_repo_refs, sau đó lần theo quá trình liệt kê cây repository đệ quy thông qua get_kernel. Tái hiện vấn đề bằng ví dụ torchrun Qwen3 FP8 với cache lạnh được cung cấp. Được xem là hoàn tất khi việc tải kernel phân tán lần đầu hoàn thành trong thời gian hợp lý mà không liên tục backoff 429.

Description

The kernel is loaded indirectly via transformers (finegrained FP8 → DeepGEMM integration) on the first forward pass, not by calling get_kernel directly. During version resolution (kernels/_versions.py → list_repo_refs) and the recursive repo tree listing, requests get 429-rate-limited. This is a multi-rank distributed run, so every rank hits the Hub API independently on startup and together we blow the 3000 api-requests / 5-min limit, stalling the first training step for many minutes (~120–207s backoff per retry).

HTTP Error 429 thrown while requesting GET https://huggingface.co/api/models/kernels-community/deep-gemm/tree/9590415046fa95a187af7ea03391d4782047170d?expand=false&recursive=true&limit=1000&cursor=ZXlKbWFXeGxYMjVoYldVaU9pSmlkV2xzWkM5MGIzSmphREl4TWkxamRURXpNaTFoWVhKamFEWTBMV3hwYm5WNEwybHVZMngxWkdVdlpHVmxjRjluWlcxdEwybHRjR3h6TDNOdE1UQXdYMkp0YTE5aWJtdGZiVzR1WTNWb0lpd2lkSEpsWlY5dmFXUWlPaUk1TURZM05HWXdNbU16TVRJNFlUWTFObU13WkRZek1tRTFPRGs0TjJKbVpHRmpPR1JqTmpNeUluMD06MjUwMDA%3D
Rate limited. Waiting 207.0s before retry [Retry 1/5].

[rank4]: Traceback (most recent call last):
[rank4]:   File "/workspace/flux_finetune_lovescape.py", line 734, in <module>
[rank4]:     main()
[rank4]:   File "/workspace/flux_finetune_lovescape.py", line 575, in main
[rank4]:     negative_ctx  = text_encoder([negative_prompt] * BATCH_SIZE).to(device)
[rank4]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
[rank4]:     return func(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/flux2/text_encoder.py", line 411, in forward
[rank4]:     output = self.model(
[rank4]:              ^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/utils/generic.py", line 907, in wrapper
[rank4]:     output = func(self, *args, **kwargs)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 492, in forward
[rank4]:     outputs: BaseModelOutputWithPast = self.model(
[rank4]:                                        ^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/utils/generic.py", line 1036, in wrapper
[rank4]:     output = func(self, *args, **kwargs)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/utils/output_capturing.py", line 252, in wrapper
[rank4]:     outputs = func(self, *args, **kwargs)
[rank4]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 424, in forward
[rank4]:     hidden_states = decoder_layer(
[rank4]:                     ^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/modeling_layers.py", line 93, in __call__
[rank4]:     return super().__call__(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1884, in _call_impl
[rank4]:     return inner()
[rank4]:            ^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1832, in inner
[rank4]:     result = forward_call(*args, **kwargs)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 318, in forward
[rank4]:     hidden_states, _ = self.self_attn(
[rank4]:                        ^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1884, in _call_impl
[rank4]:     return inner()
[rank4]:            ^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1832, in inner
[rank4]:     result = forward_call(*args, **kwargs)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 263, in forward
[rank4]:     query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
[rank4]:                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/finegrained_fp8.py", line 308, in forward
[rank4]:     return fp8_linear(
[rank4]:            ^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/finegrained_fp8.py", line 248, in fp8_linear
[rank4]:     return deepgemm_fp8_fp4_linear(
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/deepgemm.py", line 459, in deepgemm_fp8_fp4_linear
[rank4]:     deepgemm = load_deepgemm_kernel(requires_sm100=weight.dtype == torch.int8)
[rank4]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/deepgemm.py", line 169, in load_deepgemm_kernel
[rank4]:     return _load_deepgemm_kernel(requires_sm100=requires_sm100)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/deepgemm.py", line 106, in _load_deepgemm_kernel
[rank4]:     kernel = lazy_load_kernel("deep-gemm")
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/hub_kernels.py", line 409, in lazy_load_kernel
[rank4]:     kernel = get_kernel(repo_id, revision=revision, version=version, allow_all_kernels=ALLOW_ALL_KERNELS)
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/transformers/integrations/hub_kernels.py", line 469, in get_kernel
[rank4]:     return get_kernel_hub(kernel_name, revision=revision, version=version, user_agent=user_agent)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/kernels/utils.py", line 319, in get_kernel
[rank4]:     revision = select_revision_or_version(repo_id, revision=revision, version=version)
[rank4]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/kernels/_versions.py", line 85, in select_revision_or_version
[rank4]:     return resolve_version_spec_as_ref(repo_id, version).target_commit
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/kernels/_versions.py", line 49, in resolve_version_spec_as_ref
[rank4]:     versions = _get_available_versions(repo_id)
[rank4]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/kernels/_versions.py", line 12, in _get_available_versions
[rank4]:     for branch in HfApi().list_repo_refs(repo_id).branches:
[rank4]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
[rank4]:     return fn(*args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/venv/main/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 4061, in list_repo_refs
[rank4]:     hf_raise_for_status(response)
[rank4]:   File "/venv/main/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 868, in hf_raise_for_status
[rank4]:     raise _format(HfHubHTTPError, message, response) from e
[rank4]: huggingface_hub.errors.HfHubHTTPError: (Request ID: Root=1-6a2d24c0-614bb9854003adb23b4e8fc3;0d7e38c0-8542-47df-9bd3-96fdae3f917a)

[rank4]: 429 Too Many Requests: you have reached your 'api' rate limit.
[rank4]: Retry after 122 seconds (0/3000 requests remaining in current 300s window).
[rank4]: Url: https://huggingface.co/api/models/kernels-community/deep-gemm/refs.
[rank4]: We had to rate limit you, you hit the quota of 3000 api requests per 5 minutes period. Upgrade to higher plan to get higher limits (current: team). See https://huggingface.co/docs/hub/rate-limits

Steps to reproduce

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Distributed run (torchrun, multiple ranks), cold HF cache.
model_spec = "Qwen/Qwen3-8B-FP8"   # finegrained FP8 checkpoint
model = AutoModelForCausalLM.from_pretrained(model_spec, torch_dtype=None, device_map="cuda")
tok = AutoTokenizer.from_pretrained(model_spec)

inputs = tok("hello", return_tensors="pt").to(model.device)
with torch.no_grad():
    model(**inputs)   # first forward -> FP8 path lazily pulls kernels-community/deep-gemm -> 429

Expected behavior

Loading the kernel on a cold cache completes in a reasonable time without repeated 429 rate limits stalling for ~200s each

Environment

• kernels version: 0.12.3
• Python version: 3.12.13
• PyTorch version: 2.13.0.dev20260611+cu130
• CUDA version: 13.0
• GPU model: NVIDIA H200
• OS: Linux-6.8.0-90-generic-x86_64-with-glibc2.39

Additional context

No response