[RFC] Standardized Model Config for Benchmarks
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning, performance
Research direction
Start by reviewing the existing benchmark scripts and their current device-specific tensor shapes. Clarify the supported devices, baseline model profiles, canonical parameters, and memory-scaling rules before refactoring benchmarks to consume a shared configuration. Done means the configuration is agreed and existing benchmarks use it consistently.
Written by the indexing model from the issue text.
Description
🚀 The feature, motivation and pitch
As we add support for more devices, benchmarking scripts increasingly rely on device-specific tensor shapes due to VRAM constraints. This leads to fragmented benchmark setups and scattered performance results that are difficult to compare or reproduce. To make benchmarking more scalable and consistent, I propose introducing a standardized benchmark model configuration.
Motivation
Today, different kernels (and sometimes different devices) use ad-hoc shapes, which:
- Require device-specific dispatching in benchmark scripts
- Produce results that are hard to compare across hardware
- Increase maintenance burden for contributors adding new benchmarks
Proposal
Define one or more representative “mainstream” model profiles (e.g., LLaMA-/GPT-like) with canonical parameters such as: hidden_size, vocab_size, num_q_heads, num_kv_heads, etc.
All benchmark scripts would derive their shapes from this shared config, optionally using scaled-down subsets when needed to fit memory constraints, instead of inventing per-device shapes.
This would:
- Eliminate ad-hoc device-specific shapes
- Improve comparability across kernels and hardware
- Lower the barrier for contributors writing new benchmarks
- Improve reproducibility of performance results
Target Devices
Before finalizing the standardized config, it would be helpful to clarify which devices we officially want to support for benchmarking.
Here’s the list we currently have:
- NVIDIA H100 (80GB)
- Intel XPU GPU Max 1100 (48GB)
- NPU Atlas 900 A2 POD (64G)
Please feel free to comment if I missed any devices, or if there are additional targets we should consider.
Next Steps
- Confirm target benchmarking devices
- Agree on one or more baseline model profiles (e.g., LLaMA, Qwen, Gemma, etc)
- Define canonical parameters (hidden_size, vocab_size, heads, etc.)
- Specify scaling rules for memory-constrained devices
- Refactor existing benchmarks to consume the standardized config
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 ·
Maintainers usually reply within 3 days
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsPossibly taken @nazek52 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
linkedin/Liger-Kernel#1647 ·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 · 3 comments ·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
linkedin/Liger-Kernel#1661 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 ·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
author-page correction move-papers
Difficulty 1/5 Under an hour Newbie friendliness 78/100
acl-org/acl-anthology#10978 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
HKUDS/CLI-Anything#652 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Project-N-E-K-O/N.E.K.O#3813 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Dao-AILab/flash-attention#3191 ·
Maintainers usually reply within 4 days