Remove link to HEIM for diffusion model evaluation
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Newbie friendliness
- 90/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Quiet
- Domain
- documentation
Research direction
Open the Diffusers evaluation documentation at docs/diffusers/conceptual/evaluation and locate the HEIM recommendation. Remove the HEIM link and its suggestion from the list, then check the rendered page to confirm the remaining evaluation guidance and links are intact.
Written by the indexing model from the issue text.
Description
The doc page https://huggingface.co/docs/diffusers/conceptual/evaluation currently suggests using HEIM for diffusion model evaluation.
Unfortunately, HEIM is no longer officially supported or actively maintained (see stanford-crfm/helm#2161, stanford-crfm/helm#3741) and is currently not in a state that would function properly off the shelf.
From my experience, it is currently quite the pain to get some version of HEIM to somewhat function (it did involve some major manual rewrites and tons of dependency conflicts), and still I ran into trouble even reproducing the standard leaderboards.
I would hence recommend to remove HEIM from the list of suggested evaluation frameworks.
- Dominant language
- Python
- Stars
- 34.6k
- Forks
- 7.4k
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 67
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/diffusers
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
huggingface/diffusers#14864 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
huggingface/diffusers#14837 ·
Maintainers usually reply within 1 day
-
bug needs-env-info pipelines
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
huggingface/diffusers#14794 ·
Maintainers usually reply within 1 day
-
bug needs-code-example needs-env-info pipelines
Difficulty 1/5 Under an hour Newbie friendliness 86/100
huggingface/diffusers#14780 · 1 comment ·
Maintainers usually reply within 1 day
-
bug pipelines
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/diffusers#14769 · 1 comment ·
Maintainers usually reply within 1 day
All issues in huggingface/diffusers
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
solana-foundation/pay-kit#341 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
nasa/python_cmr#123 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
EleutherAI/lm-evaluation-harness#4243 ·
Maintainers usually reply within 1 day
-
area: dashboard bug perceived difficulty: 3
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Nitjsefnie-Harness-Commons/daedalus#1179 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
cusp-ai-oss/tojax#17 ·