[Feature]: eval NeUQI with typical tasks and provide usage suggestion to users
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 22/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning
Research direction
The payload names no files, only the existing NeUQI scale-initialization support and its KL-divergence evaluation data, so locate those in the repo first. The work is to run MMLU, GSM8K and GPQA with NeUQI and the current default, then compare results. Done means a written comparison and a usage recommendation for which scenarios should switch to NeUQI, which the issue does not define precisely.
Written by the indexing model from the issue text.
Description
Feature Description
Thanks to @avtc, we now support NeUQI, a new scale initialization approach, along with some evaluation data. The current evaluation mainly uses KL divergence as the metric. We should also test it on more familiar benchmarks, such as MMLU, GSM8K, and GPQA, and switch to NeUQI in the scenarios where it delivers better results
Motivation and Use Case
~
Alternatives Considered
~
Definition of Done
No response
Additional Context
No response
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 193
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 50
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/auto-round
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
intel/auto-round#2525 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 44/100
intel/auto-round#2532 ·
Maintainers usually reply within 1 day
-
[Bug]: FP8 static KV cache accuracy issuePossibly taken @xin3he claimed this 7 days ago. Openbug
intel/auto-round#2518 · 1 reaction · 1 assignee ·
Maintainers usually reply within 1 day
-
[Feature]: split special_model_handler to a folderPossibly taken @Cross2pro claimed this today. Openenhancement
Difficulty 3/5 Half a day Newbie friendliness 42/100
intel/auto-round#2514 · 2 comments ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 3/5 1-2 days Newbie friendliness 20/100
intel/auto-round#2507 ·
Maintainers usually reply within 1 day
All issues in intel/auto-round
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
pydot/pydot#601 · 4 comments ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
letsencrypt/cp-cps#361 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
CI: startup_failure because callers grant fewer permissions than ledger-app-workflows@v1 declaresPossibly taken @maykonmichel claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
LedgerHQ/app-boilerplate-rust#227 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
linuxmint/mintupdate#1107 ·