Gemma 2 / Mistral 7b v.1 issues
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- ai, machine-learning
Research direction
Start with the zero-shot evaluation flow and the compute_universal_function_vector function, then reproduce the top-1 accuracy results for Gemma 2 and Mistral 7B v0.1 across layers. Compare their indirect-effect aggregation and top-head selection with models that work; done means identifying and documenting the model-specific cause or providing a verified fix.
Written by the indexing model from the issue text.
Description
Hi, I have been running the zero-shot evaluation on several models. For models that don't already have a universal top head list, I compute the indirect effect scores for each abstractive task, and, for the tasks with 10-shot performance greater than the majority baseline, aggregate them. I then add the top k heads (k is roughly proportional to # heads in the model) list to the compute_universal_function_vector function. I then evaluate using these top heads. For most models, I seem to get good results. However, for gemma 2 (of all sizes) and Mistral 7b v.01, I almost always get a top-1 accuracy lower than 10%, with a score that doesn't change much between the layers. Do you know why the tests don't work with these models?
Thanks!
- Dominant language
- Python
- Stars
- 200
- Forks
- 47
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ericwtodd/function_vectors
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
ericwtodd/function_vectors#19 · 1 comment ·
All issues in ericwtodd/function_vectors
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100