feat(qa): tune quality QA generation and compare prompts
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
Research direction
Start by reading the prompts in src/anonymizer/engine/rewrite/qa_generation.py and the compare prompt in src/anonymizer/engine/rewrite/evaluate.py, then review the existing e2e evaluation flow. Done means quality QA questions are single-focus, generalized-but-correct answers are treated as matches, and utility-score impact is measured on the bio and legal datasets.
Written by the indexing model from the issue text.
Description
Context
From e2e testing:
- Quality QA questions occasionally ask multiple things in one question, conflating multiple meaning units. This makes the compare step unfairly strict.
- Quality compare sometimes penalizes generalized answers that preserve core meaning but differ in specificity from the original QA answer.
Scope
- Prompt-tune meaning unit extraction and/or quality QA generation to produce single-focus questions
- Guide the quality compare prompt to treat generalized-but-correct answers as matching
- Measure impact on utility scores across bio and legal datasets
Files
src/anonymizer/engine/rewrite/qa_generation.py(QA generation prompts)src/anonymizer/engine/rewrite/evaluate.py(compare prompt).
- Dominant language
- Python
- Stars
- 129
- Forks
- 21
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 21
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/Anonymizer
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
NVIDIA-NeMo/Anonymizer#327 ·
Maintainers usually reply within 1 day
-
Replacement-map generation can silently omit a requested entity, causing the rewrite to be marked unavailablePossibly taken @dundysm claimed this 32 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NVIDIA-NeMo/Anonymizer#278 · 2 comments ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
NVIDIA-NeMo/Anonymizer#6 · 1 comment ·
Maintainers usually reply within 1 day
-
task
Difficulty 4/5 3-5 days Newbie friendliness 48/100
NVIDIA-NeMo/Anonymizer#323 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
NVIDIA-NeMo/Anonymizer#304 ·
Maintainers usually reply within 1 day
All issues in NVIDIA-NeMo/Anonymizer
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Arekkazu/sgpmp-backend#744 ·
Maintainers usually reply within 1 day
-
agent/reviewer
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectbluefin/utah-packages#634 ·
Maintainers usually reply within 1 day
-
enhancement good first issue help wanted
Difficulty 2/5 1-2 days Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
home-assistant/core#188613 ·
Maintainers usually reply within 1 day
-
WebsiteOpengood first issue
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
TanbirRamim/open-source-radar#314 ·
Maintainers usually reply within 1 day