Evaluation suite - output
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
Research direction
No files, tests, or implementation entry points are named. Start by reviewing the evaluation process and the linked Evaluation Metrics Task Force context, then identify where reference/output metrics are calculated; done means raw and population reports are available as pandas data frames and can be saved as CSV, XLSX, and HTML.
Written by the indexing model from the issue text.
Description
Context
Following the evaluation process on a set of references / output pairs, an aggregated report should be produced
Describe the solution you'd like
2 separate aggregations should be proposed:
- Raw aggregation - All the calculated metrics for each reference/output pair should be made available
- Population aggregation - Core statistics of the metrics over the population of reference/output pair should be produced including
- central tendency and variation
- indication of range
- indication of cases at 5, 25, 50, 75, 95 percentile of performance for each metric
- The reports should be made available as pandas data frame with the option to save in a specified format (csv, xlsx, html)
Additional context
https://github.com/Project-MONAI/MONAI/wiki/Evaluation-metrics-task-force
@jenspetersen
@danieltudosiu
@AReinke
- Dominant language
- Python
- Stars
- 105
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Project-MONAI/MetricsReloaded
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
MorphologyOps.foreground_component() ignores connectivity parameter, always uses 1-connectivityOpen
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 50/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 52/100
All issues in Project-MONAI/MetricsReloaded
Similar issues
-
comp/tools duplicate P2 sweeper:risk-compatibility tool/mcp type/bug
Difficulty 1/5 Under an hour Newbie friendliness 88/100
NousResearch/hermes-agent#132042 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepset-ai/haystack#13092 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EverMind-AI/Raven#845 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
AstrBotDevs/AstrBot#10340 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
BasedHardware/omi#20401 · 1 comment ·
Maintainers usually reply within 1 day