Compute metrics during evaluation, not during training
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start by reading the training entry points in src/morphoclass/console/cmd_train.py around lines 139-141 and src/morphoclass/console/cmd_morphometrics.py around lines 198-200. Trace the evaluation stages and any uses of checkpoint.chk to identify assumptions about stored metrics. Done means metrics are computed during evaluation rather than training, with no remaining checkpoint metric assumptions.
Written by the indexing model from the issue text.
Description
Context
Currently we are computing all metrics during the model training. This is true both for:
- Training of TMD and DL classifiers:
https://github.com/BlueBrain/morphoclass/blob/35bbe615bbf3a0f085e2d2ca1ebcf67c4d95a868/src/morphoclass/console/cmd_train.py#L139-L141 - Training of Morphometrics classifiers:
https://github.com/BlueBrain/morphoclass/blob/35bbe615bbf3a0f085e2d2ca1ebcf67c4d95a868/src/morphoclass/console/cmd_morphometrics.py#L198-L200
Actions
- Make sure that nowhere else in the code (e.g. during evaluation) we are assuming that the
checkpoint.chkproduced duringtrainingcontains metrics scores. - Move the computation of metrics scores from
trainingstage toevaluationstages.
See also
https://github.com/BlueBrain/morphoclass/pull/45#issuecomment-1115874293
- Dominant language
- Python
- Stars
- 12
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from BlueBrain/morphoclass
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
BlueBrain/morphoclass#87 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
BlueBrain/morphoclass#86 ·
-
bug
BlueBrain/morphoclass#79 · 2 comments · 1 assignee ·
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
BlueBrain/morphoclass#56 ·
-
📜 Paper
Difficulty 3/5 1-2 days Newbie friendliness 25/100
BlueBrain/morphoclass#47 ·
All issues in BlueBrain/morphoclass
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100