Evaluate pretrained models (metrics, size, MACs)
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- data-visualization, machine-learning
Research direction
Work in scripts/evaluate_pretrained.py, which should produce results/metrics_pretrained.json and figures/curves_pretrained.png. Start by reading the script and the issues this depends on (#37, #23, #25, #26, #27), since the pretrained models and their outputs come from them. Measure params, size on disk, MACs at 64x64 via thop.profile, and latency. Done when the metrics JSON contains entries for both pretrained models.
Written by the indexing model from the issue text.
Description
The brief asks for parameter count and model size in MB.
Files: scripts/evaluate_pretrained.py
Tasks
- Accuracy, precision, recall, F1, confusion matrices -> figures
- Params, size on disk (MB), MACs at 64x64 using
thop.profile, latency - Curves figure ->
figures/curves_pretrained.png; metrics ->results/metrics_pretrained.json
Done when
Metrics JSON contains both pretrained models.
Depends on
#37 #23 #25 #26 #27
Close with a commit or PR message containing Closes #<this issue>.
- Dominant language
- Jupyter Notebook
- Stars
- 0
- Forks
- 0
- Avg merge
- 1m
- Merged PRs (30d)
- 3
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from VisionForgeCSE/CNN-For-Image-Classification
-
evaluation
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
-
evaluation good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
-
evaluation
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
training
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Optimizer factoryOpentraining
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
All issues in VisionForgeCSE/CNN-For-Image-Classification
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
nautechsystems/nautilus_trader#5287 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
zerocracy/judges-action#2714 ·
Maintainers usually reply within 8 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
gofish-graphics/gofish-graphics#1135 ·
Maintainers usually reply within 1 day
-
[BUG]: Single-color requests raise ZeroDivisionError in palette helpersPossibly taken @MokiMeow claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day