mlcommons/algorithmic-efficiency

Skip eval on train and test for self-reporting results

Open

#725 opened on Mar 26, 2024

View on GitHubContributor guide
 (2 comments) (0 reactions) (0 assignees)Python (76 forks)auto 404
Good First Issue✨ Feature Request

Repository metrics

Stars
 (422 stars)
PR merge metrics
 (PR metrics pending)

Description

Feature request: allow users to skip eval on train and test

Evaluating on the training and test sets is time-consuming and not necessary for self-reporting results. We should add a flag that allow the user to skip eval on these datasets, to make scoring faster.

Accordingly, in this scenario we should modify:

goals_reached = (
              train_state['validation_goal_reached'] and
              train_state['test_goal_reached'])

into:

goals_reached = (train_state['validation_goal_reached'])

This would speed up self-evalution even more, by stopping training when validation target is reached, avoiding unnecessary usage of computational resources.

Contributor guide

Research direction
Inspect the training loop and find where evaluation on train and test datasets is performed. Look for existing command line flags and add a new flag to skip those evaluations. Update the goals reached condition to only use validation goal reached when the flag is set. Run the training script with help to ensure the flag appears.
Tech stack
python
Domain
machine learning
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitPython
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75