feat: save the model and stop training based on `exit-duration-in-mins`

Open
#67 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
machine-learning

Research direction

Review related issues #37 and #42, then compare the requested behavior with Megatron-DeepSpeed's exit-duration-in-mins argument. Identify the training entry point and checkpoint-saving path before deciding how the configured duration should trigger a final save and clean stop; done means training stops at the limit without losing the model.

Written by the indexing model from the issue text.

Description

As discussed a long time ago in a meeting it would be really great if we had a feature to save the model and stop training after a certain time as the jobs on the JZ cluster are limited to 20 hours.

For example, in the architecture and scaling working group, they added the exit-duration-in-mins argument the library used to run trainings Megatron-DeepSpeed

related: #37 (#42)

Dominant language
Python
Stars
29
Forks
11
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from bigscience-workshop/metadata

All issues in bigscience-workshop/metadata

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.