facebookresearch/fairseq
Use symbolic link for saving best/last checkpoints
オープン
#5,490 opened on 2024/04/23
enhancementhelp wantedneeds triage
Repository metrics
- Stars
- (29,107 個のスター)
- PR merge metrics
- (PR metrics pending)
説明
🚀 Feature Request
When preparing checkpoint_best.pt and checkpoint_last.pt, create symbolic link instead of making exact copy of the checkpoint
Motivation
I'm running on a machine that somehow has bad io performance, writing the same file 3 times seems extremely inefficient.
Seems that there is plan to implement a asynchronous copying but i believe a symlink is much more efficient https://github.com/facebookresearch/fairseq/blob/bedb259bf34a9fc22073c13a1cee23192fa70ef3/fairseq/checkpoint_utils.py#L128-L141