facebookresearch/metaseq

Track all of our RNG offsets to avoid collisions

Open

#65 建立於 2022年5月8日

在 GitHub 查看
 (0 留言) (0 反應) (0 負責人)Python (6,195 star) (701 fork)batch import
enhancementgood first issue

描述

We have RNG seed offsets sprinkled through the codebase:

(base) √ metaseq % ag seed --py | grep +
cpu_tests/test_streaming_token_block_dataset.py:78:        shadow_rng = np.random.default_rng(2273 + seed)
cpu_tests/test_streaming_token_block_dataset.py:124:        shadow_rng = np.random.default_rng(2273 + seed)
metaseq/tasks/language_modeling.py:217:            with data_utils.numpy_seed(self.args.seed + epoch):
metaseq/tasks/streaming_language_modeling.py:316:            seed=1284 + self.args.seed,
metaseq/trainer.py:1052:        seed = self.cfg.common.seed + self.get_num_updates()
metaseq/data/streaming_token_block_dataset.py:96:            rng = np.random.default_rng(2273 + self.seed)
metaseq/data/iterators.py:524:                batches = shuffle_batches(list(batches), self.seed + epoch)
metaseq/data/iterators.py:532:                batches = shuffle_batches(batches, self.seed + epoch + self.shard_id)
metaseq/data/iterators.py:535:                batches = shuffle_batches(list(self.frozen_batches), self.seed + epoch)

Would be good to track these offset to avoid collisions, in cases we're assuming no collision/coupling via offsets.

貢獻者指南

技術棧
pythonnumpy
領域
machine learningdata
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
half day
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
PythonNumPy
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
50
研究方向
First, use grep or similar to find all RNG seed offsets in the codebase, building on the examples provided. Then, design a centralized registry (e.g., a dictionary or config) to store and avoid collisions. Review files such as cpu tests/test streaming token block dataset.py, metaseq/tasks/language modeling.py, and metaseq/trainer.py to understand how offsets are currently used. Finally, propose a pull request that implements the tracking mechanism.