Add tensor padding options to improve hardware utilization · CarperAI/trlx#284

(8 comments) (2 reactions) (0 assignees)Python (450 forks)batch import

contributions welcomefeature requestgood first issue

Repository metrics

Stars: (4,184 stars)
PR merge metrics: (30d に merged PR はありません)

説明

🚀 The feature, motivation, and pitch

Recent discussion on Twitter highlighted the importance of tensor padding to improve hardware util. NeMo seems to already support GPU-friendly vocab size padding from discussions with @cat-state but we should also consider adding an optional pad_to_multiple_of arg to our tokenizer calls with the accelerate/transformers backend to satisfy Tensor Core requirements.

The implementer should also provide system plots to display any improvements/findings.

Alternatives

No response

Additional context