CarperAI/trlx

Add tensor padding options to improve hardware utilization

Open

#284 opened on Feb 6, 2023

View on GitHub
 (8 comments) (2 reactions) (0 assignees)Python (450 forks)batch import
contributions welcomefeature requestgood first issue

Repository metrics

Stars
 (4,184 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

🚀 The feature, motivation, and pitch

Recent discussion on Twitter highlighted the importance of tensor padding to improve hardware util. NeMo seems to already support GPU-friendly vocab size padding from discussions with @cat-state but we should also consider adding an optional pad_to_multiple_of arg to our tokenizer calls with the accelerate/transformers backend to satisfy Tensor Core requirements.

image

The implementer should also provide system plots to display any improvements/findings.

Alternatives

No response

Additional context

No response

Contributor guide

Research direction
Investigate the tokenizer calls in trlx, identify where pad to multiple of can be added to tokenizer calls with the accelerate/transformers backend, and measure hardware utilization improvements using system plots.
Tech stack
python
Domain
machine learningaibackend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Needs investigation
Prerequisites
PythonHuggingFace Transformers
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40