Question: Why not padding to the same sequence length within the batch during the sft training phase?
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start by reading data_utils.py and the create_dataset_split function, focusing on how SFT samples are tokenized and padded. Trace how those tensors enter the training batches, then determine whether batch-level padding is supported; done means documenting the reason for the current max-length padding or defining the changes needed for dynamic padding.
Written by the indexing model from the issue text.
Description
Question: In the SFT training phase in dschat, I found that function create_dataset_split in data_utils.py will pad the samples to the maximum length. Therefore, why not dynamically padding to the maximum length of samples in the batch during training, which can significantly speed up training.
if chosen_sentence is not None:
chosen_sentence += end_of_conversation_token
chosen_token = tokenizer(chosen_sentence,
max_length=max_seq_len,
padding="max_length",
truncation=True,
return_tensors="pt")
chosen_token["input_ids"] = chosen_token["input_ids"].squeeze(0)
chosen_token["attention_mask"] = chosen_token["attention_mask"].squeeze(0)
chosen_dataset.append(chosen_token)
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepspeedai/DeepSpeedExamples
-
Difficulty 2/5 1-3 hours Newbie friendliness 52/100
deepspeedai/DeepSpeedExamples#996 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
deepspeedai/DeepSpeedExamples#995 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 52/100
deepspeedai/DeepSpeedExamples#989 ·
-
moe example 404 Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
deepspeedai/DeepSpeedExamples#984 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
deepspeedai/DeepSpeedExamples#979 · 6 comments ·
All issues in deepspeedai/DeepSpeedExamples
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
triage/confirmed
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
agentscope-ai/agentscope#2775 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
comp/desktop P3 type/bug
Difficulty 1/5 Under an hour Newbie friendliness 92/100
NousResearch/hermes-agent#118866 ·