facebookresearch/fairseq
Add an option to disable appending eos to the end of each document while preprocessing
オープン
#5,615 opened on 2025/05/26
enhancementhelp wantedneeds triage
Repository metrics
- Stars
- (29,107 個のスター)
- PR merge metrics
- (PR metrics pending)
説明
🚀 Feature Request
Add an option to disable appending EOS to the end of each document while preprocessing
Motivation
There are instances where adding EOS is not preferred like when packing data to sequence length.
Pitch
Proposing a change to add an argument to handle this