facebookresearch/fairseq
GitHub ã§èŠãAdd an option to disable appending eos to the end of each document while preprocessing
Open
#5,615 opened on 2025幎5æ26æ¥
enhancementhelp wantedneeds triage
Repository metrics
- Stars
- Â (29,107 stars)
- PR merge metrics
-  (30d ã« merged PR ã¯ãããŸãã)
説æ
ð Feature Request
Add an option to disable appending EOS to the end of each document while preprocessing
Motivation
There are instances where adding EOS is not preferred like when packing data to sequence length.
Pitch
Proposing a change to add an argument to handle this