facebookresearch/fairseq
Add an option to disable appending eos to the end of each document while preprocessing
开放
#5,615 创建于 2025年5月26日
enhancementhelp wantedneeds triage
仓库指标
- 星标
- (29,107 个星标)
- PR 合并指标
- (PR 指标待抓取)
描述
🚀 Feature Request
Add an option to disable appending EOS to the end of each document while preprocessing
Motivation
There are instances where adding EOS is not preferred like when packing data to sequence length.
Pitch
Proposing a change to add an argument to handle this