facebookresearch/fairseq

Invalid suffix of raw dataset when preprocessing without language

オープン

#1,426 opened on 2019/11/25

 (2 件のコメント) (0 件のリアクション) (0 人の担当者)Python (6,224 件のフォーク)batch import
bughelp wanted

Repository metrics

Stars
 (29,107 個のスター)
PR merge metrics
 (30d に merged PR はありません)

説明

When preprocessing using --dataset-impl raw and no source and target languages are specified, the datasets are stored under train.None-None due to this line:

https://github.com/pytorch/fairseq/blob/5349052aae4ec1350822c894fbb6be350dff61a0/preprocess.py#L218

Is this expected behavior or can we remove this suffix?

コントリビューターガイド