facebookresearch/fairseq

Improve handling of special tokens in Dictionary

Open

#1309 aperta il 26 ott 2019

Vedi su GitHub
 (5 commenti) (0 reazioni) (0 assegnatari)Python (6224 fork)batch import
enhancementhelp wantedstale

Metriche repository

Star
 (29.107 star)
Metriche merge PR
 (Nessuna PR mergiata in 30 g)

Descrizione

https://github.com/pytorch/fairseq/blob/eb68afca0208a040d4e91eceae86f5f22ca24b04/fairseq/data/dictionary.py#L178-L190

When loading dict.txt that already contains special tokens such as <s> or <pad> (which are added by default in sentencepiece), these tokens appear twice in the fairseq dictionary. They are added once in Dictionary.__init__() and a second time from the dict.txt file in Dictionary.add_from_file(). This causes weird behaviours e.g. when using the model in https://github.com/huggingface/transformers.

Ideally Dictionary would not add the special tokens manually when loading an external dict.txt that already contains them (such as in https://github.com/huggingface/transformers). But I am afraid that this can break backward compatibility for people who already trained models with this "duplicated special tokens bug".

For instance:

>> print([fairseq_model.task.dictionary[i] for i in range(15)])
['<s>', '<pad>', '</s>', '<unk>', '<unk>', '<s>', '</s>', ',', '▁the', ...]

In the fill_mask() method for roberta, this is what happens:

>> tokens = self.task.source_dictionary.encode_line(
       '<s> ' + text_spans_bpe,
       append_eos=True,
       add_if_not_exist=False,
   )
   print(tokens)
tensor([[    5,  1285, 32004,     2]])

With the first token 5 being the <s> that was added as a string and matched to the token from dict.txt and the last token 2 corresponding to dictionary.eos().

Guida contributor