facebookresearch/fairseq

Improve handling of special tokens in Dictionary

Open

#1,309 opened on Oct 26, 2019

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Python (6,224 forks)batch import
enhancementhelp wantedstale

Repository metrics

Stars
 (29,107 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

https://github.com/pytorch/fairseq/blob/eb68afca0208a040d4e91eceae86f5f22ca24b04/fairseq/data/dictionary.py#L178-L190

When loading dict.txt that already contains special tokens such as <s> or <pad> (which are added by default in sentencepiece), these tokens appear twice in the fairseq dictionary. They are added once in Dictionary.__init__() and a second time from the dict.txt file in Dictionary.add_from_file(). This causes weird behaviours e.g. when using the model in https://github.com/huggingface/transformers.

Ideally Dictionary would not add the special tokens manually when loading an external dict.txt that already contains them (such as in https://github.com/huggingface/transformers). But I am afraid that this can break backward compatibility for people who already trained models with this "duplicated special tokens bug".

For instance:

>> print([fairseq_model.task.dictionary[i] for i in range(15)])
['<s>', '<pad>', '</s>', '<unk>', '<unk>', '<s>', '</s>', ',', '▁the', ...]

In the fill_mask() method for roberta, this is what happens:

>> tokens = self.task.source_dictionary.encode_line(
       '<s> ' + text_spans_bpe,
       append_eos=True,
       add_if_not_exist=False,
   )
   print(tokens)
tensor([[    5,  1285, 32004,     2]])

With the first token 5 being the <s> that was added as a string and matched to the token from dict.txt and the last token 2 corresponding to dictionary.eos().

Contributor guide