Improve handling of special tokens in Dictionary
#1.309 geöffnet am 26. Okt. 2019
Repository-Metriken
- Stars
- (29.107 Stars)
- PR-Merge-Metriken
- (Keine gemergten PRs in 30 T)
Beschreibung
When loading dict.txt that already contains special tokens such as <s> or <pad> (which are added by default in sentencepiece), these tokens appear twice in the fairseq dictionary.
They are added once in Dictionary.__init__() and a second time from the dict.txt file in Dictionary.add_from_file().
This causes weird behaviours e.g. when using the model in https://github.com/huggingface/transformers.
Ideally Dictionary would not add the special tokens manually when loading an external dict.txt that already contains them (such as in https://github.com/huggingface/transformers).
But I am afraid that this can break backward compatibility for people who already trained models with this "duplicated special tokens bug".
For instance:
>> print([fairseq_model.task.dictionary[i] for i in range(15)])
['<s>', '<pad>', '</s>', '<unk>', '<unk>', '<s>', '</s>', ',', '▁the', ...]
In the fill_mask() method for roberta, this is what happens:
>> tokens = self.task.source_dictionary.encode_line(
'<s> ' + text_spans_bpe,
append_eos=True,
add_if_not_exist=False,
)
print(tokens)
tensor([[ 5, 1285, 32004, 2]])
With the first token 5 being the <s> that was added as a string and matched to the token from dict.txt and the last token 2 corresponding to dictionary.eos().