facebookresearch/fairseq
GitHub ã§èŠãAdd tokens to extend embeddings during fine-tuning
Open
#2,632 opened on 2020幎9æ18æ¥
enhancementhelp wantedneeds triage
Repository metrics
- Stars
- Â (29,107 stars)
- PR merge metrics
-  (30d ã« merged PR ã¯ãããŸãã)
説æ
ð Feature Request
Like add_tokens() in huggingface/transformers (link), this will allow fine-tuning add new embeddings of tokens, which will not be sperated.
Motivation
Some NLG tasks may have non-terminals, which are expected not to be separated by subwords.