better-engenhancementgood first issue
倉庫指標
- 星標
- (6,195 顆星)
- PR 合併指標
- (PR 指標待抓取)
描述
🚀 Feature Request
With #305, we now have two ways to specify a tokenizer: with the GPT2 tokenizer (provided as two files), and with the universal HF format (specified as one file). These are in two separate code paths, but they don't need to be: we could (manually) merge the two GPT2 files into the universal HF format and switch to only that, and we should.
The resulting code would be cleaner, but catching all the other places the old method is used (e.g. API and old sweeps) needs thorough review.