facebookresearch/metaseq

Unify tokenizers

Open

#308 opened on Aug 20, 2022

View on GitHub
 (1 comment) (1 reaction) (1 assignee)Python (6,195 stars) (701 forks)batch import
better-engenhancementgood first issue

Description

🚀 Feature Request

With #305, we now have two ways to specify a tokenizer: with the GPT2 tokenizer (provided as two files), and with the universal HF format (specified as one file). These are in two separate code paths, but they don't need to be: we could (manually) merge the two GPT2 files into the universal HF format and switch to only that, and we should.

The resulting code would be cleaner, but catching all the other places the old method is used (e.g. API and old sweeps) needs thorough review.

Contributor guide

Tech stack
python
Domain
backendapi
Issue type
refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Knowledge of tokenizer formatsFamiliarity with HuggingFace tokenizer
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Examine PR #305 that introduced the HuggingFace tokenizer format. Locate the two tokenizer code paths in the repository, likely in separate files or modules. Review how tokenization is used in the API and old sweeps to ensure all occurrences are updated to the unified format.