Refactor tokenizer handling in nemo.export module
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning, tooling
Research direction
Start in nemo.export.tensorrt_llm and trace how TensorRTLLM instantiates NeMo 1, NeMo 2, Hugging Face, binary, and Tiktoken tokenizers. Read sentencepiece_tokenizer.py and tiktoken_tokenizer.py, then inspect the CI checkpoint setup. Done means the tokenizer paths are simpler and CI exercises a Tiktoken-based model.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
The logic for for instantiating tokenizers in TensorRTLLM class in nemo.export.tensorrt_llm has become complex:
There is NeMo 1 and NeMo 2 checkpoint support
There are two paths: using full NeMo setup (with io.load_context) or local implementations/copies in sentencepiece_tokenizer.py and tiktoken_tokenizer.py
There are different types of tokenizers: HF, binary tokenizer.model, Tiktoken
The task is to simplify this logic (one idea is to extend all the classes above with save_pretrained method and unify constructors if possible).
There is also no way of testing a Tiktoken-based model as there is no such checkpoint in CI system. We should add it.
- Dominant language
- Python
- Stars
- 43
- Forks
- 19
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 3
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/Export-Deploy
-
nvidia-runners
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
NVIDIA-NeMo/Export-Deploy#759 ·
-
Update ED to use Dynamic Inference Engine from MCoreMay be free again @oyilmaz-nvidia claimed this 148 days ago, and no pull request is open. Open
NVIDIA-NeMo/Export-Deploy#693 · 1 assignee ·
-
Incorrect handling of reasoning traces for nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 in MBridge formatMay be free again @athitten claimed this 148 days ago, and no pull request is open. Openbug
NVIDIA-NeMo/Export-Deploy#682 · 1 comment · 1 assignee ·
-
Server returns empty answer when too many tokens requestedMay be free again @athitten claimed this 187 days ago, and no pull request is open. Openbug
NVIDIA-NeMo/Export-Deploy#652 · 1 assignee ·
-
VLM endpoint fails when there are no images in the requestMay be free again @meatybobby claimed this 219 days ago, and no pull request is open. Openbug
NVIDIA-NeMo/Export-Deploy#632 · 1 assignee ·
All issues in NVIDIA-NeMo/Export-Deploy
Similar issues
-
bug needs-triage priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
yeongseon/azure-functions-logging-python#583 ·
Maintainers usually reply within 1 day
-
documentation needs-triage priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
yeongseon/azure-functions-validation-python#546 ·
Maintainers usually reply within 1 day
-
PHP workflow streaming requests buffer the full SSE responsePossibly taken @Winoren claimed this today. Openreview: medium
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
langgenius/dify#44270 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
-
P4: low
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
jeffknupp/association#523 ·
-
bug help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
DarkStarSystems/pcons#229 ·
Maintainers usually reply within 1 day