[Performance] nemotron-hindi-4b latency is 17.0% above reference
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- machine-learning, performance
Research direction
No source file or test is named. Profile prefill, decode, KV-cache access, and synchronization on NVIDIA Thor X using the stated BF16 batch-1 workload, comparing warmed iterations with the reference. Done means identifying and correcting the regression while preserving exact token IDs and the existing performance gate.
Written by the indexing model from the issue text.
Description
This was generated by AI during triage.
Summary
nemotron-hindi-4b completes aligned BF16 generation and matches exact token IDs, but TRTMC latency is 17.0% above the reference.
Observed behavior
- Precision: TRTMC BF16, reference BF16
- Output contract: exact token IDs (matched)
- Request: batch 1, 20 new tokens
- Measurement: 3 warmups followed by 10 iterations
- Candidate p50: 717.17 ms
- Reference p50: 612.85 ms
- Regression: +17.0% (allowed margin: 5%)
The ten measured iterations are stable enough that this is not a single-sample outlier.
Expected behavior
The generation path should remain within the registered performance margin after warmup.
Acceptance criteria
- Profile prefill, decode, KV-cache access, and synchronization.
- Identify and fix the measured regression.
- Preserve exact-token output and the existing performance gate.
Reproduction platform
- Hardware: NVIDIA Thor X
- Architecture: aarch64
- TensorRT: 11.0
- GPU carveout: 48 GiB
- Dominant language
- Python
- Stars
- 254
- Forks
- 58
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 201
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/TensorRT-Model-Connect
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
NVIDIA/TensorRT-Model-Connect#1126 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
NVIDIA/TensorRT-Model-Connect#1125 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
NVIDIA/TensorRT-Model-Connect#1401 · 1 comment ·
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 68/100
NVIDIA/TensorRT-Model-Connect#1392 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
NVIDIA/TensorRT-Model-Connect#1388 · 1 comment ·
All issues in NVIDIA/TensorRT-Model-Connect
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100