Benchmark: Warmup seems to spoil first round results
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 62/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- performance, tooling
Research direction
The benchmark is in tools/strata_bench.py; the warmup that uses PROMPT[0] is around line 258, and the chat section prints per-run decode tok/s and first-token latency. Read how warmup is invoked before the timed runs, then run the chat benchmark and compare run 1 against run 2 for each question. Done when the first round no longer runs measurably faster than later rounds, or when the warmup choice is justified in the code.
Written by the indexing model from the issue text.
Description
I realize this is probably hard to fix without invalidating existing benchmarks...
I'm seeing the first benchmark round in chat being consistently way faster:
chat: 3 short questions x 21 runs, up to 100 new tokens
run 1 q1: decode 100.6 tok/s, first token 0.387 s, 100 tokens
run 1 q2: decode 75.3 tok/s, first token 0.382 s, 100 tokens
run 1 q3: decode 77.2 tok/s, first token 0.413 s, 100 tokens
run 2 q1: decode 84.6 tok/s, first token 0.43 s, 100 tokens
run 2 q2: decode 83.1 tok/s, first token 0.369 s, 100 tokens
run 2 q3: decode 84.3 tok/s, first token 0.383 s, 100 tokens
chat: 3 short questions x 21 runs, up to 100 new tokens
run 1 q1: decode 100.4 tok/s, first token 0.381 s, 100 tokens
run 1 q2: decode 77.8 tok/s, first token 0.366 s, 100 tokens
run 1 q3: decode 80.0 tok/s, first token 0.399 s, 100 tokens
run 2 q1: decode 89.9 tok/s, first token 0.413 s, 100 tokens
chat: 3 short questions x 3 runs, up to 100 new tokens
run 1 q1: decode 96.2 tok/s, first token 0.409 s, 100 tokens
run 1 q2: decode 72.2 tok/s, first token 0.39 s, 100 tokens
run 1 q3: decode 75.9 tok/s, first token 0.435 s, 100 tokens
Looking at the code, it warms up with PROMPT[0], which I'd suspect is the reason:
- Dominant language
- C++
- Stars
- 11.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Niko1221/Strata
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Niko1221/Strata#3524 · 2 comments ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 70/100
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Niko1221/Strata#3512 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 76/100
Niko1221/Strata#3501 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
category:port-update
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 2 days
-
(S1 - Need confirmation)
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
CleverRaven/Cataclysm-DDA#89194 ·
Maintainers usually reply within 1 day
-
reproduced
Difficulty 1/5 Under an hour Newbie friendliness 67/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day