[Benchmark] RTX 4070 Ti 12 GB (sm_89) + 64 GB DDR4, IQ3_S, 128K int8, images on CPU, Windows 10, engine 0.1.50

Chiusa Adatta ai principianti
  • Qualcosa non va?
#4,234 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
1/5
Tempo stimato
Meno di un'ora
Idoneità per principianti
62/100
Tipo di issue
Documentazione
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
machine-learning

Direzione di ricerca

Il rapporto cita tools/strata_bench.py e afferma di aver eseguito la suite 1 con tre esecuzioni per test. Leggi lo script per capire il formato del benchmark, poi confronta questo rapporto con altri benchmark della community. Il lavoro è completato quando l’hardware, il software, la configurazione del modello, i tempi e i limiti dichiarati sono documentati chiaramente; non è stata eseguita alcuna verifica della correttezza.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Community benchmark on NVIDIA GeForce RTX 4070 Ti

Measured on 2026-10-26 with tools/strata_bench.py (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

Hardware and software

  • GPU and VRAM: NVIDIA GeForce RTX 4070 Ti, 12 GiB VRAM
  • CPU: 13th Gen Intel(R) Core(TM) i5-13600KF (14 cores, 20 threads); installed RAM: 63.8 GiB; storage: not measured
  • OS: Windows 10 (10.0.19045); GPU driver: 621.01; CUDA: not read
  • Strata: project 0.1.51, commit 4b9e05a2; engine 0.1.50
  • Background workloads and power limits: not recorded

Model and configuration

  • Model and quantization: qwen3.8-flash-next-iq3_s
  • Context: 131072 tokens; parallel slots: 1
  • Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

Method

Run by python tools/strata_bench.py --long against the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

  • short chat: the three questions of tools/ab_engine.py, up to 100 new tokens, 3 runs
  • 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each
  • agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (cached_tokens) and timings (prompt_n)
  • concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

Results

Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range -- | -- | -- | -- | -- | -- | -- | -- short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 50.2 (44.7-60.3) | 0.77 (0.65-0.82) 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1155.7 (856.2-1160.2) | not measured | 3.50 (3.49-4.73) 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1441.7 (1383.3-1444.6) | not measured | 11.17 (11.15-11.64) 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 1443.8 (1443.1-1445.9) | not measured | 22.25 (22.22-22.25) agent session, follow-up turns | 12762 (12744-12787) at the last turn | see below | up to 64 | 3 | see below | 56.2 (46.9-68.3) | 3.06 (2.71-5.12)

Per-request data: results.json. Engine log lines from the start: engine-log-excerpt.txt (when this run started the server).

Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, nvidia-smi/rocm-smi and the OS; background load and power limits are not recorded. Total time of the suite: 223 s.

Additional context (not measured by the tool)

  • RAM: 4 x 16 GB DDR4 (Corsair CMW32GX4M2Z3600C18), XMP running at 3400 MHz CL18, 1.35 V, two DIMMs per channel (3600 MHz was not stable with this CPU's memory controller).
  • Storage: Lexar NM790 2 TB, NVMe PCIe 4.0.
  • Configuration: IQ3_S, every expert in RAM (no low-RAM mode, arena_mib 47962), --kv int8, 131072 context, --kv-streaming off (KV cache in VRAM), images on with the CPU encoder (--vision cpu, --vision-tokens 1024), French draft vocabulary (--draft-vocab fr), one slot.
  • Calibration: --calibrate run on this engine after a reboot: --pcie-frac 0.35 --spec-min-p 0.70, adaptive tier at its default. Pool workers at the engine default (9 on this 6P + 8E CPU); an interleaved A/B of 9 against 13 workers (pool_cores 2,4,6,8,10,12-19, 2 rounds) gave +0.3 %, no difference.
  • GPU package: an RTX 40 card (sm_89) running the sm_86 package, as from 0.2.5.
  • Background: idle desktop, no other GPU work during the run; power limits at stock.
Lingua principale
C++
Stelle
11.6k
Fork
1k
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Niko1221/Strata

Tutte le issue di Niko1221/Strata

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.