Some device runs take one of two speeds from process to process, unexplained

Aperta
  • Qualcosa non va?
#366 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
18/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
performance

Direzione di ricerca

Start with docs/dev/benchmarks.md and docs/dev/benchmarking.md, which hold the A40 and MI325X timings and the three-repeats-per-process method. The work is measurement: run one affected case in at least 20 fresh processes on each device at the current tree and record the wall distribution with case and date. Done when each device has one mode within 10 %, or the cause is found and the history accounts for it.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: benchmarks area: gpu performance priority: low
What happens

Some device runs take one of two speeds from process to process, and nobody knows why.

  • On an NVIDIA A40, main ran some cases at one of two speeds, so #59's rdw and radial_filament_256 ratios of 0.417 and 0.753 used the faster of two jobs. The final tree's direct ratios were 0.403 and 0.392 (2026-10).
  • On an AMD MI325X, an earlier HIP build's he_rdw_averaged walls were bimodal, 0.20 or 0.37 s.

Whether the current tree still does it is not known. Read, not run. #81 covers a similar symptom only for gnlse_raman_shock on CPU.

Why it matters

The performance history times three repeats in one process per configuration, so a mode set per process reads as a speed change between releases, by a factor of up to 1.85 on the numbers above.

To do
  • Run one affected case in at least 20 fresh processes on an A40 and an MI325X at the current tree. Plot the walls.
  • If two modes persist, look at memory placement, host-to-device transfers and device-graph capture. These are guesses, not findings.
  • Have the history flag a device row that may sit in the slower mode, or take repeats from more than one process on devices. #138 changes the same history's reading.
Done when

On each device the distribution over at least 20 processes is recorded with case and date, and either it has one mode within 10 %, or the cause is found and the history accounts for it.

Checked against commit 8a9f764.

Lingua principale
Python
Stelle
3
Fork
3
Merge medio
6h 28m
PR unite (30g)
126

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di LupoLab/solprop

Tutte le issue di LupoLab/solprop

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.