bytedance/byteps

How did you get the horovod & bytePS performance

Open

Aperta il 18 lug 2019

Vedi su GitHub
 (12 commenti) (2 reazioni) (0 assegnatari)Python (3483 star) (476 fork)batch import
distributeddocumentationgood first issue

Descrizione

I have the same hardware envs, same network, but I could not get the result as you, almost half as you. Any best practices and experience? thanks very much! for bytePS with 1 instance and 8 GPU, I have similar testing result.

Guida contributor

Tech stack
python
Dominio
machine learningperformance
Tipo issue
performance
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
4
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
over 1 week
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
stale
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
unclear
Prerequisiti
distributed trainingBytePSHorovodGPU cluster
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
10
Direzione di ricerca
The issue is a open ended query about reproducing BytePS performance benchmarks. To assist, first clarify the user's exact hardware, software versions, and network topology. Then, review BytePS configuration files and compare with published benchmarks. Check for common pitfalls like NCCL settings, network latency, or GPU affinity. Provide step by step tuning guidance and recommend running a simple allreduce test to isolate the bottleneck.