pytorch/vision

ROCM version is slow

Open

Aperta il 17 set 2020

Vedi su GitHub
 (7 commenti) (0 reazioni) (0 assegnatari)Python (15.050 star) (6858 fork)batch import
help wantedmodule: rocm

Descrizione

🐛 Bug

Inference speed is abysmal - way slower than cpu. It works as expected on NVidia. I'm not really sure whether this is torch vision or torch itself at fault.

To Reproduce

Steps to reproduce the behavior:

  1. Dockerfile: https://gist.github.com/boniek83/0bbf858a24961816557c522007a11c11
  2. Lab: https://github.com/timesler/facenet-pytorch/blob/master/examples/infer.ipynb
  3. Data for lab: https://github.com/timesler/facenet-pytorch/tree/master/data/test_images

Expected behavior

Performance should be competitive.

Environment

  • PyTorch / torchvision Version (e.g., 1.0 / 0.4.0): 1.6.0 / 0.7.0
  • OS (e.g., Linux): CentOS 7.8
  • How you installed PyTorch / torchvision (conda, pip, source): source
  • Build command you used (if compiling from source): see Dockerfile above
  • Python version: 3.6
  • GPU models and configuration: Radeon VII
  • Any other relevant information: ROCK 3.7 during runtime

cc @iotamudelta @ashishfarmer

Guida contributor

Tech stack
pythonpytorch
Dominio
performancemachine learning
Tipo issue
bug
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
4
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
over 1 week
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
stale
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
needs investigation
Prerequisiti
Understanding of PyTorch internalsExperience with AMD ROCm stackPerformance profiling skills
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
10
Direzione di ricerca
The issue reports slow inference on AMD ROCm compared to CPU and Nvidia GPUs. Investigate whether the bottleneck is in torchvision or PyTorch core by profiling ROCm kernels. The user provided a Dockerfile and a facenet notebook for reproduction. First, attempt to reproduce the performance issue in a controlled environment. Then compare execution traces with and without ROCm to identify underperforming operations. Look for recent fixes or discussions in PyTorch's ROCm GitHub issues and relevant modules like torchvision's ops.