pytorch/vision

ROCM version is slow

Open

#2,682 opened on Sep 17, 2020

View on GitHub
 (7 comments) (0 reactions) (0 assignees)Python (15,050 stars) (6,858 forks)batch import
help wantedmodule: rocm

Description

🐛 Bug

Inference speed is abysmal - way slower than cpu. It works as expected on NVidia. I'm not really sure whether this is torch vision or torch itself at fault.

To Reproduce

Steps to reproduce the behavior:

  1. Dockerfile: https://gist.github.com/boniek83/0bbf858a24961816557c522007a11c11
  2. Lab: https://github.com/timesler/facenet-pytorch/blob/master/examples/infer.ipynb
  3. Data for lab: https://github.com/timesler/facenet-pytorch/tree/master/data/test_images

Expected behavior

Performance should be competitive.

Environment

  • PyTorch / torchvision Version (e.g., 1.0 / 0.4.0): 1.6.0 / 0.7.0
  • OS (e.g., Linux): CentOS 7.8
  • How you installed PyTorch / torchvision (conda, pip, source): source
  • Build command you used (if compiling from source): see Dockerfile above
  • Python version: 3.6
  • GPU models and configuration: Radeon VII
  • Any other relevant information: ROCK 3.7 during runtime

cc @iotamudelta @ashishfarmer

Contributor guide

Tech stack
pythonpytorch
Domain
performancemachine learning
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
Understanding of PyTorch internalsExperience with AMD ROCm stackPerformance profiling skills
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
10
Research direction
The issue reports slow inference on AMD ROCm compared to CPU and Nvidia GPUs. Investigate whether the bottleneck is in torchvision or PyTorch core by profiling ROCm kernels. The user provided a Dockerfile and a facenet notebook for reproduction. First, attempt to reproduce the performance issue in a controlled environment. Then compare execution traces with and without ROCm to identify underperforming operations. Look for recent fixes or discussions in PyTorch's ROCm GitHub issues and relevant modules like torchvision's ops.