dmlc/dgl

CPU omp thread affinity becomes imbalanced after backpass

Open

#1,428 opened on Apr 8, 2020

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (12,665 stars) (2,928 forks)batch import
help wantedtopic: system performance

Description

🐛 Bug

I am running graphsage app in dgl+pytorch setting. In minigun code which spawns omp threads to execute e.g copyReduce, I see that multiple omp threads are getting executed on a cpu - somehow thread affinity is getting messed up. This happens only after first backpass which is creating extra threads in addition to the thread created in forward pass. For example, if I set OMP_NUM_THREADS=10 (for 10 cpus), then after the first fwd and bck pass I see there are 20 os threads. Then in the next fwd pass a subset of 10 threads out of the pool of 20 threads are used, leading to the affinity problem.

Environment

  • DGL Version (e.g., 1.0): latest commit
  • Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): pytorch
  • OS (e.g., Linux): Centos
  • How you installed DGL (conda, pip, source): compiled and installed from src code
  • Build command you used (if compiling from source): make -j
  • Python version: 3.6/3.7
  • CUDA/cuDNN version (if applicable):
  • GPU models and configuration (e.g. V100):
  • Any other relevant information: Intel CPUs

Additional context

Contributor guide

Tech stack
pythonpytorchcpp
Domain
performancebackend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
OpenMPthread affinityDGL internalsC++
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Investigate the minigun code in DGL that spawns OpenMP threads. The issue reports that after a backpass, extra threads are created leading to imbalance. Examine the forward/backward pass thread lifecycle and OpenMP thread binding. Consider implementing a thread pool or ensuring proper cleanup of threads. Look at the relevant source files in the minigun directory.