notadamking/RLTrader

Improve Utilization of GPU

Open

#10 opened on Jun 6, 2019

View on GitHub
 (14 comments) (3 reactions) (1 assignee)Python (1,592 stars) (535 forks)batch import
enhancementgood first issuehelp wanted

Description

This library achieves very high success rates, though it takes a very long time to optimize and train. This could be improved if we could figure out a way to utilize the GPU more during optimization/training, so the CPU can be less of a bottleneck. Currently, the CPU is being used for most of the intermediate environment calculations, while the GPU is used within the PPO2 algorithm during policy optimization.

I am currently optimizing/training on the following hardware:

  • AMD Threadripper 1920X 12 Core (24 Thread) CPU
  • Nvidia RTX 2080 8GB GPU
  • 16 GB 3000 Mhz RAM

The bottleneck on my system is definitely the CPU, which is surprising as this library takes advantage of the multi-threaded benefits of the Threadripper, and my GPU is staying around 1-10% utilization. I have some ideas on how this could be improved, but would like to start a conversation.

  1. Increase the size of the policy network (i.e. increase the number of hidden layers or increase the number of nodes in each layer)

  2. Do less work in each training loop, so the GPU loop is called more often.

I would love to hear what you guys think. Any ideas or knowledge is welcome to be shared here.

Contributor guide

Tech stack
pythontensorflow
Domain
machine learningai
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PythonTensorFlowReinforcement learning basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Review the training loop implementation in the repository, focusing on how the environment steps are computed on CPU and how the PPO2 algorithm is executed on GPU. Use profiling tools to measure CPU and GPU utilization. Explore techniques such as increasing batch sizes, parallel environment instances, or using TensorFlow's tf.data API to reduce CPU GPU synchronization overhead. The existing comments in issue #10 discuss potential approaches.