Support for multiple GPUs

Open
  • Is something wrong?
#134 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
java, python, scala, spark, tensorflow
Domain
data-engineering, distributed-systems, machine-learning

Research direction

The issue names no implementation files or tests. Start with the linked TensorFlow Java Session API and compare its session configuration with the Python visible_device_list example; done means TensorFrames tasks can run concurrently on separate available GPUs.

Written by the indexing model from the issue text.

Description

enhancement

TensorFrames should be able to run tasks concurrently on separate GPUs, if they are available. This should be possible with the latest TensorFlow java API:
https://www.tensorflow.org/api_docs/java/reference/org/tensorflow/Session.html#Session(org.tensorflow.Graph, byte[])

Assuming that all the GPUs are available for running tasks, they can be locked onto specific sessions in python by using the following code:

config = tf.ConfigProto()
config.gpu_options.visible_device_list = "0" # Or different values
session = tf.Session(config=config)
Dominant language
Scala
Stars
743
Forks
158
No merged PRs in 30d

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/tensorframes

All issues in databricks/tensorframes

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.