Add empty_cache for releasing GPU memory

Open
#892 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
44/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
csharp, python
Domain
api, machine-learning

Research direction

Start by running the supplied TorchSharp F# sample and tracing the existing tensor disposal and CUDA-facing APIs; the issue does not name a source file or test. Implement the requested empty_cache entry point consistently with the existing API, then verify that calling it after disposal releases cached GPU memory as observed with torch.cuda.empty_cache().

Written by the indexing model from the issue text.

Description

When I run the code sample below, my GPU monitoring seems to show that the memory that was allocated during execution (in a jupyter notebook) is still allocated despite the fact that I've done all I know to do to have intermediate tensors disposed

#r "nuget:TorchSharp-cuda-linux"
#r "nuget:TorchSharp"

open TorchSharp

let test () =
     use d = torch.NewDisposeScope()
     use tt = torch.randn(50000,50000,device=torch.device("cuda:7"))
     tt.MoveToOuterDisposeScope()

let test2() =
     use d2 = torch.NewDisposeScope()
     use ttt = test()
     ()

let empty_result = test2()

I get a similar result when I do the same experiment in python with pytorch

import torch

def test():
     tt = torch.randn(50000,50000,device=torch.device('cuda:7'))
     return ttt

def test2():
     ttt = test()
     return 0

empty_result = test2()

but I can free the memory by calling torch.cuda.empty_cache(). @NiklasGustafsson says

The underlying library does keep a high-watermark of allocated GPU memory, so even when you dispose of tensors, the overall allocation won't necessary go down. I'll see how I can get empty_cache() implemented

Dominant language
C#
Stars
1.9k
Forks
228
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dotnet/TorchSharp

All issues in dotnet/TorchSharp

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.