facebookresearch/fairscale
Question in offload.py: Moving activation to CPU does NOT reduce GPU memory.
オープン
#948 opened on 2022/03/04
bughelp wantedoffload_modeltriaged
Repository metrics
- Stars
- (3,411 個のスター)
- PR merge metrics
- (PR metrics pending)
説明
I use my cuda_active_bytes function to measure the GPU memory before and after the code line below. I find moving activation to CPU does NOT reduce GPU memory.
https://github.com/facebookresearch/fairscale/blob/9f347f373e32ee5cad11a40b70b8e28a74b5e2d4/fairscale/experimental/nn/offload.py#L524
def cuda_active_bytes():
torch.cuda.synchronize()
stats = torch.cuda.memory_stats()
current_active_byte = stats['active_bytes.all.current']
return current_active_byte
So actually all the activations generated by forward is still in GPU memory? If so, I think the code line above is redundant.