facebookresearch/fairscale
Question in offload.py: Moving activation to CPU does NOT reduce GPU memory.
Offen
#948 geöffnet am 04.03.2022
bughelp wantedoffload_modeltriaged
Repository-Metriken
- Stars
- (3.411 Sterne)
- PR-Merge-Metriken
- (PR-Metriken ausstehend)
Beschreibung
I use my cuda_active_bytes function to measure the GPU memory before and after the code line below. I find moving activation to CPU does NOT reduce GPU memory.
https://github.com/facebookresearch/fairscale/blob/9f347f373e32ee5cad11a40b70b8e28a74b5e2d4/fairscale/experimental/nn/offload.py#L524
def cuda_active_bytes():
torch.cuda.synchronize()
stats = torch.cuda.memory_stats()
current_active_byte = stats['active_bytes.all.current']
return current_active_byte
So actually all the activations generated by forward is still in GPU memory? If so, I think the code line above is redundant.