facebookresearch/fairscale
Question in offload.py: Moving activation to CPU does NOT reduce GPU memory.
Aberta
#948 aberto em 4 de mar. de 2022
bughelp wantedoffload_modeltriaged
Métricas do repositório
- Stars
- (3.411 estrelas)
- Métricas de merge de PR
- (Métricas PR pendentes)
Description
I use my cuda_active_bytes function to measure the GPU memory before and after the code line below. I find moving activation to CPU does NOT reduce GPU memory.
https://github.com/facebookresearch/fairscale/blob/9f347f373e32ee5cad11a40b70b8e28a74b5e2d4/fairscale/experimental/nn/offload.py#L524
def cuda_active_bytes():
torch.cuda.synchronize()
stats = torch.cuda.memory_stats()
current_active_byte = stats['active_bytes.all.current']
return current_active_byte
So actually all the activations generated by forward is still in GPU memory? If so, I think the code line above is redundant.