Can't this model be trained using multi nodes and multiple cards ?

Abierto

#3 abierto el 4/5/2023 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad: 5/5
Tiempo estimado: Más de una semana
Aptitud para principiantes: 25/100

Tipo de issue: Nueva funcionalidad
Claridad: Necesita aclaración
Estado de actividad: Estancado
Stack tecnológico: python
Área: distributed-systems, machine-learning

Línea de trabajo

Start with train.py and the multinode_runner.py and DeepSpeed launcher commands shown in the logs. Reproduce the 16B-model run across the listed nodes and inspect where memory use remains per GPU. Done would require a documented or implemented way to train the model across multiple machines and cards without OOM.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

My machine has a single node 4-card 16G graphics memory, and running the 16B model with multiple nodes will result in OOM regardless of how the number of nodes is set. Can this model be trained with multiple machines and cards
Here are some logs of the training：

2023-05-04 16:21:02,954] [INFO] [multinode_runner.py:65:get_cmd] Running on the following workers: c05r2n11,c05r2n16,c05r2n19,c05r3n02,c06r4n06,c06r4n09,c06r4n12,c06r4n17
[2023-05-04 16:21:02,955] [INFO] [runner.py:453:main] cmd = pdsh -f 1024 -w c05r2n11,c05r2n16,c05r2n19,c05r3n02,c06r4n06,c06r4n09,c06r4n12,c06r4n17 export NCCL_SOCKET_IFNAME=ib0; export UCX_MAX_EAGER_LANES=4; export UCX_MAX_RNDV_LANES=4; export UCX_ZCOPY_THRESH=auto; export UCX_RNDV_THRESH=auto; export UCX_DC_MLX5_NUM_DCI=16; export NCCL_IB_HCA=mlx5_0; export NCCL_DEBUG=info; export PYTHONPATH=/work/home/actvg1ue59/CodeGen-main; cd /work/home/actvg1ue59/CodeGen-main; /work/home/actvg1ue59/miniconda3/envs/torch/bin/python -u -m deepspeed.launcher.launch --world_info=eyJjMDVyMm4xMSI6IFswLCAxLCAyLCAzXSwgImMwNXIybjE2IjogWzAsIDEsIDIsIDNdLCAiYzA1cjJuMTkiOiBbMCwgMSwgMiwgM10sICJjMDVyM24wMiI6IFswLCAxLCAyLCAzXSwgImMwNnI0bjA2IjogWzAsIDEsIDIsIDNdLCAiYzA2cjRuMDkiOiBbMCwgMSwgMiwgM10sICJjMDZyNG4xMiI6IFswLCAxLCAyLCAzXSwgImMwNnI0bjE3IjogWzAsIDEsIDIsIDNdfQ== --node_rank=%n --master_addr=c05r2n11 --master_port=9901 train.py
c05r3n02: Warning: Permanently added 'c05r3n02,10.3.5.43' (ECDSA) to the list of known hosts.^M
c05r2n16: Warning: Permanently added 'c05r2n16,10.3.5.37' (ECDSA) to the list of known hosts.^M
c05r2n19: Warning: Permanently added 'c05r2n19,10.3.5.40' (ECDSA) to the list of known hosts.^M
c06r4n06: Warning: Permanently added 'c06r4n06,10.3.6.67' (ECDSA) to the list of known hosts.^M
c06r4n09: Warning: Permanently added 'c06r4n09,10.3.6.70' (ECDSA) to the list of known hosts.^M
c06r4n12: Warning: Permanently added 'c06r4n12,10.3.6.73' (ECDSA) to the list of known hosts.^M
c06r4n17: Warning: Permanently added 'c06r4n17,10.3.6.78' (ECDSA) to the list of known hosts.^M
c05r2n11: Currently Loaded Modulefiles:
c05r2n11: 1) compiler/devtoolset/7.3.1 3) compiler/dtk/22.10.1
c05r2n11: 2) mpi/hpcx/gcc-7.3.1
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:96:main] 0 NCCL_SOCKET_IFNAME=ib0
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:96:main] 0 NCCL_IB_HCA=mlx5_0
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:96:main] 0 NCCL_DEBUG=info
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=0
c05r2n11: [2023-05-04 16:21:07,884] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n11: [2023-05-04 16:21:07,884] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n11: [2023-05-04 16:21:07,884] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=1
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:96:main] 7 NCCL_IB_HCA=mlx5_0
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 2 NCCL_IB_HCA=mlx5_0
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:96:main] 6 NCCL_DEBUG=info
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:96:main] 7 NCCL_DEBUG=info
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 2 NCCL_DEBUG=info
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=4
c06r4n09: [2023-05-04 16:21:36,546] [INFO] [launch.py:96:main] 5 NCCL_IB_HCA=mlx5_0
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 3 NCCL_IB_HCA=mlx5_0
c06r4n09: [2023-05-04 16:21:36,546] [INFO] [launch.py:96:main] 5 NCCL_DEBUG=info
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=6
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 3 NCCL_DEBUG=info
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=2
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=3
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=7
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:123:main] dist_world_size=32
c05r3n02: [2023-05-04 16:21:36,532] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,532] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n09: [2023-05-04 16:21:36,555] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c06r4n09: [2023-05-04 16:21:36,555] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=5
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=1
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:96:main] 7 NCCL_IB_HCA=mlx5_0
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 2 NCCL_IB_HCA=mlx5_0
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:96:main] 6 NCCL_DEBUG=info
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:96:main] 7 NCCL_DEBUG=info
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 2 NCCL_DEBUG=info
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=4
c06r4n09: [2023-05-04 16:21:36,546] [INFO] [launch.py:96:main] 5 NCCL_IB_HCA=mlx5_0
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 3 NCCL_IB_HCA=mlx5_0
c06r4n09: [2023-05-04 16:21:36,546] [INFO] [launch.py:96:main] 5 NCCL_DEBUG=info
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=6
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 3 NCCL_DEBUG=info
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=2
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=3
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=7
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:123:main] dist_world_size=32
c05r3n02: [2023-05-04 16:21:36,532] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,532] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n09: [2023-05-04 16:21:36,555] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c06r4n09: [2023-05-04 16:21:36,555] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=5
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 946, in load_checkpoint_in_model
c05r2n11: load_checkpoint_in_model(
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 946, in load_checkpoint_in_model
c05r2n11: load_checkpoint_in_model(
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 946, in load_checkpoint_in_model
c05r2n11: set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
c05r2n11: set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
c05r2n11: set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
c05r2n11:
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
c05r2n11: new_value = value.to(device)new_value = value.to(device)
c05r2n11:
c05r2n11: new_value = value.to(device)
c05r2n11: new_value = value.to(device)
c05r2n11: RuntimeErrorRuntimeError: : HIP out of memory. Tried to allocate 576.00 MiB (GPU 3; 15.98 GiB total capacity; 15.83 GiB already allocated; 0 bytes free; 15.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONFHIP out of memory. Tried to allocate 576.00 MiB (GPU 1; 15.98 GiB total capacity; 15.83 GiB already allocated; 0 bytes free; 15.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONF
c05r2n11: RuntimeError
c05r2n11: RuntimeError: : HIP out of memory. Tried to allocate 576.00 MiB (GPU 2; 15.98 GiB total capacity; 15.83 GiB already allocated; 0 bytes free; 15.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONFHIP out of memory. Tried to allocate 576.00 MiB (GPU 0; 15.98 GiB total capacity; 15.83 GiB already allocated; 0 bytes free; 15.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONF

Lenguaje dominante: Python
Estrellas: 268
Forks: 11
Métricas de merge de PR: Sin PR fusionados en 30 d

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Primeros pasos

Lee el issue completo y luego la guía de contribución del proyecto.
Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
Haz un fork del repositorio y trabaja en una rama.
Abre un pull request que haga referencia al número del issue.

Más de salesforce/CodeGen2

Question of unk, bos, and eos tokens. Abierto

Dificultad 4/5 3-5 días Aptitud para principiantes 25/100

salesforce/CodeGen2#11 · Abierto el 20/7/2023
Question of eos token in codegen2-7B model Abierto

Dificultad 3/5 1-2 días Aptitud para principiantes 30/100

salesforce/CodeGen2#10 · Abierto el 27/6/2023
Will the training implementation of CodeGen 2 be released? Abierto

Dificultad 5/5 Más de una semana Aptitud para principiantes 15/100

salesforce/CodeGen2#9 · Abierto el 15/6/2023
Is CodeGen2-16B the final and complete version? Abierto

Dificultad 5/5 Más de una semana Aptitud para principiantes 15/100

salesforce/CodeGen2#7 · Abierto el 12/5/2023
How is it compared with the StarCoder Abierto

Dificultad 5/5 Más de una semana Aptitud para principiantes 20/100

salesforce/CodeGen2#6 · 1 reacción · Abierto el 9/5/2023

Todos los issues de salesforce/CodeGen2

Issues similares

validate-structure inspecte challenge/work/.terraform et rend faux les labs qui ont un provider en cache Abierto

bug

Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100

stephrobert/dsoxlab#238 · Abierto el 24/9/2026
Uniform probability densities use the upper bound as an interval width Abierto

Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100

nasa/fmdtools#40 · Abierto el 24/9/2026
_common_folder() treats a single file as the common folder; single-file archives can't be installed (4.2.8) / crash (master) Abierto

Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100

sublimehq/package_control#1780 · Abierto el 24/9/2026
Set 'Sample' input in 'autorun.py' to utilize 'Sample' Tab in eLog Abierto

Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100

pcdshub/mfx#309 · 2 comentarios · Abierto el 24/9/2026
niri config file addition does not respect existing optional include Abierto

Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100

nwg-piotr/nwg-displays#145 · Abierto el 24/9/2026

Más issues de Python