Can't this model be trained using multi nodes and multiple cards ?

Đang mở

#3 mở ngày 4/5/2023 2 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó: 5/5
Thời gian dự kiến: Hơn một tuần
Mức phù hợp với người mới: 25/100

Loại issue: Tính năng
Độ rõ ràng: Cần làm rõ
Mức độ hoạt động: Đình trệ
Công nghệ: python
Lĩnh vực: distributed-systems, machine-learning

Hướng nghiên cứu

Start with train.py and the multinode_runner.py and DeepSpeed launcher commands shown in the logs. Reproduce the 16B-model run across the listed nodes and inspect where memory use remains per GPU. Done would require a documented or implemented way to train the model across multiple machines and cards without OOM.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

My machine has a single node 4-card 16G graphics memory, and running the 16B model with multiple nodes will result in OOM regardless of how the number of nodes is set. Can this model be trained with multiple machines and cards
Here are some logs of the training：

2023-05-04 16:21:02,954] [INFO] [multinode_runner.py:65:get_cmd] Running on the following workers: c05r2n11,c05r2n16,c05r2n19,c05r3n02,c06r4n06,c06r4n09,c06r4n12,c06r4n17
[2023-05-04 16:21:02,955] [INFO] [runner.py:453:main] cmd = pdsh -f 1024 -w c05r2n11,c05r2n16,c05r2n19,c05r3n02,c06r4n06,c06r4n09,c06r4n12,c06r4n17 export NCCL_SOCKET_IFNAME=ib0; export UCX_MAX_EAGER_LANES=4; export UCX_MAX_RNDV_LANES=4; export UCX_ZCOPY_THRESH=auto; export UCX_RNDV_THRESH=auto; export UCX_DC_MLX5_NUM_DCI=16; export NCCL_IB_HCA=mlx5_0; export NCCL_DEBUG=info; export PYTHONPATH=/work/home/actvg1ue59/CodeGen-main; cd /work/home/actvg1ue59/CodeGen-main; /work/home/actvg1ue59/miniconda3/envs/torch/bin/python -u -m deepspeed.launcher.launch --world_info=eyJjMDVyMm4xMSI6IFswLCAxLCAyLCAzXSwgImMwNXIybjE2IjogWzAsIDEsIDIsIDNdLCAiYzA1cjJuMTkiOiBbMCwgMSwgMiwgM10sICJjMDVyM24wMiI6IFswLCAxLCAyLCAzXSwgImMwNnI0bjA2IjogWzAsIDEsIDIsIDNdLCAiYzA2cjRuMDkiOiBbMCwgMSwgMiwgM10sICJjMDZyNG4xMiI6IFswLCAxLCAyLCAzXSwgImMwNnI0bjE3IjogWzAsIDEsIDIsIDNdfQ== --node_rank=%n --master_addr=c05r2n11 --master_port=9901 train.py
c05r3n02: Warning: Permanently added 'c05r3n02,10.3.5.43' (ECDSA) to the list of known hosts.^M
c05r2n16: Warning: Permanently added 'c05r2n16,10.3.5.37' (ECDSA) to the list of known hosts.^M
c05r2n19: Warning: Permanently added 'c05r2n19,10.3.5.40' (ECDSA) to the list of known hosts.^M
c06r4n06: Warning: Permanently added 'c06r4n06,10.3.6.67' (ECDSA) to the list of known hosts.^M
c06r4n09: Warning: Permanently added 'c06r4n09,10.3.6.70' (ECDSA) to the list of known hosts.^M
c06r4n12: Warning: Permanently added 'c06r4n12,10.3.6.73' (ECDSA) to the list of known hosts.^M
c06r4n17: Warning: Permanently added 'c06r4n17,10.3.6.78' (ECDSA) to the list of known hosts.^M
c05r2n11: Currently Loaded Modulefiles:
c05r2n11: 1) compiler/devtoolset/7.3.1 3) compiler/dtk/22.10.1
c05r2n11: 2) mpi/hpcx/gcc-7.3.1
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:96:main] 0 NCCL_SOCKET_IFNAME=ib0
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:96:main] 0 NCCL_IB_HCA=mlx5_0
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:96:main] 0 NCCL_DEBUG=info
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n11: [2023-05-04 16:21:07,883] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=0
c05r2n11: [2023-05-04 16:21:07,884] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n11: [2023-05-04 16:21:07,884] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n11: [2023-05-04 16:21:07,884] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=1
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:96:main] 7 NCCL_IB_HCA=mlx5_0
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 2 NCCL_IB_HCA=mlx5_0
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:96:main] 6 NCCL_DEBUG=info
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:96:main] 7 NCCL_DEBUG=info
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 2 NCCL_DEBUG=info
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=4
c06r4n09: [2023-05-04 16:21:36,546] [INFO] [launch.py:96:main] 5 NCCL_IB_HCA=mlx5_0
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 3 NCCL_IB_HCA=mlx5_0
c06r4n09: [2023-05-04 16:21:36,546] [INFO] [launch.py:96:main] 5 NCCL_DEBUG=info
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=6
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 3 NCCL_DEBUG=info
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=2
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=3
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=7
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:123:main] dist_world_size=32
c05r3n02: [2023-05-04 16:21:36,532] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,532] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n09: [2023-05-04 16:21:36,555] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c06r4n09: [2023-05-04 16:21:36,555] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=5
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=1
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:96:main] 7 NCCL_IB_HCA=mlx5_0
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 2 NCCL_IB_HCA=mlx5_0
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:96:main] 6 NCCL_DEBUG=info
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:96:main] 7 NCCL_DEBUG=info
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 2 NCCL_DEBUG=info
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=4
c06r4n09: [2023-05-04 16:21:36,546] [INFO] [launch.py:96:main] 5 NCCL_IB_HCA=mlx5_0
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 3 NCCL_IB_HCA=mlx5_0
c06r4n09: [2023-05-04 16:21:36,546] [INFO] [launch.py:96:main] 5 NCCL_DEBUG=info
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=6
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n16: [2023-05-04 16:21:36,542] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n06: [2023-05-04 16:21:36,543] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n12: [2023-05-04 16:21:36,538] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:96:main] 3 NCCL_DEBUG=info
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=2
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:123:main] dist_world_size=32
c05r2n19: [2023-05-04 16:21:36,531] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=3
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=7
c05r3n02: [2023-05-04 16:21:36,531] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [4, 5, 6, 7], 'c05r2n19': [8, 9, 10, 11], 'c05r3n02': [12, 13, 14, 15], 'c06r4n06': [16, 17, 18, 19], 'c06r4n09': [20, 21, 22, 23], 'c06r4n12': [24, 25, 26, 27], 'c06r4n17': [28, 29, 30, 31]})
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:123:main] dist_world_size=32
c05r3n02: [2023-05-04 16:21:36,532] [INFO] [launch.py:123:main] dist_world_size=32
c06r4n17: [2023-05-04 16:21:36,534] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c05r3n02: [2023-05-04 16:21:36,532] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
c06r4n09: [2023-05-04 16:21:36,555] [INFO] [launch.py:103:main] WORLD INFO DICT: {'c05r2n11': [0, 1, 2, 3], 'c05r2n16': [0, 1, 2, 3], 'c05r2n19': [0, 1, 2, 3], 'c05r3n02': [0, 1, 2, 3], 'c06r4n06': [0, 1, 2, 3], 'c06r4n09': [0, 1, 2, 3], 'c06r4n12': [0, 1, 2, 3], 'c06r4n17': [0, 1, 2, 3]}
c06r4n09: [2023-05-04 16:21:36,555] [INFO] [launch.py:109:main] nnodes=8, num_local_procs=4, node_rank=5
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 946, in load_checkpoint_in_model
c05r2n11: load_checkpoint_in_model(
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 946, in load_checkpoint_in_model
c05r2n11: load_checkpoint_in_model(
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 946, in load_checkpoint_in_model
c05r2n11: set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
c05r2n11: set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
c05r2n11: set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
c05r2n11:
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
c05r2n11: File "/work/home/actvg1ue59/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
c05r2n11: new_value = value.to(device)new_value = value.to(device)
c05r2n11:
c05r2n11: new_value = value.to(device)
c05r2n11: new_value = value.to(device)
c05r2n11: RuntimeErrorRuntimeError: : HIP out of memory. Tried to allocate 576.00 MiB (GPU 3; 15.98 GiB total capacity; 15.83 GiB already allocated; 0 bytes free; 15.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONFHIP out of memory. Tried to allocate 576.00 MiB (GPU 1; 15.98 GiB total capacity; 15.83 GiB already allocated; 0 bytes free; 15.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONF
c05r2n11: RuntimeError
c05r2n11: RuntimeError: : HIP out of memory. Tried to allocate 576.00 MiB (GPU 2; 15.98 GiB total capacity; 15.83 GiB already allocated; 0 bytes free; 15.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONFHIP out of memory. Tried to allocate 576.00 MiB (GPU 0; 15.98 GiB total capacity; 15.83 GiB already allocated; 0 bytes free; 15.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONF

Ngôn ngữ chính: Python
Star: 268
Fork: 11
Chỉ số merge pull request: Không có pull request nào được merge trong 30 ngày

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Bắt đầu từ đâu

Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
Fork repository và làm thay đổi trên một nhánh.
Mở pull request có tham chiếu số hiệu của issue.

Issue khác của salesforce/CodeGen2

Question of unk, bos, and eos tokens. Đang mở

Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 25/100

salesforce/CodeGen2#11 · Mở ngày 20/7/2023
Question of eos token in codegen2-7B model Đang mở

Độ khó 3/5 1-2 ngày Mức phù hợp với người mới 30/100

salesforce/CodeGen2#10 · Mở ngày 27/6/2023
Will the training implementation of CodeGen 2 be released? Đang mở

Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 15/100

salesforce/CodeGen2#9 · Mở ngày 15/6/2023
Is CodeGen2-16B the final and complete version? Đang mở

Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 15/100

salesforce/CodeGen2#7 · Mở ngày 12/5/2023
How is it compared with the StarCoder Đang mở

Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 20/100

salesforce/CodeGen2#6 · 1 reaction · Mở ngày 9/5/2023

Tất cả issue của salesforce/CodeGen2

Issue tương tự

docs: update workflows.md for slide deck Node test discovery Đang mở

agent-ready documentation needs-triage

Độ khó 1/5 1-3 giờ Mức phù hợp với người mới 88/100

microsoft/hve-core#3012 · Mở ngày 25/9/2026
config_keys.rst says every result file PyBNF writes is alphabetical, but the credible-interval files and start_point.txt use declaration order Đang mở

documentation

Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 91/100

lanl/PyBNF#932 · Mở ngày 25/9/2026
workflow-status page template still says reusable workflows are "triggered only by workflow_call:" Đang mở

Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 92/100

LearningCircuit/local-deep-research#6804 · Mở ngày 24/9/2026
Add https://search.jeremyh.xyz/ Đang mở

instance instance add

Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 72/100

searxng/searx-instances#939 · 1 bình luận · Mở ngày 24/9/2026
Azure Sandboxes: Small and ExtraSmall tiers always fail to deploy (disk request exceeds tier maximum) Đang mở

area-deployment area-integrations triage:bot-seen

Độ khó 2/5 Nửa ngày Mức phù hợp với người mới 86/100

microsoft/aspire#20459 · 1 bình luận · Mở ngày 24/9/2026

Thêm issue về Python