使用100%A100运行报错
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 30/100
Research direction
Reproduce the vLLM deployment with 100% A100 and compare it with the working 50% A100 setup. Start with the model_runner_base.py and launcher.py log entries, inspect the dumped /tmp/err_execute_model_input_20250303-212517.pkl file, and check the ProcessGroupNCCL warning. Done means identifying the cause of the server termination and documenting or fixing the failing configuration.
Written by the indexing model from the issue text.
Description
背景 使用100%A100 用vllm部署
INFO 03-03 21:25:17 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250303-212517.pkl... INFO 03-03 21:25:17 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20250303-212517.pkl. CRITICAL 03-03 21:25:17 launcher.py:99] MQLLMEngine is already dead, terminating server process INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [39513] [rank0]:[W303 21:25:18.275990903 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
报错。
同样的步骤使用50% A100 正常运行
- Dominant language
- Python
- Stars
- 2k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from InternLM/Tutorial
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
Difficulty 1/5 Under an hour Newbie friendliness 65/100
-
test Open
Difficulty 5/5 Over a week Newbie friendliness 1/100
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
XTuner 微调报错 Open
Difficulty 4/5 3-5 days Newbie friendliness 20/100
All issues in InternLM/Tutorial
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100