verl-project/verl

vllm cannot load model after megatron training

Open

#1757 opened on May 29, 2025

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (21,533 stars) (3,940 forks)auto 404
help wanted

Description

After megatron training and convert to hf model, i want to infer using vllm, which meet problem when loading. File "/python3.11/site-packages/vllm/model_executor/models/utils.py", line 250, in _load_module raise ValueError(msg) ValueError: There is no module or parameter named 'decoder' in Qwen3ForCausalLM After model_merger.py process, the name of parameters changed.

Contributor guide

Tech stack
pythonpytorch
Domain
machine learning
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PythonPyTorchMegatronvLLM
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Investigate the parameter naming mismatch between Megatron's converted model and vLLM. The error occurs in vllm/model executor/models/utils.py at line 250, where parameter 'decoder' is missing. Check model merger.py to understand how it renames parameters after training. Examine the Qwen3ForCausalLM model definition to identify expected parameter names. The fix likely involves updating the conversion script to match vLLM's expectations or adding a parameter mapping in vLLM.