[Bug]: there is a ERROR "RuntimeError: The expanded size of the tensor (74) must match the existing size (0) at non-singleton dimension 1. Target sizes: [3, 74]. Tensor sizes: [3, 0]", when call a realtime audio req to qwen3-omni server

Your current environment

使用官方镜像：quay.io/ascend/vllm-omni:v0.24.0
torch==2.10.0+cpu
torch_npu==2.10.0
transformers==5.5.4
vLLM Version: 0.24.0
vllm-ascend Version: 0.19.1rc2.dev849+g05e095a20
vLLM-Omni Version: 0.24.0+cpu (git sha: d4a869fe)
cann-9.0.0
platforms: Ascend NPU 910B2

Your code version

ee0da84a

d4a869fe

🐛 Describe the bug

server:服务端启动命令 '''shell export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True" export HCCL_OP_EXPANSION_MODE="AIV" export HCCL_BUFFSIZE=1024 export OMP_NUM_THREADS=1

export LCCL_DETERMINISTIC=1 export HCCL_DETERMINISTIC=true export ATB_MATMUL_SHUFFLE_K_ENABLE=0 export ATB_LLM_LCOC_ENABLE=0 export CACHE_SYSTEM_PROMPT_NUM=0

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD # ubuntu export TASK_QUEUE_ENABLE=1

vllm serve /model/Qwen3-Omni-30B-A3B-Instruct
--omni
--port xxxx
--log-stats '''

部署配置vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml: ''' async_chunk: true platforms: npu: stages: - stage_id: 0 gpu_memory_utilization: 0.6 tensor_parallel_size: 2 max_num_batched_tokens: 8192 compilation_config: cudagraph_mode: FULL_DECODE_ONLY devices: "0,1" - stage_id: 1 gpu_memory_utilization: 0.6 max_num_batched_tokens: 8192 compilation_config: cudagraph_mode: PIECEWISE devices: "2" - stage_id: 2 gpu_memory_utilization: 0.3 compilation_config: cudagraph_mode: PIECEWISE devices: "2" '''

client请求方式: ''' python /vllm-omni/examples/online_serving/qwen3_omni/openai_realtime_client.py
--model /model/Qwen3-Omni-30B-A3B-Instruct
--url ws://localhost:xxxx/v1/realtime
--input-wav ./yyy.wav
--delta-dump-dir ./yyy_wavs '''

bug1: 偶现shape报错问题： ''' (StageEngineCoreProc_stage1_replica0 pid=75796) Process StageEngineCoreProc_stage1_replica0: (StageEngineCoreProc_stage1_replica0 pid=75796) Traceback (most recent call last): (StageEngineCoreProc_stage1_replica0 pid=75796) File "/usr/local/python3.12.13/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap (StageEngineCoreProc_stage1_replica0 pid=75796) self.run() (StageEngineCoreProc_stage1_replica0 pid=75796) File "/usr/local/python3.12.13/lib/python3.12/multiprocessing/process.py", line 108, in run (StageEngineCoreProc_stage1_replica0 pid=75796) self._target(*self._args, **self._kwargs) (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm-omni/vllm_omni/engine/stage_engine_core_proc.py", line 169, in run_stage_core (StageEngineCoreProc_stage1_replica0 pid=75796) engine_core.run_busy_loop() (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1265, in run_busy_loop (StageEngineCoreProc_stage1_replica0 pid=75796) self._process_engine_step() (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1304, in _process_engine_step (StageEngineCoreProc_stage1_replica0 pid=75796) outputs, model_executed = self.step_fn() (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 549, in step_with_batch_queue (StageEngineCoreProc_stage1_replica0 pid=75796) exec_future = self.model_executor.execute_model( (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 120, in execute_model (StageEngineCoreProc_stage1_replica0 pid=75796) output.result() (StageEngineCoreProc_stage1_replica0 pid=75796) File "/usr/local/python3.12.13/lib/python3.12/concurrent/futures/_base.py", line 449, in result (StageEngineCoreProc_stage1_replica0 pid=75796) return self.__get_result() (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/usr/local/python3.12.13/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result (StageEngineCoreProc_stage1_replica0 pid=75796) raise self._exception (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 98, in collective_rpc (StageEngineCoreProc_stage1_replica0 pid=75796) result = run_method(self.driver_worker, method, args, kwargs) (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 510, in run_method (StageEngineCoreProc_stage1_replica0 pid=75796) return func(*args, **kwargs) (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 351, in execute_model (StageEngineCoreProc_stage1_replica0 pid=75796) return self.worker.execute_model(scheduler_output) (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker.py", line 708, in execute_model (StageEngineCoreProc_stage1_replica0 pid=75796) output = self.model_runner.execute_model(scheduler_output, intermediate_tensors) (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/usr/local/python3.12.13/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context (StageEngineCoreProc_stage1_replica0 pid=75796) return func(*args, **kwargs) (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm-omni/vllm_omni/platforms/npu/worker/npu_ar_model_runner.py", line 482, in execute_model (StageEngineCoreProc_stage1_replica0 pid=75796) ) = self._prepare_inputs( (StageEngineCoreProc_stage1_replica0 pid=75796) ^^^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1019, in _prepare_inputs (StageEngineCoreProc_stage1_replica0 pid=75796) self._calc_mrope_positions(scheduler_output) (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm-omni/vllm_omni/worker/gpu_model_runner.py", line 398, in _calc_mrope_positions (StageEngineCoreProc_stage1_replica0 pid=75796) super()._calc_mrope_positions(scheduler_output) (StageEngineCoreProc_stage1_replica0 pid=75796) File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 2683, in _calc_mrope_positions (StageEngineCoreProc_stage1_replica0 pid=75796) self.mrope_positions.cpu[:, dst_start:dst_end] = req.mrope_positions[ (StageEngineCoreProc_stage1_replica0 pid=75796) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ (StageEngineCoreProc_stage1_replica0 pid=75796) RuntimeError: The expanded size of the tensor (74) must match the existing size (0) at non-singleton dimension 1. Target sizes: [3, 74]. Tensor sizes: [3, 0] ''' FYI：自行修改vllm/vllm/v1/worker/gpu_model_runner.py:1569行，将req_state.prompt_token_ids = new_req_data.prompt_token_ids改为req_state.prompt_token_ids = list(new_req_data.prompt_token_ids)，该报错不再出现，但会偶现请求在返回音频时卡住和返回音频与返回文本不一致的情况。 卡住时的server日志： ‘’‘ (APIServer pid=88106) INFO: 127.0.0.1:39352 - "WebSocket /v1/realtime" [accepted] (APIServer pid=88106) INFO: connection open (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-0 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-1 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-2 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-0 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-1 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-2 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-0 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-1 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-2 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-0 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-1 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) INFO 07-20 18:41:57 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-2 [rep-0] add request: rt-ws-c09becb2-78a8-4e59-b03d-a1a5aa380d23-85207a93-9637-46eb-b694-717ad157bb1c-a56d618f (APIServer pid=88106) WARNING 07-20 18:46:29 [output_processor.py:227] Error concatenating tensor for key sr; keeping last tensor (APIServer pid=88106) WARNING 07-20 18:46:40 [output_processor.py:227] Error concatenating tensor for key sr; keeping last tensor (StageEngineCoreProc_stage1_replica0 pid=88924) ERROR 07-20 18:46:40 [chunk_transfer_adapter.py:318] Failed to use custom_process_input_func for payload extraction: torch.cat(): expected a non-empty list of Tensors

‘’’

bug2: 上述配置仅修改async_chunk: false，在返回文本后，服务端卡住 logs: ''' (APIServer pid=79352) INFO: 127.0.0.1:46334 - "WebSocket /v1/realtime" [accepted] (APIServer pid=79352) INFO: connection open (APIServer pid=79352) Unrecognized keys in rope_parameters for 'rope_type'='default': {'interleaved', 'mrope_section'} (APIServer pid=79352) WARNING 07-20 15:43:56 [input_processor.py:282] Passing raw prompts to InputProcessor is deprecated and will be removed in v0.18. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). (APIServer pid=79352) INFO 07-20 15:43:56 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-0 [rep-0] add request: rt-ws-be163fb0-4bd0-49a7-9d9d-bb178e1723e5-ab52dcf2-02fe-438c-87b1-abd1b05b5185-bc08f26f (APIServer pid=79352) INFO 07-20 15:43:56 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-0 [rep-0] add request: rt-ws-be163fb0-4bd0-49a7-9d9d-bb178e1723e5-ab52dcf2-02fe-438c-87b1-abd1b05b5185-bc08f26f (Worker_TP0 pid=79430) Warning: since the loaded file is not a zipfile, only "torch.device" and "str" type parameters are currently supported for parameter types of map_location. If parameter types of map_location is "Callable[[torch.Tensor, str], torch.Tensor]" or "Dict[str, str]", which is only support for zipfile, all tensors are currently loaded onto the CPU, which may introduce problems. (Worker_TP0 pid=79430) WARNING 07-20 15:43:56 [gpu_model_runner.py:604] additional_information on request data is deprecated, use model_intermediate_buffer (Worker_TP1 pid=79431) Warning: since the loaded file is not a zipfile, only "torch.device" and "str" type parameters are currently supported for parameter types of map_location. If parameter types of map_location is "Callable[[torch.Tensor, str], torch.Tensor]" or "Dict[str, str]", which is only support for zipfile, all tensors are currently loaded onto the CPU, which may introduce problems. INFO 07-20 15:44:06 [init.py:44] Available plugins for group vllm.platform_plugins: INFO 07-20 15:44:06 [init.py:46] - ascend -> vllm_ascend:register INFO 07-20 15:44:06 [init.py:49] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load. INFO 07-20 15:44:06 [init.py:237] Platform plugin ascend is activated INFO 07-20 15:44:06 [platform.py:60] Breakable cudagraph is force disabled on Ascend because DeepSeek V4 PIECEWISE cudagraph is not supported yet. INFO 07-20 15:44:06 [init.py:44] Available plugins for group vllm.platform_plugins: INFO 07-20 15:44:06 [init.py:46] - ascend -> vllm_ascend:register INFO 07-20 15:44:06 [init.py:49] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load. INFO 07-20 15:44:06 [init.py:237] Platform plugin ascend is activated INFO 07-20 15:44:07 [platform.py:60] Breakable cudagraph is force disabled on Ascend because DeepSeek V4 PIECEWISE cudagraph is not supported yet. (Worker_TP0 pid=79430) ('Warning: torch.save with "_use_new_zipfile_serialization = False" is not recommended for npu tensor, which may bring unexpected errors and hopefully set "_use_new_zipfile_serialization = True"', 'if it is necessary to use this, please convert the npu tensor to cpu tensor for saving') (Worker_TP0 pid=79430) INFO 07-20 15:44:10 [acl_graph.py:259] Replaying aclgraph (APIServer pid=79352) INFO 07-20 15:44:10 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-1 [rep-0] add request: rt-ws-be163fb0-4bd0-49a7-9d9d-bb178e1723e5-ab52dcf2-02fe-438c-87b1-abd1b05b5185-bc08f26f (APIServer pid=79352) INFO 07-20 15:44:10 [stage_engine_core_client.py:233] [StageEngineCoreClient] stage-1 [rep-0] add request: rt-ws-be163fb0-4bd0-49a7-9d9d-bb178e1723e5-ab52dcf2-02fe-438c-87b1-abd1b05b5185-bc08f26f '''

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.