Repository metrics
- Stars
- (4,990 stars)
- PR merge metrics
- (PR metrics pending)
Description
The model to consider.
现在直接跑这个模型会报错 vllm 0.24.0+cu129 vllm-omni 0.24.0rc1 vllm-omni
vllm serve ./zcc/Qwen-Image-2512-4bit --omni --host 0.0.0.0 --port 8091 --max-model-len 8192 --max-num-seqs 4 --tensor-parallel-size 1 --gpu-memory-utilization 0.85
(APIServer pid=1356) RuntimeError: Orchestrator initialization failed: Cannot instantiate BitsAndBytesConfig with kwargs {'_load_in_4bit': True, '_load_in_8bit': False, 'bnb_4bit_compute_dtype': 'bfloat16', 'bnb_4bit_quant_storage': 'uint8', 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'llm_int8_skip_modules': ['transformer_blocks.0.img_mod.1', 'transformer_blocks.0.attn.to_q', 'transformer_blocks.0.attn.to_k', 'transformer_blocks.0.attn.to_v', 'transformer_blocks.0.attn.add_k_proj', 'transformer_blocks.0.attn.add_v_proj', 'transformer_blocks.0.attn.add_q_proj', 'transformer_blocks.0.attn.to_out.0', 'transformer_blocks.0.attn.to_add_out', 'transformer_blocks.0.img_mlp.net.0.proj', 'transformer_blocks.0.img_mlp.net.2', 'transformer_blocks.0.txt_mod.1', 'transformer_blocks.0.txt_mlp.net.0.proj', 'transformer_blocks.0.txt_mlp.net.2', 'transformer_blocks.59.img_mod.1', 'transformer_blocks.59.attn.to_q', 'transformer_blocks.59.attn.to_k', 'transformer_blocks.59.attn.to_v', 'transformer_blocks.59.attn.add_k_proj', 'transformer_blocks.59.attn.add_v_proj', 'transformer_blocks.59.attn.add_q_proj', 'transformer_blocks.59.attn.to_out.0', 'transformer_blocks.59.attn.to_add_out', 'transformer_blocks.59.img_mlp.net.0.proj', 'transformer_blocks.59.img_mlp.net.2', 'transformer_blocks.59.txt_mod.1', 'transformer_blocks.59.txt_mlp.net.0.proj', 'transformer_blocks.59.txt_mlp.net.2', 'norm_out.linear', 'proj_out'], 'llm_int8_threshold': 6.0, 'load_in_4bit': True, 'load_in_8bit': False}. Expected signature: (self, load_in_8bit: bool = False, load_in_4bit: bool = True, bnb_4bit_compute_dtype: str = 'float32', bnb_4bit_quant_storage: str = 'uint8', bnb_4bit_quant_type: str = 'fp4', bnb_4bit_use_double_quant: bool = False, llm_int8_enable_fp32_cpu_offload: bool = False, llm_int8_has_fp16_weight: bool = False, llm_int8_skip_modules: list[str] | None = None, llm_int8_threshold: float = 6.0) -> None
The closest model vllm-omni already supports.
No response
What's your difficulty of supporting the model you want?
No response
Use case and motivation
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.