whisperx doesn't return speaker when diarization is requested

Tái hiện vấn đề bằng lệnh local-ai transcript --diarize --model=whisperx-tiny được ghi trong tài liệu và kiểm tra đường dẫn backend cpu-whisperx được hiển thị trong log. So sánh đầu ra JSON chi tiết của nó với cấu trúc segment dự kiến và các backend khác hỗ trợ diarization. Hoàn tất khi các segment WhisperX có diarization bao gồm trường speaker.

LocalAI version:
4.8.2

Environment, CPU architecture, OS, and Version:

root@local-ai:~/sample-files# uname -a
Linux local-ai 6.8.0-137-generic #137-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 17 20:28:23 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

root@local-ai:~/sample-files# free -mh
               total        used        free      shared  buff/cache   available
Mem:            31Gi       948Mi        15Gi        35Mi        15Gi        30Gi
Swap:             0B          0B          0B

root@local-ai:~/sample-files# lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      10
  On-line CPU(s) list:       0-9
Vendor ID:                   GenuineIntel
  BIOS Vendor ID:            QEMU
  Model name:                Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
    BIOS Model name:         pc-q35-8.2  CPU @ 2.0GHz
    BIOS CPU family:         1
    CPU family:              6
    Model:                   79
    Thread(s) per core:      1
    Core(s) per socket:      10
    Socket(s):               1
    Stepping:                1
    BogoMIPS:                4794.44
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon re
                             p_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdr
                             and hypervisor lahf_lm abm 3dnowprefetch cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid
                              rtm rdseed adx smap xsaveopt arat vnmi umip md_clear flush_l1d arch_capabilities
Virtualization features:     
  Virtualization:            VT-x
Caches (sum of all):         
  L1d:                       320 KiB (10 instances)
  L1i:                       320 KiB (10 instances)
  L2:                        40 MiB (10 instances)
  L3:                        16 MiB (1 instance)
NUMA:                        
  NUMA node(s):              1
  NUMA node0 CPU(s):         0-9
Vulnerabilities:             
  Gather data sampling:      Not affected
  Indirect target selection: Mitigation; Aligned branch/return thunks
  Itlb multihit:             Not affected
  L1tf:                      Mitigation; PTE Inversion; VMX flush not necessary, SMT disabled
  Mds:                       Mitigation; Clear CPU buffers; SMT Host state unknown
  Meltdown:                  Mitigation; PTI
  Mmio stale data:           Mitigation; Clear CPU buffers; SMT Host state unknown
  Reg file data sampling:    Not affected
  Retbleed:                  Not affected
  Spec rstack overflow:      Not affected
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Mitigation; Clear CPU buffers; SMT Host state unknown
  Vmscape:                   Not affected

Describe the bug
Whisperx does not return the speaker id even when requested to diarize the output. I believe this is a bug as the model is declared to be diarization-capable.

In the same host and using other diarization-capable models (vibevoice-cpp-asr, vibevoice-crispasr and others) the speaker is returned correctly.

To Reproduce
Run:

/opt/local-ai/current/local-ai transcript --diarize --model=whisperx-tiny --backends-path /path-to-backends/ --models-path /path-to-models/ --log-level=debug --response-format=verbose_json --pretty-print --threads=10 ~/sample-files/micro-machines.wav

Expected behavior
The segments field in the output JSON should contain the speaker but it doesn't

Example entry:

 {
            "id": 0,
            "start": 0,
            "end": 3e-9,
            "text": " This is the micro machine representing the most miniature motorcade of micro machine.",
            "tokens": null
        },

Logs

Full output of the above command:

Aug 14 21:48:18 DEBUG GPU vendor gpuVendor="" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/system/state.go"  caller.L=87 } 
Aug 14 21:48:18 DEBUG Total available VRAM vram=0 caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/system/state.go"  caller.L=89 } 
Aug 14 21:48:18 INFO  Default capability (no GPU detected) env="LOCALAI_FORCE_META_BACKEND_CAPABILITY" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/system/capabilities.go"  caller.L=397 } 
Aug 14 21:48:18 DEBUG No system backends found caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=650 } 
Aug 14 21:48:18 DEBUG Registering backend name="llama-cpp" runFile="/mnt/vol01/local-ai/backends/cpu-llama-cpp/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="vibevoice-cpp" runFile="/mnt/vol01/local-ai/backends/cpu-vibevoice-cpp/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="whisper" runFile="/mnt/vol01/local-ai/backends/cpu-whisper/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="whisperx" runFile="/mnt/vol01/local-ai/backends/cpu-whisperx/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="cpu-crispasr" runFile="/mnt/vol01/local-ai/backends/cpu-crispasr/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="cpu-llama-cpp" runFile="/mnt/vol01/local-ai/backends/cpu-llama-cpp/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="cpu-whisper" runFile="/mnt/vol01/local-ai/backends/cpu-whisper/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="crispasr" runFile="/mnt/vol01/local-ai/backends/cpu-crispasr/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="moss-transcribe-cpp" runFile="/mnt/vol01/local-ai/backends/cpu-moss-transcribe-cpp/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="cpu-moss-transcribe-cpp" runFile="/mnt/vol01/local-ai/backends/cpu-moss-transcribe-cpp/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="cpu-vibevoice-cpp" runFile="/mnt/vol01/local-ai/backends/cpu-vibevoice-cpp/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG Registering backend name="cpu-whisperx" runFile="/mnt/vol01/local-ai/backends/cpu-whisperx/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/gallery/backends.go"  caller.L=760 } 
Aug 14 21:48:18 DEBUG [inference_defaults] applying defaults for model modelID="qwen3-vl-4b-instruct" family=map[min_p:0 repeat_penalty:1 temperature:0.7 top_k:20 top_p:0.8] caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/config/inference_defaults.go"  caller.L=90 } 
Aug 14 21:48:18 DEBUG [serving_defaults] enabling cross-request prefix cache cache_reuse=256 caller={caller.file="/home/runner/work/LocalAI/LocalAI/core/config/serving_defaults.go"  caller.L=39 } 
Aug 14 21:48:18 INFO  BackendLoader starting modelID="whisperx-tiny" backend="whisperx" model="tiny" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=235 } 
Aug 14 21:48:18 INFO  effective runtime tuning (override in the model YAML; LOCALAI_DISABLE_HARDWARE_DEFAULTS=true disables hardware auto-tuning) modelID="whisperx-tiny" context=4096 n_batch=512 n_gpu_layers=99999999 parallel="1" flash_attention="auto" f16=false caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=248 } 
Aug 14 21:48:18 DEBUG Loading model in memory from file file="/mnt/vol01/local-ai/models/tiny" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/loader.go"  caller.L=545 } 
Aug 14 21:48:18 DEBUG Loading Model with gRPC modelID="whisperx-tiny" file="/mnt/vol01/local-ai/models/tiny" backend="whisperx" options={whisperx tiny  whisperx-tiny {{}} 0x2a3b29896308 map[] 20 2 true 0} caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=59 } 
Aug 14 21:48:18 DEBUG Loading external backend uri="/mnt/vol01/local-ai/backends/cpu-whisperx/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=116 } 
Aug 14 21:48:18 DEBUG external backend is file file=&{run.sh 192 493 {0 63921523885 0x9a94de0} {2065 1709650 1 33261 0 0 0 0 192 4096 8 {1786725072 656365452} {1785927085 0} {1786724059 986318872} [0 0 0]}} caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=119 } 
Aug 14 21:48:18 DEBUG Loading GRPC Process process="/mnt/vol01/local-ai/backends/cpu-whisperx/run.sh" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=246 } 
Aug 14 21:48:18 DEBUG GRPC Service will be running id="whisperx-tiny" address="127.0.0.1:37609" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=248 } 
Aug 14 21:48:18 DEBUG GRPC Service state dir dir="/tmp/go-processmanager895863908" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=281 } 
Aug 14 21:48:18 DEBUG GRPC Service Started caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=131 } 
Aug 14 21:48:18 DEBUG Wait for the service to start up caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=144 } 
Aug 14 21:48:18 DEBUG Options options=ContextSize:4096  Seed:1420980654  NBatch:512  MMap:true  NGPULayers:99999999  Threads:10  FlashAttention:"auto" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=145 } 
Aug 14 21:48:18 DEBUG GRPC stdout id="whisperx-tiny-127.0.0.1:37609" line="Initializing libbackend for cpu-whisperx" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=318 } 
Aug 14 21:48:18 DEBUG GRPC stdout id="whisperx-tiny-127.0.0.1:37609" line="Using portable Python" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=318 } 
Aug 14 21:48:18 DEBUG GRPC stdout id="whisperx-tiny-127.0.0.1:37609" line="Added /mnt/vol01/local-ai/backends/cpu-whisperx/lib to LD_LIBRARY_PATH for GPU libraries" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=318 } 
Aug 14 21:48:19 DEBUG GRPC stderr id="whisperx-tiny-127.0.0.1:37609" line="Server started. Listening on: 127.0.0.1:37609" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=305 } 
Aug 14 21:48:20 DEBUG GRPC Service Ready caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=152 } 
Aug 14 21:48:20 DEBUG GRPC: Loading model with options options=Model:"tiny"  ContextSize:4096  Seed:1420980654  NBatch:512  MMap:true  NGPULayers:99999999  Threads:10  ModelFile:"/mnt/vol01/local-ai/models/tiny"  FlashAttention:"auto"  ModelPath:"/mnt/vol01/local-ai/models/" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/initializers.go"  caller.L=176 } 
Aug 14 21:48:22 DEBUG GRPC stderr id="whisperx-tiny-127.0.0.1:37609" line="Preparing WhisperX model, please wait" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=305 } 
Aug 14 21:48:28 DEBUG GRPC stdout id="whisperx-tiny-127.0.0.1:37609" line="2026-08-14 21:48:28 - whisperx.asr - INFO - No language specified, language will be detected for each audio file (increases inference time)" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=318 } 
Aug 14 21:48:28 DEBUG GRPC stdout id="whisperx-tiny-127.0.0.1:37609" line="2026-08-14 21:48:28 - whisperx.vads.pyannote - INFO - Performing voice activity detection using Pyannote..." caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=318 } 
Aug 14 21:48:29 DEBUG GRPC stderr id="whisperx-tiny-127.0.0.1:37609" line="Lightning automatically upgraded your loaded checkpoint from v1.5.4 to v2.6.5. To apply the upgrade to your files permanently, run `python -m lightning.pytorch.utilities.upgrade_checkpoint venv/lib/python3.10/site-packages/whisperx/assets/pytorch_model.bin`" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=305 } 
Aug 14 21:48:29 DEBUG GRPC stderr id="whisperx-tiny-127.0.0.1:37609" line="WhisperX model loaded: tiny on cpu" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=305 } 
Aug 14 21:48:29 DEBUG Model already loaded in memory model="whisperx-tiny" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/loader.go"  caller.L=641 } 
Aug 14 21:48:29 DEBUG Checking model availability model="whisperx-tiny" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/loader.go"  caller.L=653 } 
Aug 14 21:48:30 DEBUG GRPC stdout id="whisperx-tiny-127.0.0.1:37609" line="2026-08-14 21:48:30 - whisperx.asr - WARNING - Audio is shorter than 30s, language detection may be inaccurate" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=318 } 
Aug 14 21:48:30 DEBUG GRPC stdout id="whisperx-tiny-127.0.0.1:37609" line="2026-08-14 21:48:30 - whisperx.asr - INFO - Detected language: en (0.99) in first 30s of audio" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=318 } 
{
    "segments": [
        {
            "id": 0,
            "start": 0,
            "end": 3e-9,
            "text": " This is the micro machine representing the most miniature motorcade of micro machine.",
            "tokens": null
        },
        {
            "id": 1,
            "start": 3e-9,
            "end": 8e-9,
            "text": "Each one has dramatic details for a fixed-driven position, paint jobs, plus incredible micro machine pocketplace that says a police station fire station restaurant, service station, and more.",
            "tokens": null
        },
        {
            "id": 2,
            "start": 8e-9,
            "end": 1.1e-8,
            "text": "Perfect pocket portable to take any place, and there are many miniature places to play with.",
            "tokens": null
        },
        {
            "id": 3,
            "start": 1.1e-8,
            "end": 1.5e-8,
            "text": "Each one comes with its own special edition, micro machine vehicle, and fun fantastic features that miraculously move.",
            "tokens": null
        },
        {
            "id": 4,
            "start": 1.5e-8,
            "end": 1.9e-8,
            "text": "Raise the bolt lift at the airport Marina, man, the gun turret at the army base, clean your car at the car wash, raise the toll bridge.",
            "tokens": null
        },
        {
            "id": 5,
            "start": 1.9e-8,
            "end": 2.1e-8,
            "text": "And these places fit together to form a micro machine world.",
            "tokens": null
        },
        {
            "id": 6,
            "start": 2.1e-8,
            "end": 2.3e-8,
            "text": "Micro machine pocketplace that's so tremendously tiny, so perfectly precise.",
            "tokens": null
        },
        {
            "id": 7,
            "start": 2.3e-8,
            "end": 2.5e-8,
            "text": "So, doesn't we detail Joe on a pocket them all?",
            "tokens": null
        },
        {
            "id": 8,
            "start": 2.5e-8,
            "end": 2.7e-8,
            "text": "Micro machines at micro machine pocketplace that's sold separately from Gloob.",
            "tokens": null
        },
        {
            "id": 9,
            "start": 2.7e-8,
            "end": 2.9e-8,
            "text": "The smaller they are, the better they are.",
            "tokens": null
        }
    ],
    "text": " This is the micro machine representing the most miniature motorcade of micro machine.Each one has dramatic details for a fixed-driven position, paint jobs, plus incredible micro machine pocketplace that says a police station fire station restaurant, service station, and more.Perfect pocket portable to take any place, and there are many miniature places to play with.Each one comes with its own special edition, micro machine vehicle, and fun fantastic features that miraculously move.Raise the bolt lift at the airport Marina, man, the gun turret at the army base, clean your car at the car wash, raise the toll bridge.And these places fit together to form a micro machine world.Micro machine pocketplace that's so tremendously tiny, so perfectly precise.So, doesn't we detail Joe on a pocket them all?Micro machines at micro machine pocketplace that's sold separately from Gloob.The smaller they are, the better they are."
}
Aug 14 21:48:37 DEBUG Deleting process model="whisperx-tiny" force=false caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=115 } 
Aug 14 21:48:37 DEBUG Calling Free() to release GPU resources model="whisperx-tiny" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=132 } 
Aug 14 21:48:37 DEBUG Backend does not implement Free(); GPU release handled on process stop model="whisperx-tiny" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=138 } 
Aug 14 21:48:37 DEBUG GRPC stdout id="whisperx-tiny-127.0.0.1:37609" line="Received termination signal. Shutting down..." caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=318 } 
Aug 14 21:48:39 INFO  Backend process stopped id="whisperx-tiny" address="127.0.0.1:37609" process="run.sh" exitCode="0" caller={caller.file="/home/runner/work/LocalAI/LocalAI/pkg/model/process.go"  caller.L=349 } 

Additional context