Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

CTC keyword spotter hears silence: MelSpectrogram on the Neural Engine yields all-blank CTC frames

Aperta Adatta ai principianti
  • Qualcosa non va?
#1,074 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
78/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
swift
Ambito
audio-video-rtc, backend, machine-learning

Direzione di ricerca

Leggi CtcModels.load e loadDirect, che configurano le unità di calcolo dei modelli CTC. La riproduzione nell’issue mostra che MelSpectrogram funziona sulla CPU, mentre l’encoder funziona sulla Neural Engine; controlla come entrambi i loader impostano queste configurazioni. Il lavoro è terminato quando MelSpectrogram usa la CPU, l’encoder mantiene il supporto per la Neural Engine e una breve clip vocale produce un output CTC non vuoto.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

CtcModels.defaultConfiguration() loads both CTC models with .cpuAndNeuralEngine. On the Neural Engine, the CTC-110M MelSpectrogram produces features that make the encoder emit blank for every frame, whatever the audio. VocabularyBoostingSession keeps working without error, but its CTC evidence no longer depends on the audio, so rescoring decisions come down to string similarity and the boost.

Running only the MelSpectrogram on the CPU fixes it, and is as fast as or faster than the default. The encoder is fine on the Neural Engine.

This is separate from #991. The input here is explicitly zero-padded, and #991's checks used .all and .cpuOnly, which both work. Only the default, .cpuAndNeuralEngine, fails.

Environment

  • FluidAudio 0.18.0 (e152879)
  • parakeet-ctc-110m-coreml from the default download
  • macOS 27.0.1 (26A434), on Apple M1 Pro and Apple M4 Max

Reproduction

Clip: say -o clip.wav --data-format=LEF32@16000 "We deploy the app to Vercel, and the worker runs on Cloudflare."

The script below runs the two models directly with CoreML on the zero-padded 240,000-sample window and greedy-decodes ctc_head_raw_output. Output, identical on both Macs:

mel ANE, encoder ANE: (all blank)
mel ANE, encoder CPU: (all blank)
mel CPU, encoder ANE:  We deploy the Aversal and the worker runs on Cloudfler
all, all:  We deploy the Aversal and the worker runs on Cloudflure

Warm time on the M4 Max: mel CPU with encoder ANE 11 ms, default 13 ms, .all 18 ms, all CPU 39 ms.

Through FluidAudio, CtcKeywordSpotter.spotKeywordsWithLogProbs with the default CtcModels gave blank as the argmax of every frame on all 20 of our test clips. With the same CtcModels except a CPU MelSpectrogram, the log-probs decode to the speech.

repro.swift 
// swiftc -O repro.swift -o repro && ./repro ~/Library/Application\ Support/FluidAudio/Models/parakeet-ctc-110m-coreml clip.wav
import AVFoundation
import CoreML

let dir = URL(fileURLWithPath: CommandLine.arguments[1])
let file = try AVAudioFile(forReading: URL(fileURLWithPath: CommandLine.arguments[2]))  // 16 kHz mono Float32
let buffer = AVAudioPCMBuffer(pcmFormat: file.processingFormat, frameCapacity: AVAudioFrameCount(file.length))!
try file.read(into: buffer)
let audio = Array(UnsafeBufferPointer(start: buffer.floatChannelData![0], count: Int(buffer.frameLength)))
let vocab = try JSONSerialization.jsonObject(with: Data(contentsOf: dir.appendingPathComponent("vocab.json"))) as! [String: Any]

let input = try MLMultiArray(shape: [240_000], dataType: .float16)
for i in 0..<240_000 { input[i] = NSNumber(value: i < audio.count ? audio[i] : 0) }  // explicitly zero-padded
let flag = try MLMultiArray(shape: [1, 1, 1, 1], dataType: .float16)
flag[0] = 1

func load(_ name: String, _ units: MLComputeUnits) throws -> MLModel {
    let config = MLModelConfiguration()
    config.computeUnits = units
    return try MLModel(contentsOf: dir.appendingPathComponent(name), configuration: config)
}

for (label, melUnits, encoderUnits) in [("mel ANE, encoder ANE", MLComputeUnits.cpuAndNeuralEngine, MLComputeUnits.cpuAndNeuralEngine),
                                        ("mel ANE, encoder CPU", .cpuAndNeuralEngine, .cpuOnly),
                                        ("mel CPU, encoder ANE", .cpuOnly, .cpuAndNeuralEngine),
                                        ("all, all", .all, .all)] {
    let mel = try load("MelSpectrogram.mlmodelc", melUnits)
    let encoder = try load("AudioEncoder.mlmodelc", encoderUnits)
    let features = try mel.prediction(from: MLDictionaryFeatureProvider(dictionary: ["audio": input]))
        .featureValue(for: "melspectrogram_features")!.multiArrayValue!
    let logits = try encoder.prediction(from: MLDictionaryFeatureProvider(dictionary: ["melspectrogram_features": features, "input_1": flag]))
        .featureValue(for: "ctc_head_raw_output")!.multiArrayValue!  // [1, 1025, 1, 188]
    var text = "", last = -1
    for t in 0..<min(188, audio.count * 188 / 240_000 + 1) {
        let best = (0..<1025).max { logits[[0, $0, 0, t] as [NSNumber]].floatValue < logits[[0, $1, 0, t] as [NSNumber]].floatValue }!
        if best != last, best != 1024 { text += (vocab[String(best)] as? String) ?? "?" }
        last = best
    }
    print("\(label): \(text.isEmpty ? "(all blank)" : text.replacingOccurrences(of: "▁", with: " "))")
}

Workaround

Build CtcModels with the MelSpectrogram reloaded on the CPU:

let models = try await CtcModels.downloadAndLoad(variant: .ctc110m)
let cpu = MLModelConfiguration()
cpu.computeUnits = .cpuOnly
let mel = try MLModel(contentsOf: CtcModels.defaultCacheDirectory(for: .ctc110m)
    .appendingPathComponent(ModelNames.CTC.melSpectrogramPath), configuration: cpu)
let ctcModels = CtcModels(melSpectrogram: mel, encoder: models.encoder, configuration: models.configuration,
                          vocabulary: models.vocabulary, variant: models.variant)

Suggested fix

Load the MelSpectrogram with .cpuOnly in CtcModels.load and loadDirect, and keep the encoder on .cpuAndNeuralEngine. A test that a short speech clip doesn't decode to all blanks would catch a regression. We have only tested macOS 27.0.1, so it may be worth checking whether earlier macOS versions are affected too.

Because the CTC evidence ignores the audio under the default configuration, this may also explain part of the common-word replacements reported in #967.

Lingua principale
Swift
Stelle
3k
Fork
451
Merge medio
1g 4h
PR unite (30g)
60

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di FluidInference/FluidAudio

Tutte le issue di FluidInference/FluidAudio

Issue simili

Altre issue su Swift

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.