Proposal: consume the shipped MEAI speech interface for local Whisper transcription

Read Audio\SpeechToTextService.cs and Services\VoiceService.cs first, then inspect SpeechToTextServiceTests.cs and SpeechToTextNativeProofTests.cs. Run the focused STT and lifecycle suites and review the validation guide. Done means maintainers have decided whether the bounded MEAI integration and asynchronous ownership prerequisite fit the retained contracts, with unresolved build limitations documented.

SpeechToTextService currently builds a Whisper processor directly and maps its segments into OpenClaw results. Whisper.net 1.9.0 already ships an ISpeechToTextClient adapter and already brings Microsoft.Extensions.AI.Abstractions transitively. There is an opportunity to consume that standard speech boundary without replacing Whisper, changing the model, or moving capture and voice-activity detection out of OpenClaw.

I implemented a bounded pilot on my fork's proposal/meai-whisper-stt branch. It reuses the shipped adapter, explicitly pins Abstractions 10.9.0, and preserves ordered segment text and timestamps instead of flattening the response. A separate prerequisite addresses model ownership across inference and asynchronous lifecycle calls.

This is a proposal for discussion, not a merge-ready PR. Focused and required test suites pass, and a small native comparison matches the original behavior exactly on one pinned public fixture. The full repository build remains blocked by an existing npm dependency download failure. This does not claim better transcription quality, faster inference, live microphone/UI correctness, or a Talk Mode fix.

Why this boundary?

The benefit is a standard, testable speech-client boundary while keeping the existing local provider. OpenClaw can exercise conversion, mapping, cancellation, and ownership against a controlled client without creating another Whisper adapter. This is not a new cloud routing layer or a promise that arbitrary MEAI providers have interchangeable streaming semantics.

The stable Abstractions package does not mean stable speech APIs: the contracts remain experimental (MEAI001). Suppression is local to the speech service and its two direct speech-API test files. The actual Shared, WinUI, Shared.Tests, Tray.Tests, and WinNode.Cli.Tests restored graphs resolve Abstractions 10.9.0, Whisper.net 1.9.0, and Whisper.net.Runtime 1.9.1. Abstractions adds no package dependencies for the actual .NET 10 target group.

The shipped adapter at the 1.9.0 source revision, rather than development main, determines the integration:

• Each streaming update contains one complete Whisper segment, despite its TextUpdating kind. GetTextAsync would flatten boundaries, so the service uses GetStreamingTextAsync.
• SpeechLanguage receives the existing normalized two-letter language or auto; threads remain max(1, CPU count / 2). TextLanguage stays unset because this adapter enables translation for any nonempty target language.
• GetService throws NotImplementedException. The service never depends on it. The native probe records that limitation without requiring it to remain broken; broader metadata/middleware reuse needs an upstream provider fix.
• The adapter may end enumeration normally after cancellation. Checks before, during, and after enumeration prevent a canceled partial utterance from becoming a successful result.

See also the Abstractions 10.9.0 package and immutable MEAI source. No duplicate adapter, middleware, exporter, cloud fallback, ML.NET/ONNX engine replacement, or provider registry was added.

Contracts retained

The service still accepts 16 kHz mono float[], applies the original clamp/PCM16 conversion, and creates the same WAV input. It trims nonempty segments while preserving order and exact start/end times. Returned Language retains its normalized-request meaning, including auto; it is not changed to detected language.

OpenClaw continues to own model downloading/readiness, microphone capture, resampling, VAD, utterance boundaries, permissions, and sanitized command errors. stt.transcribe retains its bounded capture of at most 30 seconds; stt.listen keeps its default 30 seconds and 1-120 second range; stt.status retains its privacy-safe readiness and engine labels. There is no MCP/gateway transport or schema change.

The service owns and disposes its generated WAV stream after enumeration. The adapter must leave caller-owned streams open; the native probe checks that separately. Streaming segment updates are not evidence of live bidirectional speech support.

Separate prerequisite: asynchronous model ownership

The original service serialized transcriptions but not load, unload, or disposal. A lifecycle operation could dispose the factory while inference used it. This is a preexisting lifetime hazard, not a defect attributed to MEAI.

Simply adding a blocking semaphore wait to synchronous APIs would introduce a UI-stall risk: VoiceService.InitializeAsync is reached from the voice overlay, and disposal is reached from settings unload/shutdown. Existing pipeline draining does not prove every inference is quiescent; agent-local pipelines are not necessarily the main pipeline being drained.

The separate prerequisite commit adds asynchronous lifecycle admission on the existing gate and awaits the two owned production call sites. It does not rewrite capture, VAD, draining, or shutdown orchestration.

The client is disposed before the separately retained eager model. The pinned factory's disposal is idempotent, so both cleanup paths free its native resource once. Derived-synchronization-context tests cover off-caller admission and controlled contention; these are not a demonstration of visible WinUI responsiveness.

Shutdown remains unbounded by this service: async disposal can await the rest of an uncanceled native inference. There is no new timeout or bounded-shutdown guarantee.

Validation and provenance

The combined source commit is 15134f93, based on 273b0182745a3093c0e09f306ca8a1fff6ef3c5a. The evidence-only head retains identical production and test source.

Tests ran on the combined worktree before those commits were created. All six final source-file Git blobs were verified against the committed tree. The machine-readable evidence identifies actual executed assemblies, package hashes, original TRX hashes, case inventories, and native metadata. The lifecycle-only intermediate commit was not independently executed.

Windows x64, SDK 10.0.401, Debug, xUnit 2.9.3, VSTest:

All five final invocations exited zero and produced nonempty TRX reports whose case outcomes and counters agree. Do not sum overlapping rows as unique coverage. Shared's 33 ordinary skips are 16 DeviceIdentity, nine LocalCommandRunner, seven MXC integration cases, and the one native Whisper case that passed separately. Native opt-in variables were cleared before ordinary suites.

The evidence guide maps requirements to exact test names and reproduction commands. The new deterministic cases cover independent expected WAV bytes/options, ordered segment mapping, empty segments, queued/in-flight cancellation with controlled fakes, no partial success, sanitized errors, single-flight reuse, disposal ordering, and primary-error preservation. The real VoiceService awaiting check is a source guard plus qualified compilation, not an executed UI lifecycle test.

The record orders 16 retained build/test commands with their actual exit codes. Twelve have original TRX runner timestamps. Exact shell/build/restore boundaries were not retained and are explicitly unavailable; runner timestamps are not presented as those boundaries.

The single required full-build attempt failed fetching the existing locked node-pty-1.2.0-beta.12.tgz from npm with ERR_SSL_SSL/TLS_ALERT_HANDSHAKE_FAILURE. It was not retried or bypassed. Documentation validation and Shared/Cli/WinNodeCli/SetupEngine builds succeeded; the full build did not.

A subsequent scoped WinUI build compiled the actual changed VoiceService, then failed copying absent wxc-exec.exe. Existing Compile, VC-runtime, and setup-asset copy targets subsequently exited zero with SkipMxcNodeBridgeRestore=true. The inspected output was bin\Debug\...\win-x64, with Platform omitted. This is qualified compile/harness evidence, not shipping, MXC, app-launch, or full-build success.

An intermediate test-helper accessibility error was corrected and all final suites rerun. Earlier passing focused/native runs and the failed full build remain attached to the earlier source patch, not relabeled as final-source proof. Complete early untracked test snapshots and early assembly hashes were not retained; the final evidence supersedes those runs.

Native comparison

The native test independently freezes the original 273b018 PCM conversion, processor construction, language handling, and segment mapping. It does not call candidate conversion or mapping helpers. Both paths receive identical samples, model/runtime, thread count, and language options.

The approved public assets are ggml-tiny.bin at ggerganov/whisper.cpp revision 5359861c739e955e79d9a303bcbc70fb988958b1 and samples/jfk.wav at ggml-org/whisper.cpp revision d09f61a708f3487afa956ff578e60eae5e7a233c. The guide records their provenance, sizes, SHA256s, and the fixture's Git-blob check, including its Git header. Assets are not downloaded automatically by tests.

Both paths received 176,000 samples (11 seconds), float-byte SHA256 ebd52851100536db02d12c49fddd010372dcdc70243562e057553d476b706ae0. Actual loaded backend was CPU, with eight configured threads. The test enumerated loaded modules and recorded hashes for whisper.dll, ggml-whisper.dll, ggml-base-whisper.dll, and ggml-cpu-whisper.dll; backend identity was not inferred from settings or available files.

Hashes cover complete ordered text, start/end timestamp ticks, and language:

All four baseline/candidate pairs and both repeat comparisons matched exactly, without relaxed tolerances. Each produced one segment. Native multi-segment corpus behavior remains unverified; deterministic tests cover that mapping.

The native probe also checks caller-stream ownership, unused-model reload/unload, and pre-canceled admission with actual token identity. Its output explicitly says cancellationScope=preCanceledAdmissionOnly; nativeMidFlightAbort=unverified. Controlled fakes establish the service's in-flight cancellation behavior, not native mid-flight abort. Durations are diagnostic, not benchmarks.

No model/audio binaries, transcript payloads, raw test logs, private paths/settings, or credentials are published.

Implementation and acceptance path

1. Review the lifecycle prerequisite independently. Keep the existing owner/gate, await owned callers, retain the explicit busy-sync policy, and avoid widening capture/drain scope.
2. Add the Abstractions pin and consume the shipped adapter. Preserve eager factory ownership, normalized SpeechLanguage, unchanged threads/WAV bytes, and unset TextLanguage.
3. Map the pinned adapter's complete streaming segments without dropping timestamps/order. Retain token checks around enumeration and sanitized capability failures. Do not require unavailable metadata or introduce middleware as part of this change.
4. Bring the deterministic tests and opt-in independent native oracle with the change. Follow the guide's exact filters and asset checks in an isolated worktree/settings directory; retain exact hashes, repetitions, nonzero execution counts, and all existing STT/MCP/CLI tests.
5. Resolve the full-build environment blocker and rerun required checks before a merge-ready PR. Collect real UI lifecycle behavior and native ARM64 parity separately.

Acceptance should preserve the local engine/model, options, result shapes, readiness, and permission behavior; demonstrate safe ownership under cancellation/reload/disposal; and reject canceled partial success. Full packaging, native abort/shutdown bounds, provider interchangeability, and live bidirectional speech require separate proof.

Alternatives and remaining scope

Keeping the concrete Whisper processor is reasonable if the experimental API and unavailable metadata outweigh the value of the standard boundary. The lifecycle prerequisite can also be considered independently of MEAI adoption, although that intermediate commit still needs its own execution evidence. A small application-specific test seam is another option, at the cost of maintaining a separate contract. There is no need to replace a working local engine merely to adopt an abstraction.

Live UI/microphone/capture/drain behavior, MCP/gateway round trips, ARM64, transcription quality, and application performance remain unverified. The applicable future proof pools are windows-winui-interactive and windows-11-arm64; neither was completed here.

Upstream main was 5a59535216eea603192f2df13d4eddf1e1a20267 at closeout, three commits beyond the retained base. Those changes do not alter this speech service, its VoiceService callers, or the Shared package manifest, but this is not a full-current-main validation claim.

This is adjacent to, not a fix for, #1087, #1124, or #1097. The experimental Abstractions 10.9.0 pin aligns with the separate Windows TTS proposal, #1528; these independent implementations have not been combined or tested together.

I can put up a PR if there is interest.