Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Streaming RNNT wedges after sustained zero-PCM silence; Vulkan aborts with GGML_ASSERT(ne3 == ne13)

Abierto
  • ¿Algo no está bien?
#48 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
audio-video-rtc, backend

Línea de trabajo

Comienza en la ruta de streaming WebSocket /v1/realtime y en CacheAwareEncoder::encode, utilizando el comando de servidor proporcionado y la matriz de tiempos zero-PCM para reproducir el problema en CPU y Vulkan. Se considera terminado cuando la voz posterior a un silencio zero-PCM completo y sostenido produce deltas y un final, y el mismo patrón ya no alcanza la aserción de Vulkan reportada.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

The streaming RNNT server (Nemotron-3.5, cache-aware) wedges mid-stream after a long run of zero-PCM silence: the next real speech produces no deltas and no final until the client tears the stream down. On the Vulkan backend the same session can instead abort overnight with GGML_ASSERT(ne3 == ne13) inside CacheAwareEncoder::encode.

Reproduced in full isolation (a fresh server, a raw WebSocket client, no application code), on both the Vulkan and CPU backends, so this is in the server/streaming path, not a client bug.

Environment

  • Binary: nemo-speech 0.1.0 (manually deployed linux x86_64 vulkan build; exact source commit unknown)
  • OS/GPU: Arch Linux, kernel 7.2.3-arch1-3; Intel Integrated GPU (LNL), Mesa Vulkan 1.4.354
  • Model: nemotron-3.5 (indexed nvidia/nemotron-3.5-asr-streaming-0.6b, rev 1c8deaecc64b91f034d73e08dd8b64625eb3395d), RNNT head
  • Serve: nemo-speech serve --asr-model nemotron-3.5 --device vulkan:0|cpu --no-ui --no-warmup --asr.endpointing.enable=true --host 127.0.0.1 --port 8080

Detailed repro

Client opens GET /v1/realtime (WebSocket), sends one session.update (language=es-ES, sample_rate=16000, endpointing_ms=2500), then streams 16 kHz mono PCM16 in 160 ms frames paced in real time. The stimulus is a fixed 4.9 s Spanish clip followed by trailing silence (to trigger a .completed), then a "gap", then the same clip again.

gap contents 15s 20s 25s 60s result
wire-silent (no bytes) — 2 finals 2 finals — 2nd clip decodes fine
1 ms zero-PCM frames every 160 ms — — — 2 finals (30/45/60s) 2nd clip decodes fine
16/40/80 ms zero-PCM frames every 160 ms 2 finals 2 finals 2 finals — 2nd clip decodes fine
160 ms zero-PCM frames (full) 1 final 1 final 1 final 1 final 2nd clip produces 0 deltas, 0 finals

Notes:

  • Full 160 ms zero-PCM frames are exactly what a downstream noise gate emits while idle, so this is a real-world pattern, not synthetic.
  • input_audio_buffer.clear during the gap does not recover the stream — the wedge persists.
  • With --asr.endpointing.enable off, 60 s of full zero-PCM frames still decodes continuously (no EOU, but hundreds of deltas), so the wedge is specific to the token-silence EOU path, not the base cache-aware RNNT decoder.

Crash (Vulkan backend)

Running the same streaming pattern (real speech + long idle silences) on the Vulkan backend, the server eventually aborts:

GGML_ASSERT(ne3 == ne13) failed
/work/ggml/src/ggml-cpu/ggml-cpu.c:1270

Backtrace (excerpt, coredumpctl):

#3  ggml_abort (libggml-base.so.0)
#4  ggml_compute_forward_mul_mat (libggml-cpu.so.0)
#11 ggml_backend_sched_graph_compute_async (libggml-base.so.0)
#13 CacheAwareEncoder::EncoderBatcher::...::operator() (libnemo_speech_asr.so)
#14 CacheAwareEncoder::encode (libnemo_speech_asr.so)

Nine nemo-speech SIGABRT cores exist on this machine, all the ne3 == ne13 assert (mix of the known intermittent warmup abort and this mid-stream form). The mid-stream one occurs specifically while decoding a live stream that has sat idle in zero-PCM silence.

Hypothesis

Token-silence EOU ends a long silence run by leaving decoder/lattice state that a subsequent fire_eou + reset_utterance does not fully clear (clear also does not). The next non-silent chunk is still emitted into that wedged state, so it decodes nothing. On Vulkan the same residual state surfaces on the next CacheAwareEncoder::encode as a cache-dimensions mismatch (ne3 == ne13), aborting the process.

Lenguaje dominante
C++
Estrellas
181
Forks
42
Merge medio
1 d 8 h
PR fusionados (30 d)
8

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/NeMo-Speech.cpp

Todos los issues de NVIDIA/NeMo-Speech.cpp

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.