Host memory grows ~6.8 KB per inference on aarch64 + CUDA (not on x86_64, not on CPU)
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 45/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Ambito
- backend, machine-learning, performance
Direzione di ricerca
Inizia con la riproduzione Python fornita usando WhisperModel.transcribe e verifica la crescita lineare dell’host-RSS su aarch64 con CUDA. Esamina il percorso CudaAsyncAllocator e confrontalo con CT2_CUDA_ALLOCATOR=cub_caching, quindi traccia il codice C++ CUDA pertinente per l’inferenza. Il lavoro è completato quando la pendenza viene eliminata per inferenze ripetute a forma fissa, mentre il comportamento riportato su CPU e x86_64 rimane invariato.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Host memory grows ~6.8 KB per inference on aarch64 + CUDA (not on x86_64, not on CPU)
Summary
Running WhisperModel.transcribe() in a loop with a resident model and fixed-shape
input leaks about 6.8 KB of host RSS per call on aarch64 + CUDA. The growth is
strictly linear and is not recovered by gc.collect() or malloc_trim(0).
The same test shows no growth on x86_64 + CUDA and no growth on aarch64 + CPU,
using the same CTranslate2 version (4.8.1) and the same script.
At a realistic streaming-ASR rate (one 5-second chunk every 5 seconds) this is
~117 MB/day, which we confirmed in a 24-hour run.
This is host memory (RSS), not GPU memory — which distinguishes it from the
previously reported GPU-side growth in #1488 and SYSTRAN/faster-whisper#660.
Environment matrix
All rows: CTranslate2 4.8.1, faster-whisper 1.2.1, model Systran/faster-whisper-small,
compute_type=float16 (CPU row: float32), beam_size=1, no VAD,
input = twelve fixed 5-second 16 kHz mono WAV files cycled in order.
| # | CPU arch | Device | CUDA | CT2 build | Python | OS | Result |
|---|---|---|---|---|---|---|---|
| 1 | x86_64 (i9-14900KF, RTX 4090) | CUDA | 12.4 | official PyPI wheel | 3.10 | Ubuntu 22.04 | no growth |
| 2 | aarch64 (Jetson AGX Orin) | CPU | – | official PyPI wheel | 3.12 | Ubuntu 24.04 | no growth |
| 3 | aarch64 (Jetson AGX Orin, sm_87) | CUDA | 13.2 (JetPack 7.2) | built from source | 3.12 | Ubuntu 24.04 | +6.8 KB/call |
| 4 | aarch64 (NVIDIA GB10, sm_121) | CUDA | 13.0 (plain Ubuntu) | built from source | 3.12 | Ubuntu 24.04 | +6.8 KB/call |
Rows 3 and 4 are different machines, different GPUs, different CUDA versions, and were
built independently — yet produce the same per-call figure.
Measurements
Every 250 iterations we call gc.collect() then malloc_trim(0), and record RSS after
each. The table shows RSS after malloc_trim(0) (i.e. memory that cannot be reclaimed).
| iterations | Jetson AGX Orin (row 3) | GB10 (row 4) |
|---|---|---|
| 250 | 1533.3 MB | 808.4 MB |
| 500 | 1594.9 MB | 810.2 MB (+1.8) |
| 750 | 1596.6 MB (+1.7) | 812.0 MB (+1.8) |
| 1000 | 1598.3 MB (+1.7) | 813.6 MB (+1.6) |
| 1250 | 1600.0 MB (+1.7) | 815.2 MB (+1.6) |
+1.6–1.8 MB per 250 calls = ~6.8 KB per call, on both machines, with no sign of
levelling off. (The larger jump before 500 iterations is the allocator pool filling up;
that part does saturate. The linear part after it does not.)
x86_64 (row 1) over the same 1250 iterations: -0.8 to +0.4 MB, i.e. flat.
What we ruled out
-
Python-side accumulation —
gc.collect()reclaims 0.0 MB and the number of
objects tracked bygcgrows by +5 over 1250 iterations, on all four rows. -
glibc arena retention —
malloc_trim(0)does reclaim a steady amount every time
(7.4–66.6 MB depending on the machine), but it does not change the slope. -
The CUDA allocator implementation — we ran row 4 again with
CT2_CUDA_ALLOCATOR=cub_caching. The switch demonstrably took effect (the amount
reclaimed bymalloc_trimchanged from 31.6 MB to 7.3 MB per call), but the slope was
identical:iterations cuda_malloc_async(default)cub_caching250 +2.3 MB +2.0 MB 500 +4.1 MB +4.1 MB 750 +5.9 MB +5.4 MB 1000 +7.5 MB +7.5 MB 1250 +9.0 MB +9.3 MB So this is not
CudaAsyncAllocator/cudaMallocAsync, which is what
SYSTRAN/faster-whisper#660 points at for a different (crash) symptom. -
A broken local build — rows 3 and 4 were configured and compiled separately, on
different machines, for different compute capabilities, and agree to the same figure. -
Variable input shapes — every call uses one of the same twelve 5-second files, so
shape-keyed caches cannot explain unbounded growth.
Reproduction
import ctypes, gc, sys, time
from faster_whisper import WhisperModel
libc = ctypes.CDLL("libc.so.6")
def rss_mb():
for line in open("/proc/self/status"):
if line.startswith("VmRSS:"):
return int(line.split()[1]) / 1024
chunks = sys.argv[1:] # twelve 5-second 16 kHz mono WAV files
model = WhisperModel("small", device="cuda", compute_type="float16")
list(model.transcribe(chunks[0], language="ja", beam_size=1)[0])
gc.collect(); libc.malloc_trim(0)
base, base_obj = rss_mb(), len(gc.get_objects())
print(f"[ready] rss={base:.1f}MB obj={base_obj}")
for i in range(1, 1251):
segments, _ = model.transcribe(chunks[i % len(chunks)], language="ja", beam_size=1)
_ = "".join(s.text for s in segments)
if i % 250 == 0:
r0 = rss_mb(); gc.collect()
r1 = rss_mb(); libc.malloc_trim(0)
r2 = rss_mb()
print(f"{i:5d} rss={r0:7.1f} after_gc={r1:7.1f} after_trim={r2:7.1f} "
f"net={r2-base:+6.1f}MB gc_objects={len(gc.get_objects())-base_obj:+d}")
Build configuration used for rows 3 and 4 (stock apart from the arch list):
-DWITH_CUDA=ON -DWITH_CUDNN=ON -DWITH_MKL=OFF -DWITH_OPENBLAS=OFF -DWITH_DNNL=OFF
-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DOPENMP_RUNTIME=COMP
Note for row 4: CMake 3.28's
cuda_select_nvcc_arch_flagsdoes not recognise sm_121,
so that call was replaced with an explicit
-gencode arch=compute_121,code=sm_121. Row 3 (sm_87) was built without any patch and
produces the same figure, so this workaround is not related to the leak.
Impact
For always-on speech recognition this is the difference between a process that runs
indefinitely and one that has to be restarted. On an 8 GB Jetson Orin Nano the practical
budget is about 20 days; we currently restart the recogniser process once a day as a
workaround (reloading the model takes ~6 s, so this is cheap — but it should not be
necessary).
Related
- #1488 — GPU memory growth on x86_64 with variable input sizes. Different symptom
(device memory, not host RSS) and different trigger (variable shapes). - SYSTRAN/faster-whisper#660 — several reports there point at
ctranslate2::cuda::cudaasyncallocator::free(), but that is a crash signature; our
cub_cachingrun above shows the steady host-side growth happens without that
allocator.
- Lingua principale
- C++
- Stelle
- 4.8k
- Fork
- 557
- Merge medio
- 1g 6h
- PR unite (30g)
- 24
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di OpenNMT/CTranslate2
-
MPS: GPT-2 produces NaN logits and invalid token IDsForse già presa @TBO22 l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
OpenNMT/CTranslate2#2154 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Allow no-timestamp Whisper decoding beyond 224 tokensForse già presa @Emre-Akgul l’ha presa 127 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
OpenNMT/CTranslate2#2074 ·
I maintainer di solito rispondono entro 1 giorno
-
T5Gemma fails on MPS: incorrect beam expansion with padded batchesForse già presa @TBO22 l’ha presa oggi. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
OpenNMT/CTranslate2#2153 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 8/100
OpenNMT/CTranslate2#2152 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 55/100
OpenNMT/CTranslate2#2149 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di OpenNMT/CTranslate2
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
grizzlypeak3d/DJV#962 ·
-
cognizant-backlog
Difficoltà 2/5 1-3 ore Idoneità per principianti 61/100
KhronosGroup/OpenCL-CTS#2882 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Documentation: Fix parameter markup, legacy branch link, and phrasing in Detachable Joints and Model Command tutorialsForse già presa @patelshivani2283-lab l’ha presa oggi. Aperta
Difficoltà 1/5 1-3 ore Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 73/100
LunarG/gfxreconstruct#3440 ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
I maintainer di solito rispondono entro 1 giorno