Host memory grows ~6.8 KB per inference on aarch64 + CUDA (not on x86_64, not on CPU)

Aperta
#2,088 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, python

Direzione di ricerca

Inizia con la riproduzione Python fornita usando WhisperModel.transcribe e verifica la crescita lineare dell’host-RSS su aarch64 con CUDA. Esamina il percorso CudaAsyncAllocator e confrontalo con CT2_CUDA_ALLOCATOR=cub_caching, quindi traccia il codice C++ CUDA pertinente per l’inferenza. Il lavoro è completato quando la pendenza viene eliminata per inferenze ripetute a forma fissa, mentre il comportamento riportato su CPU e x86_64 rimane invariato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Host memory grows ~6.8 KB per inference on aarch64 + CUDA (not on x86_64, not on CPU)

Summary

Running WhisperModel.transcribe() in a loop with a resident model and fixed-shape
input
leaks about 6.8 KB of host RSS per call on aarch64 + CUDA. The growth is
strictly linear and is not recovered by gc.collect() or malloc_trim(0).

The same test shows no growth on x86_64 + CUDA and no growth on aarch64 + CPU,
using the same CTranslate2 version (4.8.1) and the same script.

At a realistic streaming-ASR rate (one 5-second chunk every 5 seconds) this is
~117 MB/day, which we confirmed in a 24-hour run.

This is host memory (RSS), not GPU memory — which distinguishes it from the
previously reported GPU-side growth in #1488 and SYSTRAN/faster-whisper#660.

Environment matrix

All rows: CTranslate2 4.8.1, faster-whisper 1.2.1, model Systran/faster-whisper-small,
compute_type=float16 (CPU row: float32), beam_size=1, no VAD,
input = twelve fixed 5-second 16 kHz mono WAV files cycled in order.

# CPU arch Device CUDA CT2 build Python OS Result
1 x86_64 (i9-14900KF, RTX 4090) CUDA 12.4 official PyPI wheel 3.10 Ubuntu 22.04 no growth
2 aarch64 (Jetson AGX Orin) CPU – official PyPI wheel 3.12 Ubuntu 24.04 no growth
3 aarch64 (Jetson AGX Orin, sm_87) CUDA 13.2 (JetPack 7.2) built from source 3.12 Ubuntu 24.04 +6.8 KB/call
4 aarch64 (NVIDIA GB10, sm_121) CUDA 13.0 (plain Ubuntu) built from source 3.12 Ubuntu 24.04 +6.8 KB/call

Rows 3 and 4 are different machines, different GPUs, different CUDA versions, and were
built independently — yet produce the same per-call figure.

Measurements

Every 250 iterations we call gc.collect() then malloc_trim(0), and record RSS after
each. The table shows RSS after malloc_trim(0) (i.e. memory that cannot be reclaimed).

iterations Jetson AGX Orin (row 3) GB10 (row 4)
250 1533.3 MB 808.4 MB
500 1594.9 MB 810.2 MB (+1.8)
750 1596.6 MB (+1.7) 812.0 MB (+1.8)
1000 1598.3 MB (+1.7) 813.6 MB (+1.6)
1250 1600.0 MB (+1.7) 815.2 MB (+1.6)

+1.6–1.8 MB per 250 calls = ~6.8 KB per call, on both machines, with no sign of
levelling off. (The larger jump before 500 iterations is the allocator pool filling up;
that part does saturate. The linear part after it does not.)

x86_64 (row 1) over the same 1250 iterations: -0.8 to +0.4 MB, i.e. flat.

What we ruled out

  • Python-side accumulation — gc.collect() reclaims 0.0 MB and the number of
    objects tracked by gc grows by +5 over 1250 iterations, on all four rows.

  • glibc arena retention — malloc_trim(0) does reclaim a steady amount every time
    (7.4–66.6 MB depending on the machine), but it does not change the slope.

  • The CUDA allocator implementation — we ran row 4 again with
    CT2_CUDA_ALLOCATOR=cub_caching. The switch demonstrably took effect (the amount
    reclaimed by malloc_trim changed from 31.6 MB to 7.3 MB per call), but the slope was
    identical:

    iterations cuda_malloc_async (default) cub_caching
    250 +2.3 MB +2.0 MB
    500 +4.1 MB +4.1 MB
    750 +5.9 MB +5.4 MB
    1000 +7.5 MB +7.5 MB
    1250 +9.0 MB +9.3 MB

    So this is not CudaAsyncAllocator / cudaMallocAsync, which is what
    SYSTRAN/faster-whisper#660 points at for a different (crash) symptom.

  • A broken local build — rows 3 and 4 were configured and compiled separately, on
    different machines, for different compute capabilities, and agree to the same figure.

  • Variable input shapes — every call uses one of the same twelve 5-second files, so
    shape-keyed caches cannot explain unbounded growth.

Reproduction

import ctypes, gc, sys, time
from faster_whisper import WhisperModel

libc = ctypes.CDLL("libc.so.6")

def rss_mb():
    for line in open("/proc/self/status"):
        if line.startswith("VmRSS:"):
            return int(line.split()[1]) / 1024

chunks = sys.argv[1:]                      # twelve 5-second 16 kHz mono WAV files
model = WhisperModel("small", device="cuda", compute_type="float16")
list(model.transcribe(chunks[0], language="ja", beam_size=1)[0])
gc.collect(); libc.malloc_trim(0)
base, base_obj = rss_mb(), len(gc.get_objects())
print(f"[ready] rss={base:.1f}MB obj={base_obj}")

for i in range(1, 1251):
    segments, _ = model.transcribe(chunks[i % len(chunks)], language="ja", beam_size=1)
    _ = "".join(s.text for s in segments)
    if i % 250 == 0:
        r0 = rss_mb(); gc.collect()
        r1 = rss_mb(); libc.malloc_trim(0)
        r2 = rss_mb()
        print(f"{i:5d}  rss={r0:7.1f}  after_gc={r1:7.1f}  after_trim={r2:7.1f}  "
              f"net={r2-base:+6.1f}MB  gc_objects={len(gc.get_objects())-base_obj:+d}")

Build configuration used for rows 3 and 4 (stock apart from the arch list):

-DWITH_CUDA=ON -DWITH_CUDNN=ON -DWITH_MKL=OFF -DWITH_OPENBLAS=OFF -DWITH_DNNL=OFF
-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DOPENMP_RUNTIME=COMP

Note for row 4: CMake 3.28's cuda_select_nvcc_arch_flags does not recognise sm_121,
so that call was replaced with an explicit
-gencode arch=compute_121,code=sm_121. Row 3 (sm_87) was built without any patch and
produces the same figure, so this workaround is not related to the leak.

Impact

For always-on speech recognition this is the difference between a process that runs
indefinitely and one that has to be restarted. On an 8 GB Jetson Orin Nano the practical
budget is about 20 days; we currently restart the recogniser process once a day as a
workaround (reloading the model takes ~6 s, so this is cheap — but it should not be
necessary).

Related

  • #1488 — GPU memory growth on x86_64 with variable input sizes. Different symptom
    (device memory, not host RSS) and different trigger (variable shapes).
  • SYSTRAN/faster-whisper#660 — several reports there point at
    ctranslate2::cuda::cudaasyncallocator::free(), but that is a crash signature; our
    cub_caching run above shows the steady host-side growth happens without that
    allocator.
Lingua principale
C++
Stelle
4.8k
Fork
557
Merge medio
1g 6h
PR unite (30g)
24

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenNMT/CTranslate2

Tutte le issue di OpenNMT/CTranslate2

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.