[BUG] nvmultiurisrcbin: removing the last source by REST causes a GPU MMU fault (Xid 31) and leaves the pipeline dead

未关闭
  • 哪里不对？
#111 5 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@wyw1267 已经在做这个了。

开始于 2026年11月3日。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
18/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
cpp, docker, python
领域
audio-video-rtc, computer-vision

调研方向

The fault is reported when the last source is removed from nvmultiurisrcbin through its REST /api/v1/stream/remove path, so start from the source-removal code in that element and the buffer lifetime of the removed source's decoder surfaces. Reproduce with the attached repro.py last script on a GPU host, then check journalctl -k for Xid 31. Done means removing the last source causes no GPU fault or downstream error, and a source added afterwards delivers frames.

由索引模型根据 Issue 内容生成。

描述

bug

Describe the bug

When nvmultiurisrcbin has one RTSP source and that source is removed through its REST API (POST /api/v1/stream/remove), the GPU reports an MMU fault (Xid 31) about 20 ms after the REST answer. nvvideoconvert then logs buffer transform failed, and nvinfer stops with Internal data stream error ... streaming stopped, reason error (-5).

The pipeline does not recover. A source added afterwards gets a 200 answer and a "new stream added" line, but no frame reaches the pipeline. The process keeps running.

Removing a source while another one stays is fine: no fault, and a source added afterwards delivers frames.

The pipeline below reads the frame on the GPU after nvinfer (nvvideoconvert, nvmultistreamtiler and the encoder). This suggests the removed source bin frees its decoder surfaces while a batch that still refers to them is in flight downstream.

Steps/Code to reproduce bug

Start an RTSP camera that serves the DeepStream sample clip:

mkdir -p clip && docker run --rm --entrypoint cat ghcr.io/nvidia/deepstream:9.2-triton-multiarch \
  /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 > clip/sample_720p.mp4
docker run -d --name cam --network host -e MTX_RTSPADDRESS=:8620 -e MTX_MOQ=no \
  -e MTX_PATHS_CAM1_RUNONINIT='ffmpeg -re -stream_loop -1 -i /v/sample_720p.mp4 -c copy -f rtsp rtsp://localhost:8620/cam1' \
  -v "$PWD/clip:/v:ro" bluenviron/mediamtx:1.21.2-ffmpeg

Save as repro.py. It uses the samples' own nvinfer config (TrafficCamNet), so nvinfer builds the engine first, which takes about 40 s:

import json
import os
import sys
import threading
import time
import urllib.error
import urllib.request

from pyservicemaker import BatchMetadataOperator, Pipeline, Probe

# usage: repro.py last     add cam-a, remove cam-a (the only camera), add cam-b
#        repro.py control  add cam-a and cam-b, remove cam-a (cam-b stays), add cam-c
MODE = sys.argv[1]
URL = sys.argv[2]
PORT = 19556
CONFIG = "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.yml"
frames = 0


class Count(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        global frames
        frames += len(list(batch_meta.frame_items))


def log(what):
    print(f"{time.strftime('%H:%M:%S')}.{int(time.time() * 1000) % 1000:03d} {what} (frames so far: {frames})", flush=True)


def post(verb, camera):
    value = {"camera_id": camera, "camera_name": camera, "camera_url": URL, "change": f"camera_{verb}"}
    body = json.dumps({"key": "sensor", "value": value}).encode()
    request = urllib.request.Request(f"http://127.0.0.1:{PORT}/api/v1/stream/{verb}", data=body, method="POST")
    opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
    for _ in range(1200):  # the REST server answers once the pipeline plays, after the engine build
        try:
            with opener.open(request, timeout=15) as response:
                return response.status
        except urllib.error.HTTPError as error:
            return error.code
        except urllib.error.URLError:
            time.sleep(0.25)
    return -1


def wait_for_frames(count, seconds):
    target, deadline = frames + count, time.monotonic() + seconds
    while frames < target and time.monotonic() < deadline:
        time.sleep(0.2)
    return frames >= target


unsynced = {"sync": False, "async": False}
pipeline = Pipeline("repro")
pipeline.add("nvmultiurisrcbin", "src", {
    "ip-address": "127.0.0.1", "port": str(PORT), "max-batch-size": 2, "width": 1280, "height": 720,
    "live-source": True, "drop-pipeline-eos": True, "select-rtp-protocol": 4, "batched-push-timeout": 33333,
    "rtsp-reconnect-interval": 5, "init-rtsp-reconnect-interval": 5, "rtsp-reconnect-attempts": -1,
})
pipeline.add("nvinfer", "det", {"config-file-path": CONFIG, "batch-size": 2})
pipeline.add("nvvideoconvert", "conv_rgba", {})
pipeline.add("capsfilter", "caps_rgba", {"caps": "video/x-raw(memory:NVMM),format=RGBA"})
pipeline.add("nvmultistreamtiler", "tiler", {"width": 1280, "height": 720, "rows": 1, "columns": 1})
pipeline.add("nvvideoconvert", "conv_nv12", {})
pipeline.add("capsfilter", "caps_nv12", {"caps": "video/x-raw(memory:NVMM),format=NV12"})
pipeline.add("nvv4l2h264enc", "enc", {"bitrate": 4000000})
pipeline.add("h264parse", "parse", {})
pipeline.add("mp4mux", "mux", {"fragment-duration": 1000})
pipeline.add("filesink", "sink", {"location": "/tmp/out.mp4", **unsynced})
pipeline.link("src", "det", "conv_rgba", "caps_rgba", "tiler", "conv_nv12", "caps_nv12", "enc", "parse", "mux", "sink")
pipeline.attach("det", Probe("count", Count()))
pipeline.start(lambda message: None)
threading.Thread(target=pipeline.wait, daemon=True).start()

log(f"add cam-a: {post('add', 'cam-a')}")
if MODE == "control":
    log(f"add cam-b: {post('add', 'cam-b')}")
log(f"frames flowing: {wait_for_frames(60, 60)}")
time.sleep(3)
log(f"remove cam-a: {post('remove', 'cam-a')}")
time.sleep(5)
before = frames
again = "cam-b" if MODE == "last" else "cam-c"
log(f"add {again}: {post('add', again)}")
time.sleep(15)
log(f"frames in the 15 s after the add: {frames - before}")
os._exit(0)  # the pipeline never ends (drop-pipeline-eos), so leave at once

Run it in the stock container (pyyaml is installed first because import pyservicemaker needs it, see #98):

docker run --rm --gpus all --network host -v "$PWD":/repro --entrypoint bash ghcr.io/nvidia/deepstream:9.2-triton-multiarch \
  -c "pip install -q --break-system-packages pyyaml && python3 /repro/repro.py last rtsp://127.0.0.1:8620/cam1"
journalctl -k --since "-3 min" | grep -i xid

last prints (UTC):

14:24:07.609 add cam-a: 200 (frames so far: 0)
14:24:48.429 frames flowing: True (frames so far: 65)
14:24:51.463 remove cam-a: 200 (frames so far: 155)
14:24:56.501 add cam-b: 200 (frames so far: 156)
14:25:11.501 frames in the 15 s after the add: 0 (frames so far: 156)

The kernel log, 22 ms after the remove answer (host time is UTC+2):

Nov 01 16:24:51.485729 kernel: NVRM: Xid (PCI:0000:01:00): 31, pid=2056812, name=python3, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_4 faulted @ 0x5_0f215000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

and stderr:

new stream removed [0:cam-a]
WARNING from src_bin_muxer: No Sources found at the input of muxer. Waiting for sources.
0:00:44.810769009     1 0x78b38c5cdd30 ERROR         nvvideoconvert gstnvvideoconvert.c:4359:gst_nvvideoconvert_transform: buffer transform failed
0:00:44.810881429     1 0x78b38c5cdd30 WARN                 nvinfer gstnvinfer.cpp:2642:gst_nvinfer_output_loop:<det> error: Internal data stream error.
0:00:44.810889459     1 0x78b38c5cdd30 WARN                 nvinfer gstnvinfer.cpp:2642:gst_nvinfer_output_loop:<det> error: streaming stopped, reason error (-5)
ERROR from det: Internal data stream error.
...
new stream added [1:cam-b:cam-b]
WARNING from src: Internal data stream error.

control (two sources, remove one, add a third) prints no Xid:

14:25:16.000 add cam-a: 200 (frames so far: 0)
14:25:17.034 add cam-b: 200 (frames so far: 0)
14:26:00.258 frames flowing: True (frames so far: 60)
14:26:03.292 remove cam-a: 200 (frames so far: 244)
14:26:08.329 add cam-c: 200 (frames so far: 392)
14:26:23.329 frames in the 15 s after the add: 675 (frames so far: 1066)

With this pipeline, last gave the Xid in 9 of 10 runs (6 of 6 in the stock image), and at most 1 frame after the add in all 10. control gave no Xid in 7 of 7.

Expected behavior

  1. Removing the last source frees nothing that a buffer still in the pipeline uses. No GPU fault, and no error from downstream elements.
  2. The element is then in the same state as one started with no sources: the muxer waits for sources, and a source added later delivers frames.

Environment details

  • Environment location: Docker on bare-metal (Ubuntu 26.04.1 LTS host, kernel 7.0.0-42, NVIDIA RTX A4000 Laptop GPU 8 GB, driver 615.71.09)
  • Method of DeepStream install: Docker, ghcr.io/nvidia/deepstream:9.2-triton-multiarch (digest sha256:94a49674eb475f0416ce49742430cfbd644258b4e6512b09c76ddd90164c6dc7), DeepStream 9.2.0, pyservicemaker 0.0.1, TensorRT 10.16.0.72, CUDA 13.2
  • Docker command: see the steps above
  • scripts/print_env.sh output, run inside the same container:
Click here to see environment details
 **git***
 Not inside a git repository
 
 ***OS Information***
 DISTRIB_ID=Ubuntu
 DISTRIB_RELEASE=24.04
 DISTRIB_CODENAME=noble
 DISTRIB_DESCRIPTION="Ubuntu 24.04.4 LTS"
 PRETTY_NAME="Ubuntu 24.04.4 LTS"
 NAME="Ubuntu"
 VERSION_ID="24.04"
 VERSION="24.04.4 LTS (Noble Numbat)"
 VERSION_CODENAME=noble
 ID=ubuntu
 ID_LIKE=debian
 HOME_URL="https://www.ubuntu.com/"
 SUPPORT_URL="https://help.ubuntu.com/"
 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
 UBUNTU_CODENAME=noble
 LOGO=ubuntu-logo
 Linux 60bb4ba654f1 7.0.0-42-generic #42-Ubuntu SMP PREEMPT_DYNAMIC Thu Oct 15 08:06:44 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
 
 ***GPU Information***
 Sun Nov  1 14:26:26 2026
 +-----------------------------------------------------------------------------------------+
 | NVIDIA-SMI 615.71.09              KMD Version: 615.71.09     CUDA UMD Version: 13.4     |
 +-----------------------------------------+------------------------+----------------------+
 | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
 | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
 |                                         |                        |               MIG M. |
 |=========================================+========================+======================|
 |   0  NVIDIA RTX A4000 Laptop GPU    Off |   00000000:01:00.0 Off |                  N/A |
 | N/A   61C    P8             22W /   80W |      13MiB /   8192MiB |      0%      Default |
 |                                         |                        |                  N/A |
 +-----------------------------------------+------------------------+----------------------+
 
 +-----------------------------------------------------------------------------------------+
 | Processes:                                                                              |
 |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
 |        ID   ID                                                               Usage      |
 |=========================================================================================|
 |  No running processes found                                                             |
 +-----------------------------------------------------------------------------------------+
 
 ***CPU***
 Architecture:                            x86_64
 CPU op-mode(s):                          32-bit, 64-bit
 Address sizes:                           39 bits physical, 48 bits virtual
 Byte Order:                              Little Endian
 CPU(s):                                  16
 On-line CPU(s) list:                     0-15
 Vendor ID:                               GenuineIntel
 Model name:                              11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz
 CPU family:                              6
 Model:                                   141
 Thread(s) per core:                      2
 Core(s) per socket:                      8
 Socket(s):                               1
 Stepping:                                1
 CPU(s) scaling MHz:                      34%
 CPU max MHz:                             5000.0000
 CPU min MHz:                             800.0000
 BogoMIPS:                                5222.40
 Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear ibt flush_l1d arch_capabilities
 Virtualization:                          VT-x
 L1d cache:                               384 KiB (8 instances)
 L1i cache:                               256 KiB (8 instances)
 L2 cache:                                10 MiB (8 instances)
 L3 cache:                                24 MiB (1 instance)
 NUMA node(s):                            1
 NUMA node0 CPU(s):                       0-15
 Vulnerability Gather data sampling:      Vulnerable
 Vulnerability Ghostwrite:                Not affected
 Vulnerability Indirect target selection: Mitigation; Aligned branch/return thunks
 Vulnerability Itlb multihit:             Not affected
 Vulnerability L1tf:                      Not affected
 Vulnerability Mds:                       Not affected
 Vulnerability Meltdown:                  Not affected
 Vulnerability Mmio stale data:           Not affected
 Vulnerability Old microcode:             Not affected
 Vulnerability Reg file data sampling:    Not affected
 Vulnerability Retbleed:                  Not affected
 Vulnerability Spec rstack overflow:      Not affected
 Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
 Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
 Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
 Vulnerability Srbds:                     Not affected
 Vulnerability Tsa:                       Not affected
 Vulnerability Tsx async abort:           Not affected
 Vulnerability Vmscape:                   Not affected
 
 ***CMake***
 /usr/bin/cmake
 cmake version 3.28.3
 
 CMake suite maintained and supported by Kitware (kitware.com/cmake).
 
 ***g++***
 /usr/bin/g++
 g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
 Copyright (C) 2023 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
 
 ***nvcc***
 /usr/local/cuda/bin/nvcc
 nvcc: NVIDIA (R) Cuda compiler driver
 Copyright (c) 2005-2026 NVIDIA Corporation
 Built on Mon_Mar_02_09:52:23_PM_PST_2026
 Cuda compilation tools, release 13.2, V13.2.51
 Build cuda_13.2.r13.2/compiler.37434383_0
 
 ***Python***
 
 ***Environment Variables***
 PATH                            : /opt/proto/bin:/opt/tritonserver/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/amazon/efa/bin:/.local/bin
 LD_LIBRARY_PATH                 : /opt/tritonserver/lib:/opt/tritonclient/lib:/usr/src/tensorrt/lib:/opt/riva/lib:/usr/local/cuda-13/lib64:/usr/lib/x86_64-linux-gnu:/opt/hpcx/ucc/lib/:/opt/hpcx/ucx/lib/:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.local/lib
 NUMBAPRO_NVVM                   :
 NUMBAPRO_LIBDEVICE              :
 CONDA_PREFIX                    :
 PYTHON_PATH                     :
 
 conda not found
 ***pip packages***
 /usr/bin/pip
 Package             Version
 ------------------- ---------------
 annotated-doc       0.0.4
 annotated-types     0.7.0
 anyio               4.12.1
 apparmor            4.0.1
 certifi             2026.2.25
 cupy-cuda13x        13.6.0
 distlib             0.4.0
 distro              1.9.0
 fastapi             0.143.3
 fastrlock           0.8.3
 filelock            3.25.2
 h11                 0.16.0
 httpcore            1.0.9
 httpx               0.27.2
 idna                3.11
 iniconfig           2.3.0
 jaraco.context      6.1.0
 jiter               0.13.0
 LibAppArmor         4.0.1
 numpy               1.26.4
 openai              1.107.3
 opentelemetry-api   1.45.1
 packaging           24.0
 partial-json-parser 0.2.1.1.post7
 pip                 24.0
 platformdirs        4.9.4
 pluggy              1.6.0
 pydantic            2.10.6
 pydantic_core       2.27.2
 Pygments            2.19.2
 pyservicemaker      0.0.1
 pytest              9.0.2
 python-apt          2.7.7+ubuntu5.3
 python-discovery    1.1.3
 scipy               1.16.3
 setuptools          78.1.1
 sniffio             1.3.1
 starlette           0.52.1
 tqdm                4.67.3
 tritonfrontend      2.67.0
 tritonserver        0.0.0
 typing_extensions   4.15.0
 typing-inspection   0.4.4
 virtualenv          21.2.0
 wheel               0.42.0

主要语言
C++
星标
310
派生
73
平均合并
8 小时 3 分钟
30 天内合并 PR
24

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA/DeepStream 的其他 Issue

查看 NVIDIA/DeepStream 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。