[Bug]: nvidia-container-cli fails intermittently with "nvcgo rpc error: timed out" (hard-coded 10s RPC timeout in libnvidia-container)

Open
  • Is something wrong?
#2,102 2 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@henry118 is already working on this.

Since Sep 17, 2026.

Assessment

This issue has not been assessed yet.

Description

bug needs-triage

Describe the bug

nvidia-container-cli intermittently fails during container startup with:

nvidia-container-cli: initialization error: nvcgo rpc error: timed out

The failure happens when the node is under CPU/IO pressure (e.g. a burst of GPU containers starting at the same time). The error is produced by the hard-coded 10 second RPC client timeout in libnvidia-container:

  • src/rpc.c (setup_client):
struct timeval timeout = {10, 0};
...
clnt_control(rpc->clt, CLSET_TIMEOUT, (char *)&timeout);

During nvc_init(), nvcgo_init() (src/nvcgo.c) forks a child process that serves an ONC RPC over a unix socketpair. The server-side handler nvcgo_init_1_svc dlopen()s the Go c-shared library libnvcgo.so (with RTLD_NOW), which boots a full Go runtime. On a loaded or slow node (cold page cache, CPU throttling, IO contention, container start storms), fork + dlopen + Go runtime init can exceed 10 s, so clnt_call() returns RPC_TIMEDOUT and the whole container start fails.

The timeout is hard-coded: it is not configurable via the config file, an environment variable, or a CLI flag, and there is no retry.

Note this is a different failure flavor from the well-known driver rpc error: timed out (#237, #202), which is caused by slow NVML initialization when persistence mode is off. The nvcgo RPC program only loads the Go cgroup helper library and does not touch NVML, so enabling persistence mode does not help here.

To Reproduce

  1. Use a node under heavy CPU/IO load (or artificially slow down process spawn / file reads, e.g. cold page cache + stress-ng).
  2. Start a GPU container (legacy mode), e.g. docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi, or launch a burst of GPU pods on Kubernetes with the legacy runtime mode.
  3. Intermittently, container creation fails with nvidia-container-cli: initialization error: nvcgo rpc error: timed out.

Expected behavior

The nvcgo RPC initialization should be resilient to slow nodes:

  • the RPC timeout should be configurable (e.g. via nvidia-container-cli config / env var), and/or
  • the default should be more tolerant, and/or
  • the call should be retried before failing the container start.

Environment (please provide the following information):

  • nvidia-container-toolkit version: v1.17.8 (also verified in the latest release v1.20.0, see below)
  • NVIDIA Driver Version: 550.x
  • Host OS: Ubuntu 22.04
  • Kernel Version: 5.15-generic
  • Container Runtime Version: containerd 1.7.x
  • CPU Architecture: x86_64
  • Kubernetes Distro and Version: K8s

Additional context

I verified that the issue is still present in the latest release:

  • nvidia-container-toolkit v1.20.0 pins third_party/libnvidia-container at 3e428194 (libnvidia-container v1.20.0).
  • In libnvidia-container v1.20.0 (and current main), src/rpc.c still contains the same hard-coded struct timeval timeout = {10, 0}; there were no timeout-related changes to src/rpc.c between v1.17.8 and v1.20.0.
  • nvcgo_init() is called unconditionally from nvc_init() in builds with WITH_NVCGO=yes (the default for official packages), so every nvidia-container-cli configure invocation pays this fork+dlopen cost and is exposed to the 10 s budget.

Workarounds we are considering on our side: switch the runtime to CDI mode (which does not invoke nvidia-container-cli), reduce node-level start bursts. A configurable timeout would still be appreciated.

@elezar @klueska

Dominant language
Go
Stars
4.6k
Forks
609
Avg merge
3d 18h
Merged PRs (30d)
37

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/nvidia-container-toolkit

All issues in NVIDIA/nvidia-container-toolkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.