[Bug]: Containerized installer removes the existing Toolkit installation before the replacement is complete and durable

Open
  • Is something wrong?
#2,153 1 comment 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@henry118 is already working on this.

Since Oct 22, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, linux
Domain
backend, devops

Research direction

Start with Installer.Install in cmd/nvidia-ctk-installer/toolkit/toolkit.go, then inspect installContentStub and installToolkitConfig in the linked files. Trace how files are prepared, synchronized, and switched into the active installation, and review the reproduction steps for the failed reinstall case. Done means failed or interrupted replacement preserves the existing installation, and success follows durable writes.

Written by the indexing model from the issue text.

Description

bug needs-triage

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

Describe the bug

The containerized Toolkit installer deletes the existing installation directory, typically /usr/local/nvidia/toolkit, before preparing its replacement. It then writes the new files directly into the live directory.

The file-writing code uses os.Create, io.Copy, permission updates, and a deferred Close. It does not explicitly sync the installed files and directories to disk before returning success. Errors from the deferred Close are also not checked.

This creates two failure windows:

  • Installation fails or is interrupted: The previous installation has already been deleted, but the replacement is incomplete. A reboot that interrupts installation during this window can leave required runtime files missing or unusable. This does not require a loss of buffered writes.
  • Power loss or hard reset before writes reach disk: Even after the write operations return successfully, data may still be in memory. After reboot, installed files may be missing, incomplete, or zero-length, depending on filesystem and writeback timing.

Containers using the affected NVIDIA runtime cannot start if required runtime files are missing or unusable. If NVIDIA is the default runtime, non-GPU containers can also be affected.

If the Toolkit installer Pod itself uses the same NVIDIA runtime, it also cannot restart to repair those files, creating a circular dependency that prevents automatic recovery.

Relevant code at commit 3e112b4af299bf72a1b23193b4f5c3321475dd6d:

The same removal and file-writing patterns are present in the v1.17.8 installer.

Closing a file does not ensure its contents are persisted to disk. Directory entries also require synchronization; see fsync(2).

To Reproduce

The early-deletion behavior was reproduced with an isolated test using the installer API:

  1. Create a temporary installation directory containing a nonempty placeholder nvidia-container-runtime file.
  2. Create an installer with WithToolkitRoot pointing to that directory and WithSourceRoot pointing to a nonexistent source directory.
  3. Call Installer.Install(&cli.Command{}, &toolkit.Options{}, "containerd").
  4. After installation returns an error, check whether the original file and directory still exist.

The installer returns an error because the source library cannot be found, but the previous installation directory and its contents have already been deleted. The test output is included below.

This test only operates on a temporary directory and verifies the early-deletion behavior. The power-loss/zero-length-file scenario has not yet been reproduced in a controlled Linux crash-recovery test.

Expected behavior

A failed or interrupted reinstall should preserve a complete, usable installation.

The replacement files and directories should be validated and durably written before replacing the active installation. Installation success should be reported only after the required file and directory updates have been persisted.

Environment (please provide the following information):

The following describes the standalone installer API test:

  • nvidia-container-toolkit version: Source commit 3e112b4af299bf72a1b23193b4f5c3321475dd6d. The corresponding installation logic was also reviewed in v1.17.8.
  • NVIDIA Driver Version: N/A — no driver is used by this test.
  • Host OS: macOS.
  • Kernel Version: Not captured for the recorded test.
  • Container Runtime Version: N/A — no running container runtime is required; "containerd" is passed as the installer API argument.
  • CPU Architecture: arm64.
  • GPU Model(s): N/A — no GPU is required.
  • Kubernetes Distro and Version: N/A — the test runs outside Kubernetes.
  • NVIDIA GPU Operator version: N/A.
  • CUDA Version: N/A.

Information to attach

Standalone test output:

install_returned_error: true
old_file_exists_after_failure: false
live_directory_exists_after_failure: false

error:
could not install toolkit components: failed to collect libraries:
error locating library: libnvidia-container.so.1: not found

nvidia-smi output and container logs are not applicable to this standalone test.

Dominant language
Go
Stars
4.6k
Forks
609
Avg merge
3d 12h
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/nvidia-container-toolkit

All issues in NVIDIA/nvidia-container-toolkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.