[Bug]: create-symlinks and update-ldcache hooks fail container create when the container rootfs is read-only

Open
  • Is something wrong?
#2,131 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@sahanasreenath is already working on this.

Since Oct 9, 2026.

  • #2132 by @sahanasreenath — open

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
go, kubernetes
Domain
devops, infrastructure

Research direction

Start from the create-symlinks and update-ldcache subcommands of nvidia-cdi-hook and the #2026 fix for update-application-profile, which already ignores read-only filesystem errors. Check how the failing write surfaces as EROFS and whether the same handling can apply to both hooks. Pull request #2132 is already open against this issue, so review it or coordinate with its author rather than starting a second patch. Done means the container starts on a read-only rootfs without the /etc and /usr/lib64 tmpfs workaround.

Written by the indexing model from the issue text.

Description

Problem

Both create-symlinks and update-ldcache write into the container root filesystem. When that filesystem is read-only, the write fails with EROFS, the hook exits 1, and container creation is aborted:

Error: container create failed: error executing hook `/usr/local/nvidia/toolkit/nvidia-cdi-hook` (exit code: 1)

This is the same class of problem as #2026, which was fixed in v1.20.1 for update-application-profile by ignoring read-only filesystem errors. The other two hooks still abort.

Generalized: a CDI hook whose only effect is to modify the container root filesystem should not prevent the container from starting when that filesystem is read-only.

Steps to reproduce
$ podman run --rm --device <cdi-device> --read-only nvidia/cuda:12.6.3-base-ubuntu22.04 nvidia-smi
Error: OCI runtime error: crun: error executing hook `/usr/local/nvidia/toolkit/nvidia-cdi-hook` (exit code: 1)

$ podman run --rm --device <cdi-device> --read-only --tmpfs /etc --tmpfs /usr/lib64 nvidia/cuda:12.6.3-base-ubuntu22.04 nvidia-smi
# succeeds

Making both /etc and /usr/lib64 writable is required; making either one writable on its own still fails. Those are the two paths the generated spec's hooks write to:

- path: /usr/local/nvidia/toolkit/nvidia-cdi-hook
  args: [nvidia-cdi-hook, create-symlinks, --link, libEGL_nvidia.so.595.58.03::/usr/lib64/libEGL_nvidia.so.0, ...]
- path: /usr/local/nvidia/toolkit/nvidia-cdi-hook
  args: [nvidia-cdi-hook, update-ldcache, --folder, /usr/lib64, --folder, /usr/lib64/vdpau]
Impact

Kubernetes workloads that set readOnlyRootFilesystem: true cannot use GPUs. This is a common hardening requirement, and it is enforced by policy on many clusters (for example OpenShift restricted-v2 style baselines and Gatekeeper/Kyverno psp-readonlyrootfilesystem constraints), so it is not something the workload owner can simply turn off.

Environment
  • nvidia-container-toolkit v1.19.0, also reproduced on v1.17.8 with regenerated CDI specs
  • CRI-O and podman/crun, OpenShift 4.x, NVIDIA GPU Operator v26.3.0
  • Driver 595.58.03, H100 NVL

Skipping the ldcache update does not make the libraries unreachable: they are bind-mounted into directories glibc already searches by default.

I have a patch following the same approach as the #2026 fix and will open a PR.

Dominant language
Go
Stars
4.6k
Forks
609
Avg merge
3d 12h
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/nvidia-container-toolkit

All issues in NVIDIA/nvidia-container-toolkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.