[Bug]: an empty --drop-in-config overwrites the engine config with drop-in-only content

Open
  • Is something wrong?
#2,123 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@efegokdemir is already working on this.

Since Oct 7, 2026.

  • #2124 by @efegokdemir — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
cli

Research direction

Start from cmd/nvidia-ctk/runtime/configure/configure.go (validateFlags and getOutputConfigPath), then pkg/config/engine/crio and pkg/config/engine/containerd where the destination is seeded with toml.NewEmpty() / ConfigWithDropIn, plus Options.Flush in the installer which shares the fallback. Decide between rejecting the empty value in validateFlags or using the source config as destination when no drop-in is in play. Done means an empty --drop-in-config either errors or preserves existing runtimes for cri-o and emits the nvidia runtime for containerd. Note an open PR (#2124) already targets this issue, so coordinate before starting.

Written by the indexing model from the issue text.

Description

Describe the bug

An empty --drop-in-config writes drop-in-only content over the top-level engine config. For cri-o that means every runtime already in crio.conf is replaced by the nvidia one; for containerd it means the nvidia runtime does not end up anywhere, and the config gains an imports glob pointing at its own directory. Both report success and exit 0.

The empty value is not a nonsense input, it is the value the code uses for docker, and validateFlags does not reject it for the other engines. docker.New returns a plain in-place config, so nothing goes wrong there. crio.New seeds its destination with toml.NewEmpty() and containerd.New seeds it with only the CRI plugin section wrapped in ConfigWithDropIn, so for those two the destination is always a drop-in tree, while getOutputConfigPath falls back to the top-level path when the drop-in path is empty.

The installer has the same fallback in Options.Flush, and RUNTIME_DROP_IN_CONFIG="" reaches it, since the flag's value source is os.LookupEnv, which reports an empty variable as set. That is the obvious way to turn drop-ins off from the GPU Operator.

To Reproduce

cri-o. Starting from a config with another runtime in it:

$ cat crio.conf
[crio]
  [crio.runtime]
    default_runtime = "crun"
    [crio.runtime.runtimes]
      [crio.runtime.runtimes.crun]
        runtime_path = "/usr/bin/crun"
        runtime_type = "oci"

$ nvidia-ctk runtime configure --runtime=crio --config=$PWD/crio.conf --drop-in-config=
INFO Wrote updated config to /tmp/dropin/crio.conf
INFO It is recommended that crio daemon be restarted.

$ cat crio.conf
[crio]

  [crio.runtime]

    [crio.runtime.runtimes]

      [crio.runtime.runtimes.nvidia]
        runtime_path = "/usr/bin/nvidia-container-runtime"
        runtime_type = "oci"

default_runtime and the crun definition are gone, so cri-o comes up with no default runtime.

containerd, starting from a v2 config with runc and a cni section:

$ nvidia-ctk runtime configure --runtime=containerd --config=$PWD/config.toml --drop-in-config=
INFO Wrote updated config to /tmp/dropin/config.toml

$ head -2 config.toml
imports = ["/tmp/dropin/*.toml"]
version = 2

$ grep -c runtimes.nvidia config.toml
0

The original content survives here, but the nvidia runtime is not in the file at all, so --runtime nvidia and RuntimeClass nvidia fail. ConfigWithDropIn.Save writes the drop-in to the top-level path and then overwrites that same path with the top-level tree, which is also where the self-referential imports glob comes from.

With the default sentinel --drop-in-config both runtimes behave correctly, which is what isolates the empty value as the trigger.

Expected behavior

Either the empty value is rejected for containerd and cri-o, the way the docker combination is already rejected in validateFlags, or the engines honour it by using the source config as the destination when no drop-in is in play. The second is the better of the two, since Options.Flush in the installer has the identical fallback and would otherwise stay broken.

Environment

  • nvidia-container-toolkit version: built from main at a672e378, and the paths are unchanged on the current tip
  • Host OS: this was reproduced with nvidia-ctk built and run locally, not on a GPU node; the code path is the TOML config handling in cmd/nvidia-ctk/runtime/configure/configure.go and pkg/config/engine/{crio,containerd}, which does not touch the driver
  • CPU Architecture: arm64

Happy to send a PR for whichever of the two shapes you prefer.

Dominant language
Go
Stars
4.6k
Forks
609
Avg merge
3d 12h
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/nvidia-container-toolkit

All issues in NVIDIA/nvidia-container-toolkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.