[Bug]: containerd cleanup removes imports for unrelated drop-ins

Open
  • Is something wrong?
#2,134 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@git-jxj is already working on this.

Since Oct 11, 2026.

  • #2135 by @git-jxj — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
backend, devops

Research direction

Start with pkg/config/engine/containerd/config_drop_in.go, especially topLevelConfig.removeImports and removeVersion, and trace how ConfigWithDropIn.Save handles cleanup through Options.Unconfigure. Reproduce with the isolated config and containerd config dump steps in the issue. Done means removing the NVIDIA drop-in preserves import patterns and settings for unrelated matching drop-ins, while cleanup still removes the top-level file when nothing else depends on it.

Written by the indexing model from the issue text.

Description

Describe the bug

Removing the NVIDIA containerd drop-in can also remove an import used by unrelated drop-ins. With a top-level config containing only version and a single imports = [".../conf.d/*.toml"], ConfigWithDropIn.Save removes that glob when its NVIDIA destination becomes empty. It does not check whether the glob still imports other files. The top-level config is then deleted as well.

An unrelated conf.d/10-custom.toml remains byte-for-byte unchanged on disk, but containerd config dump no longer loads its settings. For example, a custom debug level silently reverts to containerd's default. This affects the file-loaded cleanup path used by the installer's Options.Unconfigure, including a file config source. It does not require an empty --drop-in-config value.

To Reproduce

On main at a672e378ef91bea7dfe92c0ed274847fa1cb9236, use an isolated directory; no daemon restart is needed:

fixture=$(mktemp -d)
mkdir -p "$fixture/conf.d"
cat > "$fixture/config.toml" <<EOF
version = 3
imports = ["$fixture/conf.d/*.toml"]
EOF
cat > "$fixture/conf.d/10-custom.toml" <<'EOF'
[debug]
level = "debug"
EOF
cat > "$fixture/conf.d/99-nvidia.toml" <<'EOF'
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia]
runtime_type = "io.containerd.runc.v2"
EOF
containerd --config "$fixture/config.toml" config dump

The dump includes level = 'debug'. Run the existing file-loaded installer cleanup API from the toolkit checkout:

cat > "$fixture/cleanup.go" <<'EOF'
package main

import (
    "os"
    "path/filepath"

    "github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/container"
    "github.com/NVIDIA/nvidia-container-toolkit/pkg/config/engine/containerd"
)

func main() {
    root := os.Args[1]
    options := container.Options{
        TopLevelConfigPath: filepath.Join(root, "config.toml"),
        DropInConfig: filepath.Join(root, "conf.d", "99-nvidia.toml"),
        RuntimeName: "nvidia",
    }
    cfg, err := containerd.New(containerd.WithTopLevelConfigPath(options.TopLevelConfigPath))
    if err != nil { panic(err) }
    if err := options.Unconfigure(cfg); err != nil { panic(err) }
}
EOF
go run "$fixture/cleanup.go" "$fixture"
ls "$fixture/conf.d"
containerd --config "$fixture/config.toml" config dump

Cleanup succeeds. 99-nvidia.toml and config.toml are deleted. 10-custom.toml still exists with its original contents, but the second dump no longer includes the custom debug level.

I reproduced the same result with config version 2 and its io.containerd.grpc.v1.cri plugin name. Controls with only the NVIDIA drop-in clean up as expected; controls with multiple import patterns or another top-level setting retain their custom drop-in settings.

Expected behavior

Cleanup should remove NVIDIA runtime configuration while preserving import patterns that still load other drop-ins and preserving unrelated settings. Removing the top-level file remains reasonable when the shared glob has no other matching drop-ins and the file contains no other settings.

The relevant simplification is in pkg/config/engine/containerd/config_drop_in.go, specifically topLevelConfig.removeImports and removeVersion. A focused fix can keep the import when other matching drop-ins remain, using the local drop-in directory when host/container paths differ.

Environment

  • NVIDIA Container Toolkit: source main, a672e378ef91bea7dfe92c0ed274847fa1cb9236
  • Container runtime: containerd 2.2.1, using its real config dump parser
  • Host OS: Ubuntu 22.04.5 LTS
  • Kernel: 5.15.0-191-generic
  • CPU architecture: x86_64
  • Driver / GPU: 595.91.07 / RTX 5060 Ti; this reproduction only reads isolated configuration files and does not exercise GPU execution or restart a containerd daemon
Dominant language
Go
Stars
4.6k
Forks
609
Avg merge
3d 18h
Merged PRs (30d)
37

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/nvidia-container-toolkit

All issues in NVIDIA/nvidia-container-toolkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.