Per-step resource limits are silently ignored in rootless mode, even with cgroup v2 delegated

Open
  • Is something wrong?
#7,287 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
docker, go, kubernetes, linux
Domain
build-system, infrastructure

Research direction

Start with ToRootless() in util/rootless/specconv/specconv_linux.go, which clears spec.Linux.Resources and CgroupsPath, and with the rootless paths in the runc and containerd executors that call it. Check testLinuxResources, which skips rootless sandboxes. Done means limits are applied when the cgroup is delegated and writable, or a warning is emitted when they cannot be. The body already links two stacked PRs (#7288, #7289), so check their state before starting.

Written by the indexing model from the issue text.

Description

area/rootless kind/enhancement
Contributing guidelines and issue reporting guide
Well-formed report checklist
  • I have found a bug that the documentation does not mention anything about my problem
  • I have found a bug that there are no open or closed issues that are related to my problem
  • I have provided version/information about my environment and done my best to provide a reproducer
Description of bug

[!NOTE]
This issue and the two linked PRs were written with Claude Code. I mainly want to put a spotlight on this bug and would love to see a fix released, so resources works in rootless setups where cgroup v2 is available. I found it after my laptop froze during a build that had resource limits set.

Bug description

Per-step resource limits (docker buildx build --resource ..., Bake resources, LLB MemoryLimit and friends, added in #6569) have no effect in rootless BuildKit, and nothing tells the user.

ToRootless() in util/rootless/specconv/specconv_linux.go always sets spec.Linux.Resources = nil and spec.Linux.CgroupsPath = "". Both the runc and containerd executors call it in rootless mode. The comment there refers to a cgroup v1 error (mkdir /sys/fs/cgroup/cpuset/buildkit: permission denied). On cgroup v2, a rootless process can manage a cgroup that systemd delegated to it, and rootless runc applies limits there fine. Rootless Docker does exactly this for docker run --memory.

The daemon still advertises exec.meta.linux.resources, so clients send the limits and the build succeeds without any error or warning. testLinuxResources skips rootless sandboxes, so this is not covered by tests.

Clearing CgroupsPath also turns off resource usage sampling and OOM detection for rootless steps.

Observed: a step with --resource memory=256m --resource memory-swap=256m uses 600MB and succeeds.
Expected: the step is OOM killed like docker run --memory 256m on the same host. If limits can't be applied, the user should at least get a warning.

Reproduction

On a host with cgroup v2 and rootless Docker (the docker driver, so the daemon's embedded BuildKit):

FROM alpine
# tail keeps its whole input in memory when there is no newline, so this
# step needs about 600MB.
RUN head -c 600m /dev/zero | tail > /dev/null && echo "step used 600MB and was not killed"

$ docker buildx build --builder rootless --no-cache --progress=plain \
    --resource memory=256m --resource memory-swap=256m .
#5 [2/2] RUN head -c 600m /dev/zero | tail > /dev/null && echo "step used 600MB and was not killed"
#5 3.043 step used 600MB and was not killed
#5 DONE 3.2s

$ docker run --rm --memory 256m --memory-swap 256m alpine sh -c 'head -c 600m /dev/zero | tail > /dev/null'
$ echo $?
137

The same happens with standalone rootless buildkitd (OCI worker), in a systemd user service with Delegate=yes, and in a Kubernetes pod running moby/buildkit:rootless with --oci-worker-no-process-sandbox (v0.33.1). Inside the step, /sys/fs/cgroup does not exist.

Possible fix

Two PRs, stacked:

  • #7288 logs a warning and adds a build warning when a rootless executor ignores limits. No behaviour change otherwise.
  • #7289 applies the limits in the rootless OCI worker when the cgroup buildkitd runs in is cgroup v2, writable and has the cpu and memory controllers. Otherwise it falls back to today's behaviour plus the warning.

Neither PR covers the rest:

  • The containerd executor still drops the limits in rootless mode. Docker Engine's embedded BuildKit uses this executor when the containerd image store is on (worker label executor: containerd in the output below).
  • Docker Engine would also need to pass a delegated cgroup to runcexecutor.New in daemon/internal/builder-next for its runc path.
  • In Kubernetes, unprivileged pods get a read-only /sys/fs/cgroup, so detection fails there and limits stay off (KEP-5714).

Version information

$ docker buildx version
github.com/docker/buildx v0.38.0 0dbe87f36ed472fe36b6796c3f70b4210a731fc5

$ docker buildx inspect rootless
Name:          rootless
Driver:        docker
Nodes:
Name:             rootless
Endpoint:         rootless
Status:           running
BuildKit version: v0.34.0
Platforms:        linux/amd64, linux/amd64/v2, linux/amd64/v3
Labels:
 org.mobyproject.buildkit.worker.executor:             containerd
 org.mobyproject.buildkit.worker.network:              host
 org.mobyproject.buildkit.worker.snapshotter:          overlayfs

$ docker version --format '{{.Server.Version}}'
29.9.1

$ docker info --format 'cgroup v{{.CgroupVersion}} driver={{.CgroupDriver}} kernel={{.KernelVersion}} os={{.OperatingSystem}} {{.SecurityOptions}}'
cgroup v2 driver=systemd kernel=7.0.0-38-generic os=Ubuntu 24.04.5 LTS [name=seccomp,profile=builtin name=rootless name=cgroupns]

$ cat /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/cgroup.subtree_control
cpu memory pids
Dominant language
Go
Stars
10.4k
Forks
1.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from moby/buildkit

All issues in moby/buildkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.