[CommunityToolkit.Aspire.Hosting.K3s] Pod sandbox creation fails with cgroup.procs ENOENT on Windows 11, WSL 3.0.5 and Docker Desktop 4.88

Aperta
#2,436 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp, docker, kubernetes

Direzione di ricerca

Start with src/CommunityToolkit.Aspire.Hosting.K3s/K3sBuilderExtensions.cs, especially the container runtime arguments and K3sInitEntrypointScript described in the issue. Reproduce the failure with the listed Windows, WSL, Docker Desktop, and k3s versions, then inspect the container logs and cgroup paths. Done means pod sandboxes start successfully and CoreDNS and local-path-provisioner reach Running without cgroup errors.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

Using Windows 11, WSL 3.0.5 and Docker Desktop 4.88, the k3s cluster created by CommunityToolkit.Aspire.Hosting.K3s no longer runs reliably.

The Kubernetes node reports Ready, but pod sandbox creation repeatedly fails. CoreDNS, local-path-provisioner, the SPIRE CSI driver and other workloads remain in ContainerCreating, Pending or init states.

A representative failure is:

Failed to create sandbox for pod
failed to create containerd task
failed to create shim task
OCI runtime create failed
runc create failed
unable to apply cgroup configuration
failed to write 49865:
write /sys/fs/cgroup/kubepods/burstable/podbd669241-1f1f-4bf9-a4d9-7d7aea8c4329/5f73d605c09ef05dc6a7ca044fc2a22fa087a5f864dc14301ddc8f0e5b8ac857/cgroup.procs:
no such file or directory

The same failure occurs for besteffort pods, including local-path-provisioner and the SPIRE CSI driver.

kubelet also cannot move the k3s process into its own cgroup:

Failed to ensure state containerName="/k3s"
failed to move PID 7
(in "/docker/04a15f167a62e22903e6733541aec46384c8d1663402c4d19e29864e7e2d66fb")
to "/k3s":
failed to write 7:
write /sys/fs/cgroup/k3s/cgroup.procs:
no such file or directory

However, /sys/fs/cgroup/k3s/cgroup.procs exists and /sys/fs/cgroup/k3s/cgroup.type reports domain.

The application does not explicitly configure the cgroup namespace or mount /sys/fs/cgroup.

These settings are added by:


https://github.com/CommunityToolkit/Aspire/blob/main/src/CommunityToolkit.Aspire.Hosting.K3s/K3sBuilderExtensions.cs

.WithContainerRuntimeArgs("--privileged")
.WithContainerRuntimeArgs("--init")
.WithContainerRuntimeArgs("--userns=host")
.WithContainerRuntimeArgs("--cgroupns=host")
.WithContainerRuntimeArgs("--volume=/sys/fs/cgroup:/sys/fs/cgroup:rw")
.WithContainerRuntimeArgs("--tmpfs=/run", "--tmpfs=/var/run")

The same file defines K3sInitEntrypointScript, which moves processes from the visible root cgroup into /sys/fs/cgroup/init and enables the available cgroup v2 controllers.

This is a hypothesis rather than a confirmed root cause, but the problem may involve combining:

--cgroupns=host
--volume=/sys/fs/cgroup:/sys/fs/cgroup:rw

with the current WSL and Docker Desktop cgroup v2 implementation. Inside the k3s container, the k3s process is reported under /docker/<container-id>, while the explicitly mounted cgroup filesystem appears to expose a different WSL/Docker Desktop hierarchy. This could produce an inconsistent cgroup namespace and filesystem view, causing writes to an existing cgroup.procs file to return ENOENT.

The entrypoint script also suppresses cgroup setup failures using 2>/dev/null || true, potentially allowing k3s to start and report the node as Ready despite unsuccessful or incomplete cgroup initialisation.

Environment:

Windows edition: Windows Enterprise
Windows version: 25H2
Windows build: 26200.9711

WSL version: 3.0.5.0
WSL kernel: 6.18.58.1-1
WSLg version: 1.0.79

Ubuntu: 24.04.5 LTS
systemd: enabled

Docker Desktop: 4.88.0
Docker Engine: 29.7.2
Docker containerd: 2.3.3
Docker runc: 1.4.3
Docker cgroup version: 2
Docker cgroup driver: cgroupfs

k3s image: rancher/k3s:v1.36.0-k3s1
Kubernetes: v1.36.0+k3s1
k3s containerd: 2.2.3-k3s1
Regression

Possible. The setup worked before the managed WSL and Docker Desktop upgrade. The last confirmed working CommunityToolkit.Aspire.Hosting.K3s version was also 13.5.0; the toolkit package was not changed when the problem appeared. The environmental change was the WSL and Docker Desktop upgrade.

Steps to reproduce
1. On Windows 11 Enterprise 25H2, install or upgrade to WSL 3.0.5.0 with kernel 6.18.58.1 and Docker Desktop 4.88.0.
2. Use Ubuntu 24.04.5 LTS under WSL with systemd enabled.
3. Create a .NET 10 Aspire AppHost referencing `CommunityToolkit.Aspire.Hosting.K3s` version `13.5.0`.
4. Add a k3s cluster:


var k3sCluster = builder
    .AddK3sCluster("kubernetes", apiServerPort: 16443)
    .WithDisabledComponent("traefik");


5. Start the Aspire AppHost.
6. Identify the generated k3s container:


docker ps --format 'Name={{.Names}} Image={{.Image}} Status={{.Status}}'


7. Confirm the Kubernetes node reports `Ready`:


docker exec <k3s-container> kubectl \
  --kubeconfig=/etc/rancher/k3s/k3s.yaml \
  get nodes -o wide


8. Inspect the pods:


docker exec <k3s-container> kubectl \
  --kubeconfig=/etc/rancher/k3s/k3s.yaml \
  get pods -A -o wide


9. Observe CoreDNS and local-path-provisioner remaining in `ContainerCreating`.
10. Inspect the k3s container logs:


docker logs --tail 500 <k3s-container>


11. Observe repeated `cgroup.procs: no such file or directory` failures from runc and kubelet.
Expected behavior

The k3s cluster should create pod sandboxes successfully on the supported Windows 11, WSL and Docker Desktop environment.

The Kubernetes node should report Ready, and core components such as CoreDNS and local-path-provisioner should reach Running without cgroup-related runc failures.

Screenshots

No response

IDE and version

VS Code

IDE version

No response

Nuget packages
<Project Sdk="Aspire.AppHost.Sdk/13.6.5">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <IsAspireHost>true</IsAspireHost>
    <AspireUseCliBundle>true</AspireUseCliBundle>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="CommunityToolkit.Aspire.Hosting.K3s" Version="13.5.0" />
  </ItemGroup>
</Project>
Additional context

aspire-k3s-cgroup-diagnostics.txt

Help us help you

Possible - can work with the author(s) of this extension 😎

Lingua principale
C#
Stelle
634
Fork
206
Merge medio
1g 10h
PR unite (30g)
104

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di CommunityToolkit/Aspire

Tutte le issue di CommunityToolkit/Aspire

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.