bug: clistat only checks the nearest limit, but every parent's limit applies (cgroup v2)
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 48/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- go, linux
- Ambito
- operating-systems
Direzione di ricerca
Start by tracing the memory and CPU limit checks that read cgroup data, then compare each ancestor's usage with its own limit as described in the issue. The reproduction uses cgroup v2 and a static probe calling ContainerMemory(PrefixMebi); run that scenario to check whether the reported limit matches the one exhausted first. Done means the display reflects the ancestor with the least room left; the issue leaves open whether to report only that limit or all limits.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
For memory and CPU, clistat goes up from the process's cgroup and stops at the first limit it finds. But the kernel checks the limit of every parent cgroup, not just the nearest one. Whichever parent runs out of room first is the one that matters. This means a process can be killed for using too much memory while clistat says it is only using a small part of its limit.
This can happen even when every child limit is smaller than its parent's. If other cgroups have used up most of the parent's memory, the child hits the parent's limit first.
#38 and #39 fix the case where only one cgroup has a limit. This issue is about the general case.
Why this is wrong
- When memory is used, the kernel adds it to the cgroup and to every parent above it. If any of them goes over its limit, the allocation fails and that parent's limit is the one hit (
page_counter_try_charge()inmm/page_counter.c, v6.17). memory.currentis the memory used by a cgroup and everything below it (cgroup v2 docs). This is the number the kernel compares againstmemory.max.- Child limits can add up to more than the parent's limit (cgroup v2 docs). So a child's limit alone does not tell you if there is room left.
- CPU limits work the same way. A limit applies to the cgroup and everything below it (cgroup v2 docs).
How to reproduce
We tested this with #39 applied. The container root has a 1 GiB limit. Cgroup A, where clistat runs, has a 600 MiB limit. Cgroup B has no limit and uses 850 MiB:
docker run --rm --privileged --memory 1g -v "$PWD/probe:/probe:ro" alpine sh -c '
cd /sys/fs/cgroup && mkdir leaf A B
echo $$ > leaf/cgroup.procs && echo +memory > cgroup.subtree_control
echo 629145600 > A/memory.max
mkdir /t && mount -t tmpfs -o size=2g tmpfs /t
sh -c "echo \$\$ > B/cgroup.procs; dd if=/dev/zero of=/t/f bs=1M count=850"
sh -c "echo \$\$ > A/cgroup.procs; exec /probe"
sh -c "echo \$\$ > A/cgroup.procs; exec dd if=/dev/zero of=/t/g bs=1M count=300"'
probe is a small static binary that prints ContainerMemory(PrefixMebi).
| cgroup | Memory used | Limit |
|---|---|---|
| Container root | 895 MB | 1 GiB |
A |
0 | 600 MiB |
B |
893 MB | none |
- clistat in
Ashows9.71/600 MiB (2%). - We then used 300 MiB in
A, which is half of its own limit. The process was killed for using too much memory. The root'smemory.eventsshowsoom 1, butA's ownoomcount is still0. The root's limit was the one hit, notA's.
What should happen
clistat should show the limit that will actually be hit first. Here that is the root: about 895 MB of 1 GiB (83%), not 2%.
How to fix it
Go up from the process's cgroup to the top of the container. For each cgroup that has a limit, read usage from that same cgroup:
- Memory:
memory.current - inactive_file, compared withmemory.max. - CPU: the change in
cpu.statusage_usecover the sample time, compared withcpu.max.
Report the cgroup with the least room left (limit minus usage). This also covers #38, #39 and #41.
Open question: should clistat report only the limit that will be hit first, or every limit it finds?
Not covered by this issue
clistat cannot see limits set outside the container. For example, a Kubernetes pod-level limit is not visible from inside a container that has its own cgroup namespace. We should document this, not try to fix it here.
Generated by Coder Agents on behalf of @DanielleMaywood.
- Lingua principale
- Go
- Stelle
- 11
- Fork
- 2
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di coder/clistat
-
bug: CPU usage misses other processes that share the same CPU limit (cgroup v2)Forse già presa @DanielleMaywood l’ha presa 2 giorni fa. Aperta
Difficoltà 3/5 1-2 giorni Idoneità per principianti 67/100
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 72/100
-
Bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
-
feat: add directory-level disk usage statsForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Apertaenhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 45/100
Tutte le issue di coder/clistat
Issue simili
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 76/100
I maintainer di solito rispondono entro 1 giorno
-
[flaky test] TestValidatorCommitterDropsEmptyStatusBatch: submits to the mock VC before its stream is registeredForse già presa @liran-funaro l’ha presa oggi. Apertacoordinator flaky-test race-condition testing
Difficoltà 2/5 1-3 ore Idoneità per principianti 83/100
hyperledger/fabric-x-committer#876 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
garethgeorge/backrest#1426 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 67/100
I maintainer di solito rispondono entro 1 giorno
-
enhancement nav-pilot
Difficoltà 2/5 Mezza giornata Idoneità per principianti 62/100
I maintainer di solito rispondono entro 1 giorno