bug: clistat only checks the nearest limit, but every parent's limit applies (cgroup v2)
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go, linux
- Domain
- operating-systems
Research direction
Start by tracing the memory and CPU limit checks that read cgroup data, then compare each ancestor's usage with its own limit as described in the issue. The reproduction uses cgroup v2 and a static probe calling ContainerMemory(PrefixMebi); run that scenario to check whether the reported limit matches the one exhausted first. Done means the display reflects the ancestor with the least room left; the issue leaves open whether to report only that limit or all limits.
Written by the indexing model from the issue text.
Description
Summary
For memory and CPU, clistat goes up from the process's cgroup and stops at the first limit it finds. But the kernel checks the limit of every parent cgroup, not just the nearest one. Whichever parent runs out of room first is the one that matters. This means a process can be killed for using too much memory while clistat says it is only using a small part of its limit.
This can happen even when every child limit is smaller than its parent's. If other cgroups have used up most of the parent's memory, the child hits the parent's limit first.
#38 and #39 fix the case where only one cgroup has a limit. This issue is about the general case.
Why this is wrong
- When memory is used, the kernel adds it to the cgroup and to every parent above it. If any of them goes over its limit, the allocation fails and that parent's limit is the one hit (
page_counter_try_charge()inmm/page_counter.c, v6.17). memory.currentis the memory used by a cgroup and everything below it (cgroup v2 docs). This is the number the kernel compares againstmemory.max.- Child limits can add up to more than the parent's limit (cgroup v2 docs). So a child's limit alone does not tell you if there is room left.
- CPU limits work the same way. A limit applies to the cgroup and everything below it (cgroup v2 docs).
How to reproduce
We tested this with #39 applied. The container root has a 1 GiB limit. Cgroup A, where clistat runs, has a 600 MiB limit. Cgroup B has no limit and uses 850 MiB:
docker run --rm --privileged --memory 1g -v "$PWD/probe:/probe:ro" alpine sh -c '
cd /sys/fs/cgroup && mkdir leaf A B
echo $$ > leaf/cgroup.procs && echo +memory > cgroup.subtree_control
echo 629145600 > A/memory.max
mkdir /t && mount -t tmpfs -o size=2g tmpfs /t
sh -c "echo \$\$ > B/cgroup.procs; dd if=/dev/zero of=/t/f bs=1M count=850"
sh -c "echo \$\$ > A/cgroup.procs; exec /probe"
sh -c "echo \$\$ > A/cgroup.procs; exec dd if=/dev/zero of=/t/g bs=1M count=300"'
probe is a small static binary that prints ContainerMemory(PrefixMebi).
| cgroup | Memory used | Limit |
|---|---|---|
| Container root | 895 MB | 1 GiB |
A |
0 | 600 MiB |
B |
893 MB | none |
- clistat in
Ashows9.71/600 MiB (2%). - We then used 300 MiB in
A, which is half of its own limit. The process was killed for using too much memory. The root'smemory.eventsshowsoom 1, butA's ownoomcount is still0. The root's limit was the one hit, notA's.
What should happen
clistat should show the limit that will actually be hit first. Here that is the root: about 895 MB of 1 GiB (83%), not 2%.
How to fix it
Go up from the process's cgroup to the top of the container. For each cgroup that has a limit, read usage from that same cgroup:
- Memory:
memory.current - inactive_file, compared withmemory.max. - CPU: the change in
cpu.statusage_usecover the sample time, compared withcpu.max.
Report the cgroup with the least room left (limit minus usage). This also covers #38, #39 and #41.
Open question: should clistat report only the limit that will be hit first, or every limit it finds?
Not covered by this issue
clistat cannot see limits set outside the container. For example, a Kubernetes pod-level limit is not visible from inside a container that has its own cgroup namespace. We should document this, not try to fix it here.
Generated by Coder Agents on behalf of @DanielleMaywood.
- Dominant language
- Go
- Stars
- 11
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from coder/clistat
-
bug: CPU usage misses other processes that share the same CPU limit (cgroup v2)Possibly taken @DanielleMaywood claimed this 2 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 67/100
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
-
Bug
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
feat: add directory-level disk usage statsMay be free again A pull request for this issue was closed without being merged. Openenhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
Similar issues
-
[flaky test] TestValidatorCommitterDropsEmptyStatusBatch: submits to the mock VC before its stream is registeredPossibly taken @liran-funaro claimed this today. Opencoordinator flaky-test race-condition testing
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
hyperledger/fabric-x-committer#876 ·
Maintainers usually reply within 1 day
-
kind/features
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
TencentBlueKing/bk-nodemgr#3804 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
cloudnative-pg/plugin-barman-cloud#1187 ·
Maintainers usually reply within 1 day
-
[deliverclient] Remove the unused common/deliverclient packagePossibly taken @liran-funaro claimed this today. Opencode-hygiene
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
hyperledger/fabric-x-common#236 ·
Maintainers usually reply within 1 day
-
consensus feature I2 S4 U2
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 1 day