No check for CPU quota or thermal throttling: a throttled run looks statistically stable

Open
  • Is something wrong?
#274 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
linux, python
Domain
performance, testing-qa

Research direction

Read format_checks() in _cli.py, the PowerSupply operation, and get_cgroup_cpu_quota() from #265. The issue leaves open whether to add an OPERATIONS check, capture counters around runs, store them in metadata, or combine these approaches; first agree on that design. Done means pyperf check can identify a benchmark run affected by Linux CPU quota or thermal throttling, while distinguishing unavailable thermal checks from zero throttling.

Written by the indexing model from the issue text.

Description

pyperf system show registers ten checks and pyperf check warns on four statistical conditions, but nothing in either looks at whether the benchmark was throttled while it ran. I think that is worth adding, and I would rather agree the shape with you than send a PR you did not want.

The statistical checks cannot substitute for this

format_checks() in _cli.py warns when the standard deviation reaches 10% of the mean, when there are not enough samples for a stable result, when the minimum or maximum is 50% off the mean, or when the shortest raw value is under 1 ms.

Sustained throttling trips none of them. A CPU quota that binds for the whole run moves the mean and leaves the relative spread where it was, so the result comes back looking stable, and it is simply wrong. The advice offered on an unstable result is to rerun with more runs and values, and to run pyperf system tune. Neither helps here, because the machine was not jittery, it was capped.

This is the case I keep running into on CI runners and in containers, where a quota is normal and invisible to the benchmark.

The precedent is already in the tree

PowerSupply reads /sys/class/power_supply/*/online, reports "the power cable is unplugged" and advises "The power cable must be plugged". That is the same kind of fact: an environmental state that invalidates a measurement rather than a tunable, surfaced by system show as advice. Throttling differs only in that nothing reads it yet.

What there is to read, on Linux

CPU quota throttling. cgroup v2 cpu.stat carries throttled_usec and nr_throttled. A delta across the run is not a risk of an effect, it is the effect. Two details that bite: a limit set on an ancestor cgroup also binds, which is why get_cgroup_cpu_quota() from #265 walks the hierarchy for the smallest limit, and the counters need the same treatment, because the process is usually not in the root cgroup and the root's counters stay at zero while a descendant is throttled hard.

Thermal throttling. /sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count and package_throttle_count on x86. These are absent on most Arm, where a Raspberry Pi needs vcgencmd get_throttled instead, so "could not be checked" has to be a distinct outcome from "zero", or the check quietly passes on the machines where thermal throttling is most likely.

Two design questions

  1. Does this belong in OPERATIONS next to PowerSupply, or does it need sampling around the run? system show can report that a quota is set and currently binding, which is useful on its own, but it cannot produce the delta, and the delta is the part that says this run was spoiled. My guess is both: an operation that reports a binding quota, and counters captured around the benchmark.

  2. #265 put the quota in metadata. Would you want the throttle counters in metadata too, so an already saved result can be judged after the fact, or only as a run-time warning?

I am happy to write whichever you prefer, or to drop it if you would rather pyperf stayed out of this. I can also add a demonstration here, a pyperf timeit under systemd-run --scope -p CPUQuota=... showing pyperf check reporting no warning at all, if that would be more convincing than the argument.

For completeness, since it comes up in search: #243 asks for system {tune,show} on macOS and mentions thermal pressure as one of several macOS variance sources. That is a different request and this one is Linux-only, so I have kept it separate rather than adding to that thread.

Dominant language
Python
Stars
967
Forks
112
Avg merge
5d 8h
Merged PRs (30d)
9

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from psf/pyperf

All issues in psf/pyperf

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.