No check for CPU quota or thermal throttling: a throttled run looks statistically stable

オープン
  • 何か問題がありますか？
#274 コメント 5 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
linux, python
領域
performance, testing-qa

調査の方向性

Read format_checks() in _cli.py, the PowerSupply operation, and get_cgroup_cpu_quota() from #265. The issue leaves open whether to add an OPERATIONS check, capture counters around runs, store them in metadata, or combine these approaches; first agree on that design. Done means pyperf check can identify a benchmark run affected by Linux CPU quota or thermal throttling, while distinguishing unavailable thermal checks from zero throttling.

索引モデルが issue の本文から書いたものです。

説明

pyperf system show registers ten checks and pyperf check warns on four statistical conditions, but nothing in either looks at whether the benchmark was throttled while it ran. I think that is worth adding, and I would rather agree the shape with you than send a PR you did not want.

The statistical checks cannot substitute for this

format_checks() in _cli.py warns when the standard deviation reaches 10% of the mean, when there are not enough samples for a stable result, when the minimum or maximum is 50% off the mean, or when the shortest raw value is under 1 ms.

Sustained throttling trips none of them. A CPU quota that binds for the whole run moves the mean and leaves the relative spread where it was, so the result comes back looking stable, and it is simply wrong. The advice offered on an unstable result is to rerun with more runs and values, and to run pyperf system tune. Neither helps here, because the machine was not jittery, it was capped.

This is the case I keep running into on CI runners and in containers, where a quota is normal and invisible to the benchmark.

The precedent is already in the tree

PowerSupply reads /sys/class/power_supply/*/online, reports "the power cable is unplugged" and advises "The power cable must be plugged". That is the same kind of fact: an environmental state that invalidates a measurement rather than a tunable, surfaced by system show as advice. Throttling differs only in that nothing reads it yet.

What there is to read, on Linux

CPU quota throttling. cgroup v2 cpu.stat carries throttled_usec and nr_throttled. A delta across the run is not a risk of an effect, it is the effect. Two details that bite: a limit set on an ancestor cgroup also binds, which is why get_cgroup_cpu_quota() from #265 walks the hierarchy for the smallest limit, and the counters need the same treatment, because the process is usually not in the root cgroup and the root's counters stay at zero while a descendant is throttled hard.

Thermal throttling. /sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count and package_throttle_count on x86. These are absent on most Arm, where a Raspberry Pi needs vcgencmd get_throttled instead, so "could not be checked" has to be a distinct outcome from "zero", or the check quietly passes on the machines where thermal throttling is most likely.

Two design questions

  1. Does this belong in OPERATIONS next to PowerSupply, or does it need sampling around the run? system show can report that a quota is set and currently binding, which is useful on its own, but it cannot produce the delta, and the delta is the part that says this run was spoiled. My guess is both: an operation that reports a binding quota, and counters captured around the benchmark.

  2. #265 put the quota in metadata. Would you want the throttle counters in metadata too, so an already saved result can be judged after the fact, or only as a run-time warning?

I am happy to write whichever you prefer, or to drop it if you would rather pyperf stayed out of this. I can also add a demonstration here, a pyperf timeit under systemd-run --scope -p CPUQuota=... showing pyperf check reporting no warning at all, if that would be more convincing than the argument.

For completeness, since it comes up in search: #243 asks for system {tune,show} on macOS and mentions thermal pressure as one of several macOS variance sources. That is a different request and this one is Linux-only, so I have kept it separate rather than adding to that thread.

主要言語
Python
スター
967
フォーク
112
平均マージ
5日 8時間
マージ済み PR（30日）
9

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

psf/pyperf のほかの issue

psf/pyperf の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。