cpufreq collector hangs forever on arm64 with cppc_cpufreq, leaking a goroutine and an fd per scrape
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 48/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- go, linux
Línea de trabajo
Comienza con procfs/sysfs/system_cpu.go, especialmente con SystemCpufreq y parseCpufreqCpuinfo, y luego reproduce el collector de cpufreq en arm64 con cppc_cpufreq, observando las goroutines, los descriptores de archivo y la finalización del scrape. Se considera terminado cuando un archivo de policy ilegible o bloqueado no puede dejar los scrapes bloqueados indefinidamente ni provocar fugas de recursos; verifica el comportamiento ante fallos y las métricas del collector.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
What did you do?
Ran the cpufreq collector on two arm64 machines (NVIDIA GB10 / DGX Spark) whose scaling driver is cppc_cpufreq, with 20 CPUs and 20 cpufreq policies.
The collector hangs permanently. It does not merely get slow: a goroutine blocked at process start was still blocked 1229 minutes (20.5 h) later, and every subsequent scrape leaks another goroutine and another open file descriptor.
Observed via Grafana Alloy's embedded prometheus.exporter.unix (Alloy v1.18.1, arm64, go1.26.5, github.com/prometheus/procfs v0.21.0). The stack is entirely inside node_exporter/procfs, but to be upfront: I have not tested standalone node_exporter on this hardware, so I cannot claim the standalone binary reproduces it — I would expect it to, since nothing in the path is Alloy-specific.
What did you expect to see?
The scrape completing, or at worst the collector failing and reporting node_scrape_collector_success{collector="cpufreq"} 0.
What did you see instead?
Roughly 40–60% of scrapes never return. From /debug/pprof/goroutine?debug=2:
goroutine 185317 [IO wait, 1229 minutes]:
internal/poll.runtime_pollWait(0xeb287c0f7e00, 0x72)
/usr/local/go/src/runtime/netpoll.go:351 +0xa0
internal/poll.(*pollDesc).wait(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*FD).Read(...)
/usr/local/go/src/internal/poll/fd_unix.go:165 +0x234
os.(*File).Read(...)
/usr/local/go/src/os/file.go:144 +0x68
os.ReadFile(...)
/usr/local/go/src/os/file.go:871 +0xb4
github.com/prometheus/procfs/internal/util.ReadUintFromFile(...)
/go/pkg/mod/github.com/prometheus/[email protected]/internal/util/parse.go:85 +0x20
github.com/prometheus/procfs/sysfs.parseCpufreqCpuinfo(...)
/go/pkg/mod/github.com/prometheus/[email protected]/sysfs/system_cpu.go:284 +0x154
github.com/prometheus/procfs/sysfs.FS.SystemCpufreq.func1()
/go/pkg/mod/github.com/prometheus/[email protected]/sysfs/system_cpu.go:249 +0x40
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93 +0x4c
It is stuck in runtime_pollWait — the descriptor was registered with the netpoller and the kernel never marks it readable, so os.ReadFile never returns. Because SystemCpufreq waits on its errgroup, the whole NodeCollector.Collect blocks with it and the HTTP scrape times out.
The concurrency appears to be what triggers it. SystemCpufreq reads every policy in parallel (system_cpu.go:249), deliberately, per the comment about the kernel's intentional 50 ms per-CPU delay. Reading the same files sequentially from a shell is fast and never hangs:
$ for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq; do timeout 3 cat $c; done
# all 20 return in < 200 ms total, no process ever enters D state
So a manual check "proves" the machine is healthy, which made this quite misleading to track down.
Impact
It is not only missing metrics — it is an unbounded resource leak, since each hung scrape leaves a goroutine and an fd behind forever. After ~21 h on a 120 s scrape interval:
| host A | host B | unaffected x86 host | |
|---|---|---|---|
| alloy goroutines | 2557 | 2128 | 313 |
open fds / pointing at cpufreq |
1083 / 740 | 820 / 543 | — |
| RSS | ~800 MB | ~800 MB | — |
| scrapes hanging (of 12) | 5 | 4 | 0 |
Restarting the process reclaims it (goroutines 2612 → 273, cpufreq fds 760 → 0, RSS 800 → 386 MB), but the leak starts over immediately.
Workaround
Disabling the collector (--no-collector.cpufreq, or disable_collectors = ["cpufreq"] in Alloy) fixes it completely and instantly — 0 hung scrapes out of 12, max scrape time 30 s → 0.13 s.
Worth noting the trigger only became visible after a same-day upgrade of both the kernel (6.17.0-1021 → 6.17.0-1029-nvidia) and Alloy (1.17.x → 1.18.1) on these two hosts, so I cannot cleanly attribute it to either one; both hosts are the only ones in the fleet with both changes. What is clear is where it blocks. A 50 ms/CPU kernel delay cannot produce a 20-hour block, so something on the kernel/firmware side is failing to ever complete the read, and the parallel access is what exposes it. cppc_cpufreq reads go through the PCC firmware mailbox, which is a plausible place for concurrent access to serialize badly.
Even if the kernel is at fault, node_exporter arguably should not be able to leak unboundedly because of one unreadable sysfs file — a timeout or context on the per-policy reads would contain the damage.
System information
Linux 6.17.0-1029-nvidia aarch64 (Ubuntu 24.04, #29-Ubuntu SMP PREEMPT_DYNAMIC)
NVIDIA GB10 (DGX Spark class), 20 CPUs, 20 cpufreq policies
scaling_driver: cppc_cpufreq
Reproduced on two physically different machines from two different OEMs (Lenovo ThinkStation PGX and ASUS Ascent GX10), so it is not a single-vendor firmware quirk.
node_exporter version
Embedded in Grafana Alloy v1.18.1 (revision 6012ec4, goarch=arm64, goversion=go1.26.5), which vendors github.com/prometheus/procfs v0.21.0.
- Lenguaje dominante
- Go
- Estrellas
- 13.8k
- Forks
- 2.7k
- Merge medio
- 1 d 4 h
- PR fusionados (30 d)
- 7
Preparar el entorno
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de prometheus/node_exporter
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
prometheus/node_exporter#3830 · 3 comentarios · 1 reacción ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
prometheus/node_exporter#3823 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
prometheus/node_exporter#3817 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
prometheus/node_exporter#3761 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 68/100
prometheus/node_exporter#1767 ·
Los mantenedores suelen responder en 1 día
Todos los issues de prometheus/node_exporter
Issues similares
-
area: global bug dx priority: low
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
Los mantenedores suelen responder en 1 día
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
grafana/mcp-grafana#1267 ·
Los mantenedores suelen responder en 1 día
-
automation models
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
coverage-gap good-first-pattern help wanted
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
GoogleCloudPlatform/k8s-aibom#114 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
txn2/mcp-data-platform#1984 ·
Los mantenedores suelen responder en 1 día