ConnectX Power through DCGM

Open
  • Is something wrong?
#322 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp
Domain
observability

Research direction

Start by reading DCGM’s field reference and the existing ConnectX monitoring path to see whether a NIC power field already exists, then trace how exporter fields are defined and exposed. Check DOCA Telemetry PHY capability per device and prototype with doca_telemetry_phy --pci-addr <BDF> --get-system-temp-and-power-info to confirm whether the reported watts are board input power. Done means DCGM exposes a documented ConnectX power sample with units, capability gating, and a test or exporter example; the issue names no source files, so ask maintainers for the intended field/API entry point.

Written by the indexing model from the issue text.

Description

Human

Currently DCGM exporter provides gpu power, grace power, but for NVIDIA ConnectX NIC power, i would need to use an publicly avialable doca_telemetry_phy CLI. It is an poor user experience. In comparsion, AMD DCGM-exporter equivalent is adding AMD NIC power as an first class primtive to their exporter https://github.com/ROCm/device-metrics-exporter/issues/714

Can this be implemented into dcgm such that we can get the sampling of ConnectX NIC power? Can the ETA be provided for this feature?

+viz @Ankur-singh

AI deep research slop

Yes: NVIDIA documents a ConnectX power-sensor API and CLI sample through DOCA Telemetry PHY. The documented output is watts plus a sensor name, but support must be checked per device, and the documentation does not establish that every card reports total board input power including optics ([NVIDIA guide](https://networking-docs.nvidia.com/doca/archive/3-5-0/doca-telemetry-phy)). DCGM has ConnectX monitoring, but no documented ConnectX power or energy field ([DCGM field reference](https://docs.nvidia.com/datacenter/dcgm/latest/dcgm-api/dcgm-api-field-ids.html)).The NVIDIA command to tryDOCA 3.5 includes the doca_telemetry_phy sample with --get-system-temp-and-power-info; this option is absent from the inspected 3.4 guide ([3.5 documentation](https://networking-docs.nvidia.com/doca/archive/3-5-0/doca-telemetry-phy), [3.4 documentation](https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-telemetry-phy)). After building the sample, substitute the actual NIC BDF:sudo /tmp/doca-telemetry-phy-build/doca_telemetry_phy \
  --pci-addr 0000:03:00.0 \
  --get-system-temp-and-power-info
The documented interface provides:
power: sensor reading in watts.
power_sensor_name: identifies the power sensor.
Temperature and historical maximum temperature ([API structure](https://docs.nvidia.com/doca/api/3-5-0/doca-libraries-api/annotated.html)).
NVIDIA lists fwctl and firmware baselines of 28.43.1000 for CX7, 32.43.1000 for BF3, and 40.43.1000 for CX8; these are library prerequisites, not a guarantee of power-sensor support, which has a separate capability check ([prerequisites and capability API](https://networking-docs.nvidia.com/doca/archive/3-5-0/doca-telemetry-phy)). The full build commands are in the attached research.Important traps
MPEIN.pci_power is slot-power capability, not measured consumption. Linux uses it to print “PCIe slot advertised sufficient power,” so collecting that value as NIC watts would be wrong ([Linux implementation](https://github.com/torvalds/linux/blob/ac9c34d1e45a4c25174ced4fc0cfc33ff3ed08c7/drivers/net/ethernet/mellanox/mlx5/core/events.c)).
mlxlink -m does not establish board electrical power. Its documented readings include RX/TX optical power in dBm, laser bias current, and maximum module power class ([MFT documentation](https://docs.nvidia.com/networking/display/mftv4320/mlxlink+utility)).
DOCA library support does not mean an exporter already exposes it. The DTS 3.5 PHY provider is documented as transceiver-temperature-only, while its separate BlueField bf_ptm provider has power counters ([DTS guide](https://networking-docs.nvidia.com/doca/archive/3-5-0/doca-telemetry-service-guide)).
Firmware provenance matters. CX8 firmware history includes a fix adding a “fixed estimated power value for Hvdd”; that does not prove this API includes the estimate, but it warrants asking NVIDIA which rails are measured versus estimated ([firmware history](https://networking-docs.nvidia.com/connectx8fwrn/40.49.1014/bug-fixes-history)).
Dominant language
C++
Stars
813
Forks
115
Avg merge
6d 10h
Merged PRs (30d)
1

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/DCGM

All issues in NVIDIA/DCGM

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.