rapidsai/cuml

[BUG] Label-permutation invariance appears inconsistent for `homogeneity_score`, `completeness_score`, `v_measure_score`, and `mutual_info_score` in `cuml.metrics`

Open

#7,199 opened on Sep 9, 2025

View on GitHub
 (3 comments) (0 reactions) (1 assignee)Python (644 forks)auto 404
buggood first issue

Repository metrics

Stars
 (5,223 stars)
PR merge metrics
 (PR metrics pending)

Description

Describe the bug The following cuml.metrics clustering metrics change when class/cluster label identities are permuted (e.g., flipping 0 ↔ 1 in the predicted labels), whereas they should be invariant to any permutation of label values:

  • homogeneity_score
  • completeness_score
  • v_measure_score
  • mutual_info_score

According to the scikit-learn definitions that cuML mirrors, these metrics are independent of the absolute label values—a permutation of class or cluster labels must not change the score. This is documented for each metric and for the combined function in scikit-learn. scikit-learn

In practice, with fixed y_true and y_pred, replacing y_pred by 1 - y_pred yields substantially different scores (often ~4× larger), indicating a violation of permutation invariance.

Steps/Code to reproduce bug Minimal script (CPU NumPy arrays to avoid any host/device ambiguity).

import cupy as cp
from cuml.metrics import (
    homogeneity_score, completeness_score,
    mutual_info_score, v_measure_score
)

y_true = cp.array([0, 0, 0, 1, 1, 0, 1])
y_pred = cp.array([1, 0, 1, 0, 1, 1, 0])

funcs = [
    homogeneity_score,
    completeness_score,
    mutual_info_score,
    v_measure_score,
]

for f in funcs:
    s1 = f(y_true, y_pred)
    # permute label identities (0 ↔ 1)
    s2 = f(y_true, 1 - y_pred)   
    print(f.__name__, "s1=", s1, "s2=", s2)
    # Expected: s1 == s2 (up to floating tol)
    # Actual (observed): s1 != s2, often s2 ≈ 4 * s1

Detailed Traceback(Actual Output)

Expected behavior Scores should be identical (within numerical tolerance) when class/cluster label identities are permuted. E.g., metric(y_true, y_pred) should equal metric(y_true, 1 - y_pred) for the four metrics above, per the standard definitions in scikit-learn.

Environment details:

  • Environment location: Bare-metal
  • Linux Distro/Architecture: [Ubuntu 20.04 x86_64]
  • GPU Model/Driver: [A800, driver 525.147.05]
  • CUDA: [12.9, V12.9.86]
  • Python: [3.12.11]
  • cuML version where it reproduces: 25.08
  • Method of cuDF & cuML install: conda
    • Due to the such long result, I prefer to paste the command line I used to create the virtual environment. conda create -n rapids-25.08 -c rapidsai -c conda-forge -c nvidia
      rapids=25.08 python=3.12 'cuda-version>=12.0,<=12.9'

Additional context

  • scikit-learn homogeneity_score: “This metric is independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score value in any way.” homogeneity_score
  • scikit-learn completeness_score: same invariance statement. completeness_score
  • scikit-learn v_measure_score: defined as harmonic mean of homogeneity and completeness; also documented as independent of label values (and symmetric in swapping labels_true/labels_pred). v_measure_score
  • scikit-learn mutual_info_score: same invariance statement. mutual_info_score

Contributor guide