Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

multiclass_metrics_fn "hits@n" does not allow specifying n and returns hard-coded HITS@1/5/10

Fermée
  • Quelque chose ne va pas ?
#1,248 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 3 jours

@Karnis052 y travaille déjà.

Depuis le 11/10/2026.

  • #1289 par @Karnis052 — ouverte

Évaluation

Difficulté
4/5
Temps estimé
1-2 jours
Accessibilité débutants
48/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
python
Domaine
machine-learning

Piste de recherche

Start by reading the implementation and docstring for multiclass_metrics_fn() in the PyHealth metrics package, focusing on the hits@n and mean_rank branches. Check the existing metrics tests for their conventions, then add coverage for a caller-selected cutoff and invalid values once the intended API is settled. Done means the API, returned metric names, validation, tests, and documented accepted metrics agree.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

multiclass_metrics_fn() accepts "hits@n" as a metric, but there is currently no way for the caller to specify the value of n.

The current implementation handles the literal "hits@n" and always calculates three hard-coded cutoffs:

elif metric == "hits@n":
    argsort = np.argsort(-y_prob, axis=1)
    ranking = np.array([
        np.where(argsort[i] == y_true[i])[0][0]
        for i in range(len(y_true))
    ]) + 1

    output["HITS@1"] = np.count_nonzero(ranking <= 1) / len(ranking)
    output["HITS@5"] = np.count_nonzero(ranking <= 5) / len(ranking)
    output["HITS@10"] = np.count_nonzero(ranking <= 10) / len(ranking)

This makes the @n portion of the metric name misleading because callers cannot actually select n.

For example, there is currently no supported way to request HITS@3 or HITS@20.

Reproduction

import numpy as np
from pyhealth.metrics import multiclass_metrics_fn

y_true = np.array([0, 1, 2])

y_prob = np.array([
    [0.8, 0.1, 0.1],
    [0.2, 0.5, 0.3],
    [0.4, 0.3, 0.3],
])

result = multiclass_metrics_fn(
    y_true,
    y_prob,
    metrics=["hits@n"],
)

print(result)

Actual behavior

Passing:

metrics=["hits@n"]

does not provide any way to specify n.

Instead, the function always adds three fixed entries to the output:

HITS@1
HITS@5
HITS@10

As a result, callers cannot request another valid cutoff such as HITS@3, HITS@20, etc.

There is also a mismatch between the requested metric name and the returned metric names: the caller requests one metric ("hits@n"), while the function produces three differently named output entries.

Expected behavior

The public API should make the HITS cutoff explicit and provide a way to specify the desired value of n.

For example, one possible interface would be:

metrics=["hits@3"]

which would return something like:

{
    "HITS@3": ...
}

Similarly:

metrics=["hits@20"]

would calculate HITS@20 rather than requiring a hard-coded cutoff in the implementation.

Alternatively, the function could expose a separate argument for the requested n.

If the intended API is specifically to calculate HITS@1, HITS@5, and HITS@10 together, the accepted metric name and documentation should make that fixed behavior explicit instead of exposing it as "hits@n".

Possible fix

One approach would be to recognize metric names beginning with "hits@", parse the requested cutoff, validate it, and calculate that cutoff dynamically.

Conceptually:

elif metric.startswith("hits@"):
    n = int(metric.split("@", 1)[1])

    argsort = np.argsort(-y_prob, axis=1)
    ranking = np.array([
        np.where(argsort[i] == y_true[i])[0][0]
        for i in range(len(y_true))
    ]) + 1

    output[f"HITS@{n}"] = np.count_nonzero(ranking <= n) / len(ranking)

The implementation would also need appropriate validation for malformed or invalid values of n.

This is only one possible API design; the maintainers may prefer to preserve the current combined HITS@1/5/10 behavior and document it explicitly instead.

Documentation

The multiclass_metrics_fn() docstring currently lists the accepted metrics, but "hits@n" is not included in that list even though it is implemented.

The ranking metrics should be documented once the intended API is established.

Additional context

"mean_rank" is implemented immediately after "hits@n" and calculates both:

mean_rank
mean_reciprocal_rank

but "mean_rank" is also absent from the documented list of accepted metrics.

I searched the existing PyHealth issues for terms including hits@n, HITS@10, and mean_reciprocal_rank and did not find an existing issue covering this behavior.

Langage dominant
Python
Étoiles
1.7k
Forks
803
Merge moyen
5 j 16 h
PR mergées (30 j)
58

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de sunlabuiuc/PyHealth

Toutes les issues de sunlabuiuc/PyHealth

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.