Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

multiclass_metrics_fn "hits@n" does not allow specifying n and returns hard-coded HITS@1/5/10

Geschlossen
  • Stimmt etwas nicht?
#1,248 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 3 Tagen

@Karnis052 arbeitet bereits daran.

Seit 11.10.2026.

  • #1289 von @Karnis052 — offen

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
48/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
machine-learning

Rechercherichtung

Start by reading the implementation and docstring for multiclass_metrics_fn() in the PyHealth metrics package, focusing on the hits@n and mean_rank branches. Check the existing metrics tests for their conventions, then add coverage for a caller-selected cutoff and invalid values once the intended API is settled. Done means the API, returned metric names, validation, tests, and documented accepted metrics agree.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

multiclass_metrics_fn() accepts "hits@n" as a metric, but there is currently no way for the caller to specify the value of n.

The current implementation handles the literal "hits@n" and always calculates three hard-coded cutoffs:

elif metric == "hits@n":
    argsort = np.argsort(-y_prob, axis=1)
    ranking = np.array([
        np.where(argsort[i] == y_true[i])[0][0]
        for i in range(len(y_true))
    ]) + 1

    output["HITS@1"] = np.count_nonzero(ranking <= 1) / len(ranking)
    output["HITS@5"] = np.count_nonzero(ranking <= 5) / len(ranking)
    output["HITS@10"] = np.count_nonzero(ranking <= 10) / len(ranking)

This makes the @n portion of the metric name misleading because callers cannot actually select n.

For example, there is currently no supported way to request HITS@3 or HITS@20.

Reproduction

import numpy as np
from pyhealth.metrics import multiclass_metrics_fn

y_true = np.array([0, 1, 2])

y_prob = np.array([
    [0.8, 0.1, 0.1],
    [0.2, 0.5, 0.3],
    [0.4, 0.3, 0.3],
])

result = multiclass_metrics_fn(
    y_true,
    y_prob,
    metrics=["hits@n"],
)

print(result)

Actual behavior

Passing:

metrics=["hits@n"]

does not provide any way to specify n.

Instead, the function always adds three fixed entries to the output:

HITS@1
HITS@5
HITS@10

As a result, callers cannot request another valid cutoff such as HITS@3, HITS@20, etc.

There is also a mismatch between the requested metric name and the returned metric names: the caller requests one metric ("hits@n"), while the function produces three differently named output entries.

Expected behavior

The public API should make the HITS cutoff explicit and provide a way to specify the desired value of n.

For example, one possible interface would be:

metrics=["hits@3"]

which would return something like:

{
    "HITS@3": ...
}

Similarly:

metrics=["hits@20"]

would calculate HITS@20 rather than requiring a hard-coded cutoff in the implementation.

Alternatively, the function could expose a separate argument for the requested n.

If the intended API is specifically to calculate HITS@1, HITS@5, and HITS@10 together, the accepted metric name and documentation should make that fixed behavior explicit instead of exposing it as "hits@n".

Possible fix

One approach would be to recognize metric names beginning with "hits@", parse the requested cutoff, validate it, and calculate that cutoff dynamically.

Conceptually:

elif metric.startswith("hits@"):
    n = int(metric.split("@", 1)[1])

    argsort = np.argsort(-y_prob, axis=1)
    ranking = np.array([
        np.where(argsort[i] == y_true[i])[0][0]
        for i in range(len(y_true))
    ]) + 1

    output[f"HITS@{n}"] = np.count_nonzero(ranking <= n) / len(ranking)

The implementation would also need appropriate validation for malformed or invalid values of n.

This is only one possible API design; the maintainers may prefer to preserve the current combined HITS@1/5/10 behavior and document it explicitly instead.

Documentation

The multiclass_metrics_fn() docstring currently lists the accepted metrics, but "hits@n" is not included in that list even though it is implemented.

The ranking metrics should be documented once the intended API is established.

Additional context

"mean_rank" is implemented immediately after "hits@n" and calculates both:

mean_rank
mean_reciprocal_rank

but "mean_rank" is also absent from the documented list of accepted metrics.

I searched the existing PyHealth issues for terms including hits@n, HITS@10, and mean_reciprocal_rank and did not find an existing issue covering this behavior.

Vorherrschende Sprache
Python
Sterne
1.7k
Forks
803
Ø Merge
5 T. 7 Std.
Gemergte PRs (30 T.)
58

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus sunlabuiuc/PyHealth

Alle Issues in sunlabuiuc/PyHealth

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.