Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

kg_emb models: default L3 penalty has no coefficient, negative scores are summed, and ns other than "adv" breaks TransE and RotatE

Aperta
  • Qualcosa non va?
#1,346 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
numpy, python, pytorch
Ambito
machine-learning

Direzione di ricerca

Start with pyhealth/medcode/pretrained_embeddings/kg_emb/models/kg_base.py, then read distmult.py, complex.py, transe.py, and rotate.py alongside the linked RotatE reference implementation. Check the reported regularization, negative scoring, and non-adv sampling behaviors, and add the hand-set embedding tests the issue proposes. Done means those tests cover the reported defects and the models train and score consistently across supported options.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

These are pre-existing defects in pyhealth/medcode/pretrained_embeddings/kg_emb/models. On master the code is unreachable, since kg_emb fails to import (#952); #1202 makes the module importable again and leaves the behaviour described here unchanged. In the rejection loops it replaces the deprecated np.in1d(..., assume_unique=True) with np.isin(...) without that flag: the draws are made with replacement, and on NumPy's sort path the flag dropped repeated draws of a valid negative, so the negatives drawn on larger graphs change, but the non-termination described below does not. Fixing these defects changes trained models and reported metrics, so, as @jhnwu3 agreed on Discord on 27 Oct, they are tracked here rather than in that PR. Links point to master at 0bfc345.

1. The default L3 penalty of DistMult and ComplEx has no coefficient

DistMult and ComplEx default to use_regularization="l3" (distmult.py#L21, complex.py#L22). KGEBaseModel.forward adds l3_regularization() to the loss with no weight (kg_base.py#L307-L308), and that term sums |E|^3 + |R|^3 over the whole entity and relation tables (distmult.py#L33-L34, complex.py#L44-L45), not over the embeddings of the batch.

The penalty then dominates the loss. On a synthetic graph of 200 entities in 10 clusters of 20, with one relation ("same cluster", 2,000 of the 3,800 ordered within-cluster pairs, split into 1,600 training and 400 test triples), trained for 60 epochs with Adam (lr 1e-3, batch 64) and otherwise default model arguments, on the #1202 branch: the mean training loss approaches ln 2 = 0.6931 and stays there (0.6999 for DistMult and 0.6932 for ComplEx at epoch 10, 0.6931 at epoch 60), every entity embedding entry shrinks below 1e-3 in absolute value, and the filtered test MRR, over head and tail prediction, stays at the level of an untrained model:

Model MRR, init seed 0 MRR, init seed 1
DistMult, untrained 0.031 0.024
DistMult, default L3 0.024 0.029
DistMult, use_regularization=None 0.319 0.304
ComplEx, untrained 0.023 0.030
ComplEx, default L3 0.027 0.032
ComplEx, use_regularization=None 0.269 0.265
Script (runs on the #1202 branch; the MRR is PyHealth's own filtered evaluation) 
"""Filtered test MRR of DistMult and ComplEx with and without their default
L3 penalty, on a synthetic clustered graph.

Runs on the branch of sunlabuiuc/PyHealth#1202, where kg_emb imports.
Consumes nothing on disk; prints one line per run.

The graph has 200 entities in 10 clusters of 20 and one relation, "same
cluster": 2,000 of the 3,800 ordered pairs of distinct entities within a
cluster, split 80/20. The models keep their default arguments apart from
use_regularization. The filtered MRR is PyHealth's: forward in eval mode,
which filters every known train or test triple, then multiclass_metrics_fn,
over head and tail prediction.
"""
import numpy as np
import polars as pl
import torch

from pyhealth.medcode.pretrained_embeddings.kg_emb.datasets import KnownTriples
from pyhealth.medcode.pretrained_embeddings.kg_emb.models import ComplEx, DistMult
from pyhealth.metrics import multiclass_metrics_fn
from pyhealth.processors import KGTripleProcessor

N_ENTITIES, N_CLUSTERS, N_TRIPLES = 200, 10, 2000
TRAIN_FRACTION = 0.8
EPOCHS, BATCH, LR = 60, 64, 1e-3
GRAPH_SEED = 0
INIT_SEEDS = (0, 1)


def make_graph():
    size = N_ENTITIES // N_CLUSTERS
    pairs = [
        (c * size + i, 0, c * size + j)
        for c in range(N_CLUSTERS)
        for i in range(size)
        for j in range(size)
        if i != j
    ]
    order = np.random.default_rng(GRAPH_SEED).permutation(len(pairs))[:N_TRIPLES]
    triples = [pairs[k] for k in order]
    n_train = int(TRAIN_FRACTION * len(triples))
    return triples[:n_train], triples[n_train:]


class Part:
    """Stands in for a set_task part: a processor fitted on the training
    triples, and the known triples of both parts."""

    def __init__(self, train, test):
        processor = KGTripleProcessor(num_entities=N_ENTITIES, num_relations=1)
        processor.fit([{"triple": t} for t in train], "triple")
        self.input_processors = {"triple": processor}
        ids = pl.DataFrame(
            train + test, schema=["head_id", "relation_id", "tail_id"], orient="row"
        )
        self.known_triples = KnownTriples.from_ids(ids)


def filtered_mrr(model, test):
    model.eval()
    with torch.no_grad():
        out = model(triple=torch.tensor(test))
    scores = multiclass_metrics_fn(
        out["y_true"].numpy(), out["y_prob"].numpy(), metrics=["mean_rank"]
    )
    return scores["mean_reciprocal_rank"]


def build(cls, part, seed, regularization):
    torch.manual_seed(seed)
    np.random.seed(seed)  # training negatives are drawn with np.random
    return cls(part, use_regularization=regularization)


def train(model, triples):
    optimizer = torch.optim.Adam(model.parameters(), lr=LR)
    x = torch.tensor(triples)
    model.train()
    epoch_means = []
    for _ in range(EPOCHS):
        order = torch.randperm(len(x))
        losses = []
        for start in range(0, len(x), BATCH):
            loss = model(triple=x[order[start:start + BATCH]])["loss"]
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            losses.append(loss.item())
        epoch_means.append(float(np.mean(losses)))
    return epoch_means


if __name__ == "__main__":
    train_triples, test_triples = make_graph()
    part = Part(train_triples, test_triples)
    print(f"{len(train_triples)} train / {len(test_triples)} test triples")
    for seed in INIT_SEEDS:
        for cls in (DistMult, ComplEx):
            untrained = build(cls, part, seed, "l3")
            print(f"seed {seed} {cls.__name__} untrained: MRR {filtered_mrr(untrained, test_triples):.3f}")
            for regularization in ("l3", None):
                model = build(cls, part, seed, regularization)
                means = train(model, train_triples)
                print(
                    f"seed {seed} {cls.__name__} use_regularization={regularization}: "
                    f"MRR {filtered_mrr(model, test_triples):.3f}, mean loss at epochs "
                    f"10/20/60 {means[9]:.4f}/{means[19]:.4f}/{means[-1]:.4f}, "
                    f"max |E| {model.E_emb.detach().abs().max():.1e}",
                    flush=True,
                )

The RotatE reference implementation (DeepGraphLearning/KnowledgeGraphEmbedding) multiplies its regularization term by an argument that defaults to 0. Two related points in the same lines: the KGEBaseModel docstring says regularization is "False by default" (kg_base.py#L22), and any value other than None or "l3" calls self.regularization() with no argument (kg_base.py#L309-L310), which raises TypeError at the first training step, since every model's regularization takes a batch. #1202 corrects the docstring, not this behaviour.

A possible fix: a named coefficient, 0 or small by default; the penalty computed on the embeddings of the batch, as in the N3 regularizer of Lacroix et al. (2018); and a ValueError in __init__ for unsupported values.

2. Head and tail negative scores are added before the loss

In training, forward adds the i-th head-corrupted score to the i-th tail-corrupted score (kg_base.py#L288), then applies the self-adversarial weighting or logsigmoid to that sum (kg_base.py#L290-L294). The loss therefore depends on s(h'_i, r, t) + s(h, r, t'_i), which is the score of no triple, and a hard negative paired with an easy one gets almost no gradient. The RotatE reference scores each negative on its own, alternating head-corrupted and tail-corrupted batches. Concatenating the two score tensors along the negative dimension before the weighting would give a per-negative loss.

3. ns != "adv" is broken for TransE and RotatE
  • TransE.calc returns +||h + r - t|| unless ns == "adv" (transe.py#L44-L47), whereas the loss, the ranking metrics and inference() treat a higher score as more plausible. Training with ns="uniform" or ns="normal" therefore optimizes the opposite of the TransE objective.
  • RotatE.calc reads self.r_emb_range (rotate.py#L40), which KGEBaseModel.__init__ defines only when ns == "adv" (kg_base.py#L65-L70), so any other value raises AttributeError at the first forward pass.

The four models default to ns="adv", which is not affected. A possible fix: return -distance, or margin - distance, for every ns in TransE; in RotatE, compute the phase scale independently of ns, or reject other values in __init__.

Smaller points in the same code
  • The rejection loops that draw training negatives (kg_base.py#L153, #L169) never end when a pair's filter set covers every entity, which only a degenerate graph produces.
  • The loss returned in evaluation (kg_base.py#L331-L332) treats every candidate as a negative, the true entity included, so it should not serve as a validation criterion.

I can open a PR for these once #1202 is merged, with tests on hand-set embeddings for each.

Lingua principale
Python
Stelle
1.7k
Fork
803
Merge medio
5g 16h
PR unite (30g)
58

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di sunlabuiuc/PyHealth

Tutte le issue di sunlabuiuc/PyHealth

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.