Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

kg_emb models: default L3 penalty has no coefficient, negative scores are summed, and ns other than "adv" breaks TransE and RotatE

オープン
  • 何か問題がありますか？
#1,346 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
numpy, python, pytorch
領域
machine-learning

調査の方向性

Start with pyhealth/medcode/pretrained_embeddings/kg_emb/models/kg_base.py, then read distmult.py, complex.py, transe.py, and rotate.py alongside the linked RotatE reference implementation. Check the reported regularization, negative scoring, and non-adv sampling behaviors, and add the hand-set embedding tests the issue proposes. Done means those tests cover the reported defects and the models train and score consistently across supported options.

索引モデルが issue の本文から書いたものです。

説明

These are pre-existing defects in pyhealth/medcode/pretrained_embeddings/kg_emb/models. On master the code is unreachable, since kg_emb fails to import (#952); #1202 makes the module importable again and leaves the behaviour described here unchanged. In the rejection loops it replaces the deprecated np.in1d(..., assume_unique=True) with np.isin(...) without that flag: the draws are made with replacement, and on NumPy's sort path the flag dropped repeated draws of a valid negative, so the negatives drawn on larger graphs change, but the non-termination described below does not. Fixing these defects changes trained models and reported metrics, so, as @jhnwu3 agreed on Discord on 27 Oct, they are tracked here rather than in that PR. Links point to master at 0bfc345.

1. The default L3 penalty of DistMult and ComplEx has no coefficient

DistMult and ComplEx default to use_regularization="l3" (distmult.py#L21, complex.py#L22). KGEBaseModel.forward adds l3_regularization() to the loss with no weight (kg_base.py#L307-L308), and that term sums |E|^3 + |R|^3 over the whole entity and relation tables (distmult.py#L33-L34, complex.py#L44-L45), not over the embeddings of the batch.

The penalty then dominates the loss. On a synthetic graph of 200 entities in 10 clusters of 20, with one relation ("same cluster", 2,000 of the 3,800 ordered within-cluster pairs, split into 1,600 training and 400 test triples), trained for 60 epochs with Adam (lr 1e-3, batch 64) and otherwise default model arguments, on the #1202 branch: the mean training loss approaches ln 2 = 0.6931 and stays there (0.6999 for DistMult and 0.6932 for ComplEx at epoch 10, 0.6931 at epoch 60), every entity embedding entry shrinks below 1e-3 in absolute value, and the filtered test MRR, over head and tail prediction, stays at the level of an untrained model:

Model MRR, init seed 0 MRR, init seed 1
DistMult, untrained 0.031 0.024
DistMult, default L3 0.024 0.029
DistMult, use_regularization=None 0.319 0.304
ComplEx, untrained 0.023 0.030
ComplEx, default L3 0.027 0.032
ComplEx, use_regularization=None 0.269 0.265
Script (runs on the #1202 branch; the MRR is PyHealth's own filtered evaluation) 
"""Filtered test MRR of DistMult and ComplEx with and without their default
L3 penalty, on a synthetic clustered graph.

Runs on the branch of sunlabuiuc/PyHealth#1202, where kg_emb imports.
Consumes nothing on disk; prints one line per run.

The graph has 200 entities in 10 clusters of 20 and one relation, "same
cluster": 2,000 of the 3,800 ordered pairs of distinct entities within a
cluster, split 80/20. The models keep their default arguments apart from
use_regularization. The filtered MRR is PyHealth's: forward in eval mode,
which filters every known train or test triple, then multiclass_metrics_fn,
over head and tail prediction.
"""
import numpy as np
import polars as pl
import torch

from pyhealth.medcode.pretrained_embeddings.kg_emb.datasets import KnownTriples
from pyhealth.medcode.pretrained_embeddings.kg_emb.models import ComplEx, DistMult
from pyhealth.metrics import multiclass_metrics_fn
from pyhealth.processors import KGTripleProcessor

N_ENTITIES, N_CLUSTERS, N_TRIPLES = 200, 10, 2000
TRAIN_FRACTION = 0.8
EPOCHS, BATCH, LR = 60, 64, 1e-3
GRAPH_SEED = 0
INIT_SEEDS = (0, 1)


def make_graph():
    size = N_ENTITIES // N_CLUSTERS
    pairs = [
        (c * size + i, 0, c * size + j)
        for c in range(N_CLUSTERS)
        for i in range(size)
        for j in range(size)
        if i != j
    ]
    order = np.random.default_rng(GRAPH_SEED).permutation(len(pairs))[:N_TRIPLES]
    triples = [pairs[k] for k in order]
    n_train = int(TRAIN_FRACTION * len(triples))
    return triples[:n_train], triples[n_train:]


class Part:
    """Stands in for a set_task part: a processor fitted on the training
    triples, and the known triples of both parts."""

    def __init__(self, train, test):
        processor = KGTripleProcessor(num_entities=N_ENTITIES, num_relations=1)
        processor.fit([{"triple": t} for t in train], "triple")
        self.input_processors = {"triple": processor}
        ids = pl.DataFrame(
            train + test, schema=["head_id", "relation_id", "tail_id"], orient="row"
        )
        self.known_triples = KnownTriples.from_ids(ids)


def filtered_mrr(model, test):
    model.eval()
    with torch.no_grad():
        out = model(triple=torch.tensor(test))
    scores = multiclass_metrics_fn(
        out["y_true"].numpy(), out["y_prob"].numpy(), metrics=["mean_rank"]
    )
    return scores["mean_reciprocal_rank"]


def build(cls, part, seed, regularization):
    torch.manual_seed(seed)
    np.random.seed(seed)  # training negatives are drawn with np.random
    return cls(part, use_regularization=regularization)


def train(model, triples):
    optimizer = torch.optim.Adam(model.parameters(), lr=LR)
    x = torch.tensor(triples)
    model.train()
    epoch_means = []
    for _ in range(EPOCHS):
        order = torch.randperm(len(x))
        losses = []
        for start in range(0, len(x), BATCH):
            loss = model(triple=x[order[start:start + BATCH]])["loss"]
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            losses.append(loss.item())
        epoch_means.append(float(np.mean(losses)))
    return epoch_means


if __name__ == "__main__":
    train_triples, test_triples = make_graph()
    part = Part(train_triples, test_triples)
    print(f"{len(train_triples)} train / {len(test_triples)} test triples")
    for seed in INIT_SEEDS:
        for cls in (DistMult, ComplEx):
            untrained = build(cls, part, seed, "l3")
            print(f"seed {seed} {cls.__name__} untrained: MRR {filtered_mrr(untrained, test_triples):.3f}")
            for regularization in ("l3", None):
                model = build(cls, part, seed, regularization)
                means = train(model, train_triples)
                print(
                    f"seed {seed} {cls.__name__} use_regularization={regularization}: "
                    f"MRR {filtered_mrr(model, test_triples):.3f}, mean loss at epochs "
                    f"10/20/60 {means[9]:.4f}/{means[19]:.4f}/{means[-1]:.4f}, "
                    f"max |E| {model.E_emb.detach().abs().max():.1e}",
                    flush=True,
                )

The RotatE reference implementation (DeepGraphLearning/KnowledgeGraphEmbedding) multiplies its regularization term by an argument that defaults to 0. Two related points in the same lines: the KGEBaseModel docstring says regularization is "False by default" (kg_base.py#L22), and any value other than None or "l3" calls self.regularization() with no argument (kg_base.py#L309-L310), which raises TypeError at the first training step, since every model's regularization takes a batch. #1202 corrects the docstring, not this behaviour.

A possible fix: a named coefficient, 0 or small by default; the penalty computed on the embeddings of the batch, as in the N3 regularizer of Lacroix et al. (2018); and a ValueError in __init__ for unsupported values.

2. Head and tail negative scores are added before the loss

In training, forward adds the i-th head-corrupted score to the i-th tail-corrupted score (kg_base.py#L288), then applies the self-adversarial weighting or logsigmoid to that sum (kg_base.py#L290-L294). The loss therefore depends on s(h'_i, r, t) + s(h, r, t'_i), which is the score of no triple, and a hard negative paired with an easy one gets almost no gradient. The RotatE reference scores each negative on its own, alternating head-corrupted and tail-corrupted batches. Concatenating the two score tensors along the negative dimension before the weighting would give a per-negative loss.

3. ns != "adv" is broken for TransE and RotatE
  • TransE.calc returns +||h + r - t|| unless ns == "adv" (transe.py#L44-L47), whereas the loss, the ranking metrics and inference() treat a higher score as more plausible. Training with ns="uniform" or ns="normal" therefore optimizes the opposite of the TransE objective.
  • RotatE.calc reads self.r_emb_range (rotate.py#L40), which KGEBaseModel.__init__ defines only when ns == "adv" (kg_base.py#L65-L70), so any other value raises AttributeError at the first forward pass.

The four models default to ns="adv", which is not affected. A possible fix: return -distance, or margin - distance, for every ns in TransE; in RotatE, compute the phase scale independently of ns, or reject other values in __init__.

Smaller points in the same code
  • The rejection loops that draw training negatives (kg_base.py#L153, #L169) never end when a pair's filter set covers every entity, which only a degenerate graph produces.
  • The loss returned in evaluation (kg_base.py#L331-L332) treats every candidate as a negative, the true entity included, so it should not serve as a validation criterion.

I can open a PR for these once #1202 is merged, with tests on hand-set embeddings for each.

主要言語
Python
スター
1.7k
フォーク
803
平均マージ
5日 7時間
マージ済み PR（30日）
58

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

sunlabuiuc/PyHealth のほかの issue

sunlabuiuc/PyHealth の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。