RecallGain metric

未关闭
  • 哪里不对？
#96 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
停滞
技术栈
pandas, python
领域
machine-learning

调研方向

The issue proposes changes to metric calculation and cross validation, but names no files, tests, or entry points. Start by locating the existing metric implementations and the calc_metrics and cross_validate APIs. Done means RecallGain works for a single reference model and multiple reference models, with the proposed per-user and cross-validation behavior.

由索引模型根据 Issue 内容生成。

描述

enhancement
Feature Description

Given test interactions, we can exclude all Positives that would have been recommended by the reference model (e.g. popular model). After that we can calculate Recall as usual. Resulting metrics is interpreted as a gain to overall Recall which is achieved by the new algorithm in comparison with the reference algorithm.

Why this feature?

Useful for both candidate-generators selection and simple validation protocol in case of strong popularity bias in data (e.g. calculating RecallGain from a Popular model).

Additional context
recall_gain = RecallGain(k=10, k_ref=20)

# one metric
value = recall_gain.calc(reco=recos, ref_recos=ref_recos)
per_user = recall_gain.calc_per_user(reco=recos, ref_recos=ref_recos)

# calc_metrics
calc_metrics(
    metrics = {"precision": precision, "intersecton": intersecton, "recall_gain": recall_gain},
    reco=recos,
    interactions=df_test,
    ref_recos: Union[pd.Datafame, Dict[Hashable, pd.DataFrame]]=ref_recos
)
# here we can pass multiple models with:
ref_recos = {"one": ref_recos_one, "two": ref_recos_two}
# result dict will have `intersecton_one`, `intersecton_two`, `recall_gain_one`, `recall_gain_two` as keys

# cross_validate
cv_results = cross_validate(
    dataset=dataset,
    splitter=splitter,
    models=models,
    metrics=metrics,
    k=10,
    filter_viewed=True,
    ref_models=["one", "two"], # just selecting keys from `models` argument
    validate_ref_models=False  # optionally exclude ref_models from other metrics
)
主要语言
Python
星标
457
派生
55
30 天内没有已合并 PR

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

MTSWebServices/RecTools 的其他 Issue

查看 MTSWebServices/RecTools 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。