RecallGain metric

Open
  • Is something wrong?
#96 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
pandas, python
Domain
machine-learning

Research direction

The issue proposes changes to metric calculation and cross validation, but names no files, tests, or entry points. Start by locating the existing metric implementations and the calc_metrics and cross_validate APIs. Done means RecallGain works for a single reference model and multiple reference models, with the proposed per-user and cross-validation behavior.

Written by the indexing model from the issue text.

Description

enhancement
Feature Description

Given test interactions, we can exclude all Positives that would have been recommended by the reference model (e.g. popular model). After that we can calculate Recall as usual. Resulting metrics is interpreted as a gain to overall Recall which is achieved by the new algorithm in comparison with the reference algorithm.

Why this feature?

Useful for both candidate-generators selection and simple validation protocol in case of strong popularity bias in data (e.g. calculating RecallGain from a Popular model).

Additional context
recall_gain = RecallGain(k=10, k_ref=20)

# one metric
value = recall_gain.calc(reco=recos, ref_recos=ref_recos)
per_user = recall_gain.calc_per_user(reco=recos, ref_recos=ref_recos)

# calc_metrics
calc_metrics(
    metrics = {"precision": precision, "intersecton": intersecton, "recall_gain": recall_gain},
    reco=recos,
    interactions=df_test,
    ref_recos: Union[pd.Datafame, Dict[Hashable, pd.DataFrame]]=ref_recos
)
# here we can pass multiple models with:
ref_recos = {"one": ref_recos_one, "two": ref_recos_two}
# result dict will have `intersecton_one`, `intersecton_two`, `recall_gain_one`, `recall_gain_two` as keys

# cross_validate
cv_results = cross_validate(
    dataset=dataset,
    splitter=splitter,
    models=models,
    metrics=metrics,
    k=10,
    filter_viewed=True,
    ref_models=["one", "two"], # just selecting keys from `models` argument
    validate_ref_models=False  # optionally exclude ref_models from other metrics
)
Dominant language
Python
Stars
457
Forks
55
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from MTSWebServices/RecTools

All issues in MTSWebServices/RecTools

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.