NicolasHug/Surprise

Potential numerical instability in pearson sim

Open

#216 opened on Oct 15, 2018

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Python (6,098 stars) (1,043 forks)batch import
help wanted

Description

Following #214, there might be some numerical instability in pearson similarity computation, probably caused by the sqrt function receiving either negative values, NaN or infinity.

I couldn't reproduce the issue myself.

Contributor guide

Tech stack
python
Domain
machine learning
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
Pythonbasic linear algebranumpyfloating point arithmetic
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
Investigate the Pearson similarity computation in the Surprise library, likely in the file `surprise/similarities.py` or similar. Look for the sqrt function call and ensure it doesn't receive negative values or NaN. Check the linked issue #214 for context. Attempt to reproduce the instability by testing with datasets that have zero variance or near zero similarities. Consider adding a small epsilon or clamping values before sqrt.