piskvorky/gensim

WordEmbeddingsKeyedVectors.add() doesn't clear `vectors_norm`, causing `IndexError` on later `most_similar()`

Open

#2,532 opened on Jun 18, 2019

View on GitHub
 (17 comments) (0 reactions) (0 assignees)Python (15,144 stars) (4,349 forks)batch import
Hacktoberfestbugdifficulty easygood first issueimpact MEDIUMreach LOW

Description

As reported in a StackOverflow question/answer: https://stackoverflow.com/a/56641265/130288

An adapted version of the asker's minimal test case (which could become a unit test):

import numpy as np
from gensim.models.keyedvectors import WordEmbeddingsKeyedVectors

kv = WordEmbeddingsKeyedVectors(vector_size=3)
kv.add(entities=['a', 'b'],
       weights=[np.random.rand(3), np.random.rand(3)])
kv.most_similar('a')  # works

kv.add(entities=['c'], weights=[np.random.rand(3)])
kv.most_similar('c')  # fails with `IndexError`

Clearing the vectors_norm property (with either del or assignment-to-None) should be sufficient to trigger re-calculation upon the next most_similar().

Contributor guide

Tech stack
pythonnumpy
Domain
machine learning
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
pythonbasic numpyfamiliarity with gensim
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Look at the file `gensim/models/keyedvectors.py`, specifically the `add()` method and the `most similar()` method. The issue is that `add()` does not clear the cached `vectors norm` property, causing `most similar()` to fail with `IndexError` when called after adding new entities. The fix is to set `self.vectors norm = None` inside `add()` after modifying the vectors. The provided test case can be used to verify the fix. No maintainer clarification needed; the solution is straightforward.
WordEmbeddingsKeyedVectors.add() doesn't clear `vectors_norm`, causing `IndexError` on later `most_similar()` · piskvorky/gensim#2532 | Good First Issue