piskvorky/gensim

bug in gensim.summarization.mz_entropy.mz_keywords

Open

#2,523 opened on Jun 10, 2019

View on GitHub
 (1 comment) (1 reaction) (0 assignees)Python (15,144 stars) (4,349 forks)batch import
Hacktoberfestbugdifficulty easygood first issueimpact LOWreach LOW

Description

Problem statement:

It seems to be a bug if the text is too short and number of words is lower than blocksize. In my case the values were: n_words (232.0) and blocksize (1024).

Log:

gensim\summarization\mz_entropy.py:127: RuntimeWarning: invalid value encountered in double_scalars
  - __log_combinations(n_words, blocksize)

Dirty solution:

Override blocksize value from the default 1024 to something lower:

mz_keywords(text, blocksize=128)

Contributor guide

Tech stack
python
Domain
data
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
under 1 hour
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Pythonbasic knowledge of gensim summarization module
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
The bug occurs at gensim/summarization/mz entropy.py line 127 when n words (or num words) is less than blocksize. The fix should check if n words < blocksize and either adjust blocksize or skip the division. The issue mentions overriding blocksize as a dirty solution, but a proper fix would involve handling this edge case in the function. Look at the function signature and the log combinations helper to understand the math. Consider raising a warning or adjusting blocksize automatically.