piskvorky/gensim

Improving computational time in LDASeqModel

Open

#1545 opened on Aug 22, 2017

View on GitHub
 (9 comments) (1 reaction) (0 assignees)Python (15,144 stars) (4,349 forks)batch import
Hacktoberfestdifficulty mediumfeaturegood first issueperformance

Description

Hi,

I was training a LDASeqModel on ~40k documents through 20 years (20 time slices in my case) and after 2 days of computation, still didn't finish.

I realized that only LDAModel is used in this class. Shouldn't be better to use LDAMulticore since it can also handle the case of single core ?

Thanks

Contributor guide

Tech stack
python
Domain
datamachine learningperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Pythongensim
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50
Research direction
Investigate the LDASeqModel class in gensim's source code to identify where LDAModel is used and assess the feasibility of replacing it with LDAMulticore. Check the LDAMulticore implementation to ensure it supports the same interface. Consider any potential side effects, such as changes in behavior or dependencies. Reference the issue comments for any maintainer feedback.