LatentDirichletAllocation: Modernization of Inference

Firstly, I'm rather new around here; I hope this is the correct place for proposing replacement of algorithms. If it isn't, it would be awful kind of someone to point me in the right direction.

Summary

LatentDirichletAllocation uses a very old (2003) algorithm to estimate its parameters. I tried using the sklearn function for my research, but found that it was unsatisfactory compared to newer algorithms. I have implemented some newer algorithms, and would be happy to get one of them in a format conforming to sklearn's requirments and contribute them. I only have experience in using batch estimation (so no online), but I'd be happy to look into newer online algorithms as well.

Proposed New Algorithm

--Collapsed Variational Bayes (Zeroth Order Taylor Expansion)--

Introduced in 2012 in Asuncion et al, the Zeroeth Order Taylor approximation to Collapsed Variational Bayes (often abbreviated CVB0) is a much more modern algorithm than the original variational bayes algorithm proposed in Blei Ng Jordan, 2003.

It is essentially a deterministic version of Collapsed Gibbs Sampling (not introduced in Wilson and Chew, 2010, but I find their description quite understandable), wherein the topic-word propensity matrix (called $\beta$ in Blei et al and sometimes called $\phi$ in newer articles) as well as the document-topic prevalence vector (called $\theta$ in Blei et al) are analytically integrated out. This results in a posterior only on the word-topic associations (called $z$ in Blei et al), over which a variational distribution is defined. Unfortunately, this cost function is not analytic, but taylor series approximations yield good empirical results.

Advantages of New Algorithm

Based on my personal experience and reviewing relevant literature, I see the benefits of the new algorithm as threefold:

1. Considerably faster -- Asuncion et al report a tenfold increase. I have observed much higher in practice on certain datasets (though, admittedly, I did not spend much time optimizing my cpp implementations of either algorithm).
2. More accurate -- Theoretically, the process of integrating out certain parameters results in a lower bound on the loglikelihood that is strictly better than the lower bound induced by traditional VB. Empirically, taylor approximations of this bound are still able to beat traditional VB.
3. Increase in convexity of cost surface -- This is something I have not seen in the literature, but have observed myself. It is very important in my research that running the algorithm twice should produce the same result, something which seems not to occur with traditional VB. The newer algorithm seems to be much better at avoiding multimodality of the cost surface.

\vspace{5em}

I've benefited very much over the last several years from using sklearn, both as an academic in research and classes, as well as during internships in industry. I feel an obligation to contribute, now that I can, should these improvements be desired. My implementation is in C, but I can easily create a python wrapper for it (or re-write in python, if that's desired).