Online Bayesian Moment Matching for Latent Dirichlet Allocation

when: may 20, 2014 from 11 am to noon

where: A00

speaker: Pascal Poupart

title: Online Bayesian Moment Matching for Latent Dirichlet Allocation

abstract: Latent Dirichlet Allocation (LDA) is one of the most important models for the unsupervised discovery of latent components such as topics, entities and relations based on unlabeled corpora. The estimation of LDA parameters is challenging since maximum likelihood training leads to a non-convex optimization problem and Bayesian learning yields a posterior that consists of a mixture of exponentially many Dirichlet components. As a result, practitioners typically resort to approximations based on Gibbs sampling, variational Bayes or expectation propagation. While these approximations often work well in practice, it is not clear how to detect convergence in Gibbs sampling and local optima may trap both variational Bayes and expectation propagation. Hence, there is a need for more robust techniques. In this talk, I will describe a Bayesian moment matching technique that performs exact inference with respect to a set of initial distributions (different from the prior). The approach starts with a partially defined prior where only the first, second and third order moments are specified. The remaining moments of the prior are selected gradually on a need basis as the data of the corpus is processed. This lazy approach allows us to select moments that ensure tractability of the computation. As a result, the algorithm scales linearly with the amount of data and performs exact Bayesian learning with respect to this incrementally specified prior. The approach will be demonstrated on a topic modeling task with social media data. In this talk, I will describe a Bayesian moment matching technique to estimate the parameters of the Latent Dirichlet Allocation (LDA) model. While Bayesian learning yields a posterior with exponentially large mixture of Dirichlets, the full posterior is rarely used in practice. A few moments (i.e, mean and variance) are often sufficient to estimate the parameters and some form of confidence. We propose a moment matching technique to compute the first and second order moments of the posterior in an online fashion. While moment matching leads to an approximation, we show that there exists a set of initial distributions (different from the prior) for which exact inference yields the same first and second order moments in the posterior. The approach compares favorably to existing parameter estimation techniques for LDA both in terms of accuracy and time on synthetic and real data from social media.