Score Matching for Models with Latent Variables

Abstract: Undirected graphical models such as Markov random fields or Boltzmann machines prove useful in many signal processing and machine learning tasks. However, parameter estimation in these models is difficult due to the intractable normalizing constant in their probability density functions. One powerful technique for parameter estimation in such models is score matching. This technique makes use of an objective function which is independent of the normalizing constant and constitutes locally consistent estimators for the parameters of such models. However, score matching is only applicable to fully-observed models. We extend the applicability of score matching to models with latent variables. Our estimators are unbiased, based on Monte Carlo integration. Unbiased gradient estimators open the way to optimization through stochastic approximation. We demonstrate the performance of our methodology on Gamma Markov random fields and restricted Boltzmann machines.

Bio: Onur Dikmen received the B.Sc., M.Sc., and Ph.D. degrees in computer engineering from Bogaziçi University, Istanbul, Turkey. He worked at Télécom ParisTech, France, as a CNRS Research Associate. He is currently with the Department of Information and Computer Science at Aalto University, Finland. His research interests include statistical signal processing, Bayesian statistics, and approximate inference. He works on Bayesian source modeling and nonnegative matrix factorization for source separation.

Host: Sohan Seth