Nonparametric Estimation of Latent Distributions in Mixture Models

We discuss a nonparametric estimation method of the latent (mixing) distribution in mixture models. Lindsay’s theorem tells that the maximum likelihood estimate of the mixing distribution is given by a discrete distribution whose support consists of distinct points, the number of which is no more than the sample size. This provides a framework for determining the number of mixture components from data. However, it is vulnerable to overfitting because of the flexibility of the nonparametric estimation. We propose an objective function with one parameter, the minimization of which becomes the maximum likelihood estimation or the kernel vector quantization in special cases. Generalizing Lindsay's theorem, we prove the existence and discreteness of the optimal mixing distribution and devise an algorithm to calculate it. It is demonstrated that with an appropriate choice of the parameter, the proposed method is less prone to overfitting than the maximum likelihood method. Furthermore, we show the connection between the unifying estimation framework and the rate-distortion theory.