Abstract: Unsupervised learning of feature hierarchies is often a good initialization for supervised training of deep architectures. In existing deep learning methods these feature hierarchies are built layer by layer in a greedy fashion using auto-encoders or restricted Boltzmann machines. Both yield encoders which compute linear projections followed by a smooth thresholding function. In this work we demonstrate that these encoders fail to find stable features when the required computation is in the exclusive-or class. To overcome this limitation we propose a two-layer encoder which is not restricted in the type of features it can learn. The proposed encoder can be regularized by an extension of previous work on contractive regularization. We demonstrate the advantages of two-layer encoders qualitatively as well as on commonly used benchmark datasets.
Bio: Hannes Schulz is a PhD student at the university of Bonn, Germany. His main research interests are deep learning and computer vision.
Host: Tapani Raiko
Last updated on 26 Nov 2012 by Sohan Seth - Page created on 26 Nov 2012 by Sohan Seth