Loading...

Representation Learning by Deep Networks and Information Theory

Haji Miri, Mohammad Sina | 2021

504 Viewed
  1. Type of Document: M.Sc. Thesis
  2. Language: Farsi
  3. Document No: 53804 (19)
  4. University: Sharif University of Technology
  5. Department: Computer Engineering
  6. Advisor(s): Soleymani Baghshah, Mahdieh
  7. Abstract:
  8. Representation learning refers to mapping the input data to another space, usually with lower dimensions than the input space. This task can be helpful in improving the performance of methods in downstream tasks, compression, and improving sample generation in generative models. Representation learning is a problem connected to information theory, and information theory's concepts and quantities are used widely in representation learning models. Besides, the representation learning problem is closely related to latent variable generative models. These models usually learn useful representations in their process of training, implicitly or explicitly. So, the usage of latent variable generative models is common for learning representations. One of the most popular generative models for representation learning is the variational autoencoder framework.Various characteristics can be considered for the quality of a representation. One of these characteristics is the disentanglement of a representation. Disentangled representations are more interpretable and can help generative models with more control over their generation process.This research is focused on deep disentangled representation learning based on the variational autoencoder framework. The proposed method is capable of disentangling class-related and class-independent factors of variation in data. It employs an attention mechanism in its latent space in order to improve the process of extracting class-related factors from data. Also, the multimodality of data distribution is dealt with by utilizing mixture models as learnable prior distributions and incorporating the Bhattacharyya coefficient in the objective function to prevent highly overlapping mixtures. The model's encoder is further trained in a semi-supervised manner, with a small fraction of labeled data, to improve representations' interpretability.Quantitative and qualitative experiments on three datasets show that the proposed method learns disentangled representations. This is apparent in the quantitative evaluation on a disentanglement metric called Factor Score. Particularly, the proposed method improves this metric by 0.04 in comparison to some prior methods. Also, the method's qualitative evaluation shows the quality of generated samples and the disentanglement of their factors of variation
  9. Keywords:
  10. Deep Learning ; Information Theory ; Representation Learning ; Representation Disentanglement ; Generative Models with Latent Space

 Digital Object List

 Bookmark

...see more