Notes on the use of autoencoders

Construction an autoencoder (A) is a unsupervised learning NN technique in which an input X is mapped to itself X->A->X. Importantly there are multiple layers in this NN which contains in the interior a "bottleneck" which has a capacity smaller than the input and output layers. The purpose of the bottleneck is to form a set of latent variables encoding the salient part of the samples while filtering out the noise. Since 1990s, there have been introduced several varients.

More flexible than PCA

Like PCA, autoencoders can be use for dimentionality reduction, but have the added benefit that datapoints are not necessarily compressed into a hyperplane as the encoding is allowed to be non-linear.

Survey and Overview

• Definition Wikipedia
• Intro via anomaly detection Anomaly Detection with Autoencoders Made Easy
• Applied Deep Learning Part 3 (Autoencoders)

Variational Autoencoders

• Explanation and theory, old-ish example uses Caffe Tutorial on Variational Autoencoders
• Generation of of sample data using variational autoencoders with example in python-keras Generating new faces with Variational Autoencoders.
• Variational Autoencoder example Intuitively Understanding Variational Autoencoders

KL Divergence

• How to use Kullback-Leibler divergence (KL divergence) with Keras?
• Kullback-Leibler Divergence Explained
• As of Tensorflow 1.15, KL divergence is implemented directly see Keras docs

Disentangled Variational Autoencoders

• Introduction to disentangled VAE What a disentangled Net We Weave: Representation Learning in VAEs (Pt.1).
• With Great Power Comes Poor Latent Codes: Representation Learning in VAEs (Pt. 2).
• InfoVAE InfoVAE: Information Maximizing Variational Autoencoders
  

Variations

• Use of ConvLSTM Anomaly Detection in Videos using LSTM Convolutional Autoencoder
• Example of convolutional autoencoders Convolutional Autoencoders for Image Noise Reduction
• multi input
• SOM-VAE described in SOM-VAE: Interpretable Discrete Representation Learning on Time Series\.

Related

• Video frame prediction with encoding and decoding layers and a bottleneck London TensorFlow Meetup - Convolutional LSTMs for video prediction

Howto

A variational autoencoder implementation in keras with a custom training step, also applicable is Custom Training Using the Stragegy Class. Withing a custom loss function, it's sometimes necessary to get the epoch number.

Handling the tradeoff reconstruction error and KL divergence example in Balancing Reconstruction vs KL Loss Variational Autoencoder on Cross Validated stack exchange.