Collecting Links

• LMVP: Vido predictor with Leaked Motion Information
• Geometry-Based Bext Frame Prediction from Monocular Video
• Predicting Future Frames using Retrospective Cycle GAN
• Next-frame prediction with Conv-LSTM

Collecting links

• Variational autoencoder implementation in keras with a custom training step
• Custom Training Using the Strategy Class.
• Withing a custom loss function, it's sometimes necessary to get the epoch number.
• Some reserch in determining, controlling the VAE loss function to avoid posterior collapse in Re-balancing Variational Autoencoder Loss for Molecule Sequence Generation.
• A review of Dropout as applied to RNNs
• Dropout with LSTM Networks for Time Series Forecasting

Overcoming error: no video mode activated

A bug certainly. There are at least 2 reasons. It's connected to the use of GRUB_TIMEOUT_STYLE part of the fix is to unset this as described here

#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10

Additionally, there is a scan of menu fonts that happen during boot, but if the boot partition is different than the /usr partition, then those fonts will not be found causing an error. A workaround is to copy those fonts to /boot as described in comment #24 of the bug thread

   for dir in "${pkgdatadir}" "`echo '/boot/grub' | sed "s,//*,/,g"`" /usr/share/grub ; do
        for basename in unicode unifont ascii; do
            path="${dir}/${basename}.pf2"
            if is_path_readable_by_grub "${path}" > /dev/null ; then
                font_path="${path}"
            else
                continue
            fi
            break 2
        done
   done

After performing these updates, run update-grub

VFIO setup

Setting up a VM using VFIO allows for near full rate use of the GPU and a responsive desktop; desireable. The important aspects include

• hardware that support IOMMU
• a GPU that can be dedicated to the guest
• a configuration that assigs resources, especially by restricting access to the prospective GPU from the host video driver.

Links

• Ubuntu setup discussed but unfortunatly is system wide rather than via boot option in Ubuntu 18.04 – VFIO PCIe Passthrough
• Arch-linux analog setup and docs in PCI passthrough via OVMF and Can I load the vfio-pci module using a kernel parameter? reddit question

Example /etc/grub.d/40_custom entry

Assign and reserve a GPU via boot option. Assumes a specific partitioning setup; LVM and a root LV in a vg0 VG, AMD hardware; those specifics are not required, the important part is turning on iommu, and assigning vfio ids as kernel arguments.

menuentry "VFIO" {
  root="hd0,gpt2"
  linux	/vmlinuz-4.15.0-96-generic root=/dev/mapper/vg0-root ro amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 vfio-pci.ids=10de:1b80,10de:10f0
  initrd /initrd.img-4.15.0-96-generic
}

There are at least 5 dependencies here.

1. Software dependency on TensorFlow version.
2. TensorFlow dependency on python version.
3. TensorFlow dependency on cuda and cuDNN
4. Cuda dependency on NVidia drivers.
5. NVidia driver dependency on NVidia hardware.

Software Dependency

Starting with an (opensource) software install, the typical method calls for installation of various support frameworks in particular TensorFlow. Assuming pip rather than anaconda

pip install tensorflow_gpu

Usually the required version of cuda is that version that was the most up-to-date version when reference tensorflow is compiled.

Tensorflow python requirements

*Python version 3.8+ is supported only by TensorFlow 2.0+
*Python version 3.7 is supported by TensorFlow 1.15.x

Cuda Library Requirements of TensorFlow

*TensorFlow 2.0 requires cuda 10.2
*TensorFlow 1.15.x requires cuda 10.0
*TensorFlow 1.12.x requires cuda 9.2

NN Support Library (cuDNN) Depencies

Likewise there is a cuDNN, cuda set of dependencies. Typically there will be multiple compatible versions, each one is a separate download available from the NVidia developer website (see below); additionally so are the install instructions

*cuda version 10.2 is only compatible with cuDNN version 7.6.5
*cuda versions 10.1, 10.0, 9.2, 9.0 support cuDNN versions 7.6.5, 7.6.4, 7.6.3, 7.6.2, 7.6.1 7.6.0, 7.5.1, 7.5.0
*cuda versions 10.0, 9.2, 9.0 support cuDNN versions 7.4.2, 7.4.1, 7.3.1, 7.3.0
*cuda versions 9.2, 9.0, 8.0 support cuDNN versions 7.1.4, 7.1.3, 7.1.2
*cuda versions 9.1, 9.0, 8.0 supports cuDNN version 7.0.5
*cuda version 9.0 supports cuDNN version 7.0.4
*cuda versions 8.0, 7.5 support cuDNN versions 6.0, 5.1, 5.0
*cuda 7.0 and later supports cuDNN versions 4 and 3
*cuda 6.5 and later supports cuDNN version 2

Cuda NVidia driver compatibility and driver-hardware compatibility

Finally, different versions of the cuda library require different versions of the NVidia drivers while each driver requires a minimum hardware architecture. Generally newer libraries require newer hardware as they leverage newer hardware capabilities. Refer to the NVidia cuda compatibility page for details.

Construction an autoencoder (A) is a unsupervised learning NN technique in which an input X is mapped to itself X->A->X. Importantly there are multiple layers in this NN which contains in the interior a "bottleneck" which has a capacity smaller than the input and output layers. The purpose of the bottleneck is to form a set of latent variables encoding the salient part of the samples while filtering out the noise. Since 1990s, there have been introduced several varients.

More flexible than PCA

Like PCA, autoencoders can be use for dimentionality reduction, but have the added benefit that datapoints are not necessarily compressed into a hyperplane as the encoding is allowed to be non-linear.

Survey and Overview

• Definition Wikipedia
• Intro via anomaly detection Anomaly Detection with Autoencoders Made Easy
• Applied Deep Learning Part 3 (Autoencoders)

Variational Autoencoders

• Explanation and theory, old-ish example uses Caffe Tutorial on Variational Autoencoders
• Generation of of sample data using variational autoencoders with example in python-keras Generating new faces with Variational Autoencoders.
• Variational Autoencoder example Intuitively Understanding Variational Autoencoders

KL Divergence

• How to use Kullback-Leibler divergence (KL divergence) with Keras?
• Kullback-Leibler Divergence Explained
• As of Tensorflow 1.15, KL divergence is implemented directly see Keras docs

Disentangled Variational Autoencoders

• Introduction to disentangled VAE What a disentangled Net We Weave: Representation Learning in VAEs (Pt.1).
• With Great Power Comes Poor Latent Codes: Representation Learning in VAEs (Pt. 2).
• InfoVAE InfoVAE: Information Maximizing Variational Autoencoders
  

Variations

• Use of ConvLSTM Anomaly Detection in Videos using LSTM Convolutional Autoencoder
• Example of convolutional autoencoders Convolutional Autoencoders for Image Noise Reduction
• multi input
• SOM-VAE described in SOM-VAE: Interpretable Discrete Representation Learning on Time Series\.

Related

• Video frame prediction with encoding and decoding layers and a bottleneck London TensorFlow Meetup - Convolutional LSTMs for video prediction

Howto

A variational autoencoder implementation in keras with a custom training step, also applicable is Custom Training Using the Stragegy Class. Withing a custom loss function, it's sometimes necessary to get the epoch number.

Handling the tradeoff reconstruction error and KL divergence example in Balancing Reconstruction vs KL Loss Variational Autoencoder on Cross Validated stack exchange.