Generative models

Deep Learning for Engineering

Master’s Degree in Electrical Engineering
Ivan Aldaya
 Table of Contents

▶ Introduction to generative models

▶ Autoregressive models

▶ Variational autoencoders

▶ Generative adversarial networks

▶ Other generative approaches

▶ Bibliography

1/20
 Introduction to generative models
General description generative models
Emulating models
In most machine learning applications
X System Y
the model emulates the system in the
presence of the same input
Model Y

Generating models

X System Y
In generative models, the model is
fed with a random input to generate
an output similar to that of the system Random Model Y
2/20
input
 Introduction to generative models
Applications of generative models in engineering

Variational autoencoder

Ma, W., Cheng, F., Xu, Y., Wen, Q., & Liu, Y. (2019). Probabilistic representation and inverse design
of metamaterials based on a deep generative model with semi-supervised learning strategy.
Advanced Materials, 31(35), 1901111.
3/20
 Introduction to generative models
Applications of generative models in engineering

Wen, F., Jiang, J., & Fan, J. A. (2020). Robust freeform metasurface design based on progressively
4/20 growing generative networks. Acs Photonics, 7(8), 2098-2104.
 Introduction to generative models
Main approaches for generative models

Autorregresive Variational Generative

generative models autoencoders advensarial networks

Variation
of the Gen. Disc.
latent
space

In autoregressive models We train a VA and then A generator and a

the elements are com- variations are performed discriminator are sim-
puted from previous on the latent space ultaneously trained to
values. generate new samples
5/20
 Table of Contents

▶ Introduction to generative models

▶ Autoregressive models

▶ Variational autoencoders

▶ Generative adversarial networks

▶ Other generative approaches

▶ Bibliography

6/20
 Autoregressive models
Basics of autoregressive models

• Autorregresive generative models are similar to forecasting

- These models exploit the correlation between time/space values
• The tool is very close to recursive neural networks
y0 y1 y2 yN y0 y1 y2 yN

h0 h1 h2 hN h0 h1 h2 hN

x0 x1 x2 xN x0
Recurrent neural network Auroregressive generative model
• LSTM and GRU units can be adopted
7/20 • Generally used for 1D series/traces
 Autoregressive models
Operation of autoregressive models
Autoregressive models still require training to find the model coefficients
y0 y1 y2 yN • We use known output sequences/traces

• We find the coefficients of hi to

h0 h1 h2 hN minimize the difference between the
model and system outputs
y0 y1 y2 yN
x0
• Once the model is trained, we use
a modified value of the input to h0 h1 h2 hN
generate a new sequence/trace

x0 x0 {y0, y1, y2, ... , yN}

8/20 x0
 Table of Contents

▶ Introduction to generative models

▶ Autoregressive models

▶ Variational autoencoders

▶ Generative adversarial networks

▶ Other generative approaches

▶ Bibliography

9/20
 Variational autoencoders
Operation of Variational autoencoders

In the case of variational autoencoder it is better to start from the training to

understand the operation principle:
Training stage Training stage

Real Latent Real Random New

sample Encoder space Decoder sample input Decoder sample

• The coefficients of the encoder are optimized • Once trained we can use
• The hyperparameters should be tuned: one of these approaches:
- Architectural hyperparametes (CNN, RNN,...) - Use a random input
10/20
- Regularization mechanisms should be used - Use a mod. latent val.
 Variational autoencoders
Operation of Variational autoencoders

• It is important to understand the latent space:

• For example, for the MNIST base, we get:

• It is worth noting
that the latent repre-
sentation is depen-
dent on the AE
• Some points in the
latent space do not
correspond to valid
samples

https://tiao.io/post/tutorial-on-variational-autoencoders-with-a-concise-keras-implementation/
11/20
 Table of Contents

▶ Introduction to generative models

▶ Autoregressive models

▶ Variational autoencoders

▶ Generative adversarial networks

▶ Other generative approaches

▶ Bibliography

12/20
 Generative adversarial networks
Operation of variational autoencoders

An generative adversarial network is composed of two modules:

• A generator that creates samples from random entries
• A discriminator that asses whether the synthetic samples look real
Discrimin.
loss
Real
samples Discriminator

Generator
Random Synthetic loss
input Generator sample

• Generator and discipminator may have different architectures (RNN, CNNs)

• Since we have two networks, we have two figures of merits
13/20
 Generative adversarial networks
Training of generative adversarial networks

The training is composed of two iterative stages:

Discrimin.
Training the discriminator Real loss
samples Discriminator
Generator
Random Synthetic loss
input Generator sample
Discrimin.
Training the generator Real loss
samples Discriminator
Generator
Random Synthetic loss
input Generator sample
14/20
 Generative adversarial networks
Training of variational autoencoders

The training process: The loss function:

1- Initialize the weights randomly We need to define a combined loss:
2- While not converge: • Minimax loss:
2.1- Update the weights of the L = Ex[log(D(x))]+Ez[log(1−D(G(z)))]
discriminator keeping the - D(x) discriminator probability of
parameters of the generator detecting real
constant - Ex the expected value of real samples
2.2- Update the weights of the - G(z) is the output of the generator
generator keeping the - Ez the expected value of real samples
parameters of the discri- • Wasserstein loss:
minator constan L = D(x) − D(G(z))
15/20
 Table of Contents

▶ Introduction to generative models

▶ Autoregressive models

▶ Variational autoencoders

▶ Generative adversarial networks

▶ Other generative approaches

▶ Bibliography

16/20
 Other generative approaches
Other advanced generative approaches

Adversial AEs Boltzmann machines Transformers

Hidden layer
Output

Output
Input

Enc. Dec. Enc. Dec.

Enc. Dec.
Visible layer
Dec. Disc. • Boltzmann machines Enc. Dec.
• Restricted Boltzmann Enc.
VAE AAE
Dec.
machines Input
Karakus, E., & Kose, H. (2020).
Conditional restricted Boltzmann https://www.pluralsight.com/
machine as a generative model resources/blog/ai-and-data/
Makhzani, A., et al. (2015). for body-worn sensor signals. what-are-transformers-
Adversarial autoencoders. IET Signal Processing, 14(10), generative-ai
17/20 arXiv preprint arXiv:1511.05644. 725-736.
 Table of Contents

▶ Introduction to generative models

▶ Autoregressive models

▶ Variational autoencoders

▶ Generative adversarial networks

▶ Other generative approaches

▶ Bibliography

18/20
 Bibliography

1. Goodfellow, I. (2016). Deep learning (Vol. 196). MIT press.

2. Alpaydin, E. (2021). Machine learning. MIT press.

3. chrome-extension:// efaidnbmnnnibpcajpcglclefind-
mkaj/https://deepgenerativemodels.github.io/assets/slides/cs236 lecture3.pdf

4. https://tiao.io/post/tutorial-on-variational-autoencoders-with-a-concise-keras-
implementation/

5. https://developers.google.com/machine-learning/gan

19/20
 Exercise

Discuss what figure of merit is often used in VAEs.

Identify at least one research article (or conference) that employs the VAEs and GANs in
your area.

Tip: for the sake of clarity, I recommend to build a table to summarize the information.

20/20