A deeper dive into
loading data
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Maham Faisal Khan
Senior Data Science Content Developer
Back to our animals dataset
import pandas as pd
pd.read_csv('animals.csv')
animal_name hair feathers eggs milk predator fins legs tail type
skimmer 0 1 1 0 1 0 2 1 2
gull 0 1 1 0 1 0 2 1 2
seahorse 0 0 1 0 0 1 0 1 4
tuatara 0 0 1 0 1 0 4 1 3
squirrel 1 0 0 1 0 0 2 1 1
Type key: mammal (1), bird (2), reptile (3), fish (4), amphibian (5), bug (6), invertebrate (7).
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Back to our animals dataset: defining features
import numpy as np
# Define input features
features = animals[animals.columns[1:-1]])
X = np.array(features).float()
print(X)
features = np.array(
[[0, 1, 1, 0, 1, 0, 2, 1],
[0, 0, 1, 0, 0, 1, 0, 1],
[0, 1, 1, 0, 1, 0, 2, 1],
[1, 0, 0, 1, 0, 0, 2, 1],
[0, 0, 1, 0, 1, 0, 4, 1]])
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Back to our animals dataset: defining target values
# Define target features (ground truth)
target = animals[animals.columns[-1]]
y = np.array(target).float()
array([2, 4, 2, 1, 3])
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Recalling TensorDataset
import torch
from torch.utils.data import TensorDataset
# Instantiate dataset class
dataset = TensorDataset(torch.tensor(X).float(), torch.tensor(y).float())
# Access an individual sample
sample = dataset[0]
input_sample, label_sample = sample
print('input sample:', input_sample)
print('label_sample:', label_sample)
input sample: tensor([0., 1., 1., 0., 1., 0., 2., 1.])
label_sample: tensor(2.)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Recalling DataLoader
from torch.utils.data import DataLoader
batch_size = 2
shuffle = True
# Create a DataLoader
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=shuffle)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Recalling DataLoader
# Iterate over the dataloader
for batch_inputs, batch_labels in dataloader:
print('batch inputs', batch_inputs)
print('batch labels', batch_labels)
batch inputs: tensor([[0., 0., 1., 0., 0., 1., 0., 1.],
[0., 1., 1., 0., 1., 0., 2., 1.]])
batch labels: tensor([4., 2.])
batch inputs: tensor([[0., 1., 1., 0., 1., 0., 2., 1.],
[1., 0., 0., 1., 0., 0., 2., 1.]])
batch labels: tensor([2., 1.])
batch inputs: tensor([[0., 0., 1., 0., 1., 0., 4., 1.]])
batch labels: tensor([3.])
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Let's practice!
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Evaluating model
performance
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Maham Faisal Khan
Senior Data Science Content Developer
Training, validation and testing
Raw dataset is usually split in three subsets:
Percent of data Role
Training 80-90% Used to adjust the model's parameters
Validation 10-20% Used for hyperparameter tuning
Testing 5-10% Only used once to calculate final metrics
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Model evaluation metrics
In this video, we'll focus on evaluating: In classification, accuracy measures how
Loss well model correctly predicts ground truth
Training labels
Validation
Accuracy
Training
Validation
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Calculating training loss
For each epoch: training_loss = 0.0
we sum up the loss for each iteration of for i, data in enumerate(trainloader, 0):
the training set dataloader # Run the forward pass
...
at the end of the epoch, we calculate the
mean training loss # Calculate the loss
loss = criterion(outputs, labels)
# Calculate the gradients
...
# Calculate and sum the loss
training_loss += loss.item()
epoch_loss = training_loss / len(trainloader)
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Calculating validation loss
After the training epoch, we iterate over the validation set and calculate the average
validation loss
validation_loss = 0.0
model.eval() # Put model in evaluation mode
with torch.no_grad(): # Speed up the forward pass
for i, data in enumerate(validationloader, 0):
# Run the forward pass
...
# Calculate the loss
loss = criterion(outputs, labels)
validation_loss += loss.item()
epoch_loss = validation_loss / len(validationloader)
model.train()
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Overfitting
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Calculating accuracy with torchmetrics
import torchmetrics
# Create accuracy metric using torch metrics
metric = torchmetrics.Accuracy(task="multiclass", num_classes=3)
for i, data in enumerate(dataloader, 0):
features, labels = data
outputs = model(features)
# Calculate accuracy over the batch
acc = metric(output, labels.argmax(dim=-1))
# Calculate accuracy over the whole epoch
acc = metric.compute()
print(f"Accuracy on all data: {acc}")
# Reset the metric for the next epoch (training or validation)
metric.reset()
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Let's practice!
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Fighting overfitting
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Maham Faisal Khan
Senior Data Science Content Developer
Reasons for overfitting
Overfitting: the model does not generalize to unseen data.
model memorizes training data
good performances on the training set / poor performances on the validation set
Possible causes:
Problem Solutions
Dataset is not large enough Get more data / use data augmentation
Model has too much capacity Reduce model size / add dropout
Weights are too large Weight decay
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Fighting overfitting
Strategies:
Reducing model size or adding dropout layer
Using weight decay to force parameters to remain small
Obtaining new data or augmenting data
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
"Regularization" using a dropout layer
Randomly zeroes out elements of the input tensor during training
model = nn.Sequential(nn.Linear(8, 4),
nn.ReLU(),
nn.Dropout(p=0.5))
features = torch.randn((1, 8))
model(i)
tensor([[1.4655, 0.0000, 0.0000, 0.8456]], grad_fn=<MulBackward0>)
Dropout is added after the activation function
Behaves differently during training and evaluation; we must remember to switch modes
using model.train() and model.eval()
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Regularization with weight decay
optimizer = optim.SGD(model.parameters(), lr=1e-3, weight_decay=1e-4)
Optimizer's weight_decay parameter takes values between zero and one
Typically small value, e.g. 1e-3
Weight decay adds penalty to loss function to discourage large weights and biases
The higher the parameter, the less likely the model is to overfit
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Data augmentation
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Let's practice!
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Improving model
performance
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Maham Faisal Khan
Senior Data Science Content Developer
Steps to maximize performance
Overfit the training set
can we solve the problem?
sets a performance baseline
Reduce overfitting
improve performances on the validation
set
Fine-tune hyperparameters
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Step 1: overfit the training set
Modify the training loop to overfit a single data point (batch size of 1)
features, labels = next(iter(trainloader))
for i in range(1e3):
outputs = model(features)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
should reach 1.0 accuracy and 0 loss
helps findings bugs in the code
Goal: minimize the training loss
create large enough model
use a default learning rate
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Step 2: reduce overfitting
Goal: maximize the validation accuracy
Experiment with:
Dropout
Data augmentation
Weight decay
Reducing model capacity
Keep track of each hyperparameter and
report maximum validation accuracy
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Step 2: reduce overfitting
Original model overfitting the Model with too much regularization
training data
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Step 3: fine-tune hyperparameters
Grid search Random search
for factor in range(2, 6): factor = np.random.uniform(2, 6)
lr = 10 ** -factor lr = 10 ** -factor
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Let's practice!
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Wrap-up video
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Maham Faisal Khan
Senior Data Science Content Developer
Summary
Chapter 1 Chapter 3
Discovered deep learning Manipulated the architecture of a neural
network
Created small neural networks
Played with learning rate and momentum
Discovered linear layers and activation
functions Learned about transfer learning
Chapter 2 Chapter 4
Created and used loss functions Learned about dataloaders
Calculated derivatives and use Reduced overfitting
backpropagation Evaluated model performance
Trained a neural network
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Next steps
Course
Intermediate Deep Learning with
PyTorch
Learn
Probability and statistics
Linear algebra
Calculus
Practice
Pick a dataset on Kaggle
Check out DataCamp workspace
Train a neural network
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
Let's practice!
INTRODUCTION TO DEEP LEARNING WITH PYTORCH
