Regression
Course: Artificial Intelligence
Fundamentals
Instructor: Marco Bonzanini
Machine Learning Tasks
Supervised Unsupervised
Discrete Data
Classification Clustering
(predict a label) (group similar items)
Continuous Data
Dimensionality
Regression Reduction
(predict a quantity) (reduce n. of variables)
Machine Learning Tasks
Supervised Unsupervised
Discrete Data
Classification Clustering
(predict a label) (group similar items)
Continuous Data
Dimensionality
Regression Reduction
(predict a quantity) (reduce n. of variables)
Classification vs Regression
• Classification:
- supervised
- requires a set of labelled training samples
- predicts a label
• Regression:
- supervised
- requires a set of labelled training samples
- predicts a quantity
Section Agenda
• Introduction to Regression Analysis
• Regression Algorithms
• Linear Regression
• Evaluation
Introduction to
Regression
Regression Analysis
• Set of process for estimating the relationship
between variables
• Heavily studied in statistics
• Can be used to better understand relationships
between variables, although “correlation does not
imply causation”
• Relationship with ML: predictions / forecasting
Applications
• Predict the price of a house
• Predict student’s grades
(time on books vs time on Facebook)
• Predict crop yields
• … Predict a quantity
Variables
• Dependent variables:
- variables that we want to forecast
- their values depend on something else
- denoted as Y (desired output)
• Independent variables:
- variables that explain the other one
- its values are independent
- denoted as X (input)
- a.k.a. features, predictors, covariates
Regression Model
• A function that describes the relationship between
dependent and independent variables
• b: unknown parameters
Regression Example
Regression Example
Price
Problem:
predict the price of a house
given its size
Size
Regression Example
Price
Training data:
samples (size, price)
Size
Regression Example
Price
Training data:
samples (size, price)
$200K
Size
100 sq. m
Regression Example
Price
Training:
find the curve
that best fits the data
Size
Regression Example
Price
Prediction:
given the size of a new house
Size
120 sq. m
Regression Example
Price
$250K
Prediction:
given the size of a new house
estimate the price
Size
120 sq. m
Regression Example
Price
$250K Estimated
$220K Real
Size
Regression Example
Price
$250K
$220K Error:
difference between
estimated and real
Size
Regression
Algorithms
Linear Regression
Linear Regression
• From school (calculus?):
• In statistics:
Linear Regression
• From school (calculus?):
• In statistics: Independent variable
Linear Regression
• From school (calculus?):
• In statistics: Dependent variable
Linear Regression
• From school (calculus?):
• In statistics: Slope
Linear Regression
• From school (calculus?):
• In statistics: Intersect
Linear Regression Error
Price
Actual value
Error:
difference between
estimated and actual
Estimated value
Size
Linear Regression Error
Price
Error:
difference between
estimated and actual
Size
Linear Regression Error (2)
Linear Regression Error (2)
Actual value
Estimated value
Error a.k.a. residual
Linear Regression
Ingredients
• Training data
• Residual error of sample i
• Sum of Squared Residuals
Linear Regression
Approach
• Minimise the SSR function
• a.k.a. Least-Squares Regression
• Treat SSR as “cost function”
• e.g. run Gradient Descent (forward reference)
Evaluation
Regression Evaluation
• Focus on the “error” (smaller = better)
• Mean Absolute Error (MAE)
• Mean Squared Error (MSE)
• Root Mean Squared Error (RMSE)
Mean Absolute Error
Mean Squared Error
Root Mean Squared Error
MAE vs MSE vs RMSE
• One better than the others?
• Squared: penalise bigger errors more
• RMSE: same unit as the variable being considered
Questions?
