Lecture 8: 2 February, 2023
Madhavan Mukund
https://www.cmi.ac.in/~madhavan
Data Mining and Machine Learning
January–April 2023
Linear regression
Training input is
{(x1 , y1 ), (x2 , y2 ), . . . , (xn , yn )}
~
Each input xi is a vector (xi1 , . . . , xik )
Add xi0 = 1 by convention
yi is actual output
How far away is our prediction h✓ (xi ) from
the true answer yi ?
Define a cost (loss) function
n
1X
J(✓) = (h✓ (xi ) yi ) 2
2
i=1
Essentially, the sum squared error (SSE) -
Justified via MLE
Divide by n, mean squared error (MSE)
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 2 / 22
The non-linear case
What if the relationship is
not linear?
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 3 / 22
The non-linear case
What if the relationship is
not linear?
Here the best possible
explanation seems to be a
quadratic
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 3 / 22
The non-linear case
What if the relationship is
not linear?
Here the best possible
explanation seems to be a
quadratic
Non-linear : cross
dependencies
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 3 / 22
The non-linear case
What if the relationship is
not linear?
Here the best possible
explanation seems to be a
quadratic
Non-linear : cross
dependencies
Input xi : (xi1 , xi2 )
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 3 / 22
The non-linear case
What if the relationship is
not linear?
Here the best possible
explanation seems to be a
quadratic
Non-linear : cross
dependencies
Input xi : (xi1 , xi2 )
Quadratic dependencies:
y = ✓0 + ✓1 xi1 + ✓2 xi2 + ✓11 xi21 + ✓22 xi22 + ✓12 xi1 xi2
--
linear quadratic
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 3 / 22
The non-linear case
Recall how we fit a line
⇥ ⇤ ✓0
1 xi 1
✓1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 4 / 22
The non-linear case
Recall how we fit a line
⇥ ⇤ ✓0
1 xi 1
✓1
For quadratic, add new
coefficients and expand
parameters
2 3
⇥ ⇤ ✓0
1 xi1 xi21 4 ✓1 5
✓2
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 4 / 22
The non-linear case
Input (xi1 , xi2 )
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 5 / 22
The non-linear case
Input (xi1 , xi2 )
For the general quadratic
case, we are adding new
derived “features”
xi 3 = xi21
xi 4 = xi22
xi 5 = x i 1 xi 2
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 5 / 22
The non-linear case
Original input matrix
2 3
1 x11 x1 2
6 1 x21 x2 2 7
6 7
6 ··· 7
6 7
6 1 xi xi 2 7
6 1 7
4 ··· 5
1 xn 1 xn 2
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 6 / 22
The non-linear case
Expanded input matrix
2 3
1 x1 1 x1 2 x121 x122 x 1 1 x1 2
6 7
6 1 x2 1 x2 2 x221 x222 x 2 1 x2 2 7
6 7
6 ··· 7
6 7
6 1 xi 1 xi 2 xi21 xi22 x i 1 xi 2 7
6 7
4 ··· 5
1 x n 1 xn 2 xn21 xn22 xn 1 xn 2
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 7 / 22
The non-linear case
Expanded input matrix
2 3
1 x1 1 x1 2 x121 x122 x 1 1 x1 2
6 7
6 1 x2 1 x2 2 x221 x222 x 2 1 x2 2 7
6 7
6 ··· 7
6 7
6 1 xi 1 xi 2 xi21 xi22 x i 1 xi 2 7
6 7
4 ··· 5
1 x n 1 xn 2 xn21 xn22 xn 1 xn 2
New columns are computed
and filled in from original
inputs
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 7 / 22
Exponential parameter blow-up
Cubic derived features
Jesic
xi31 , xi32 , xi33 ,
xi21 xi2 , xi21 xi3 ,
xi22 xi1 , xi22 xi3 ,
xi23 xi1 , xi23 xi2 ,
x i 1 xi 2 x i 3 ,
7
xi21 , xi22 , xi23 ,
Quadrate
x i 1 xi 2 , xi 1 x i 3 , x i 2 xi 3 ,
x i 1 , x i 2 , xi 3 . - Linear
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 8 / 22
Higher degree polynomials
How complex a polynomial
should we try?
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 9 / 22
Higher degree polynomials
How complex a polynomial
should we try?
Aim for degree that
minimizes SSE
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 9 / 22
Higher degree polynomials
How complex a polynomial
should we try?
Aim for degree that
minimizes SSE
As degree increases,
features explode
exponentially
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 9 / 22
Overfitting
Need to be careful about
adding higher degree terms
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 10 / 22
Overfitting
Need to be careful about
adding higher degree terms
For n training points,can
always fit polynomial of
degree (n 1) exactly
However, such a curve
would not generalize well to
new data points
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 10 / 22
Overfitting
Need to be careful about
adding higher degree terms
For n training points,can
always fit polynomial of
degree (n 1) exactly
However, such a curve
would not generalize well to
new data points
Overfitting — model fits
training data well, performs
poorly on unseen data
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 10 / 22
Regularization
Need to trade o↵ SSE
against curve complexity
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 11 / 22
Regularization
Need to trade o↵ SSE
against curve complexity
So far, the only cost has
been SSE
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 11 / 22
Regularization
Need to trade o↵ SSE
against curve complexity
So far, the only cost has
been SSE
Add a cost related to
parameters (✓0 , ✓1 , . . . , ✓k )
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 11 / 22
Regularization
Need to trade o↵ SSE
against curve complexity
So far, the only cost has
been SSE
Add a cost related to
parameters (✓0 , ✓1 , . . . , ✓k )
Minimize, for instance
n k
1X X
(zi yi )2 + ✓j2
2
i=1 j=1
5-
SSE
coefficients
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 11 / 22
Regularization
Need to trade o↵ SSE
against curve complexity
So far, the only cost has
been SSE
Add a cost related to
parameters (✓0 , ✓1 , . . . , ✓k )
Minimize, for instance
n k
1X X
(zi yi )2 + ✓j2
2
i=1 j=1
Second term penalizes curve
complexity
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 11 / 22
Regularization
Variations on regularization
Change the contribution of coefficients
to the loss function
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 12 / 22
Regularization
Variations on regularization
Change the contribution of coefficients
to the loss function
Ridge regression:
k
X
Coefficients contribute ✓j2
j=1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 12 / 22
Regularization
Variations on regularization
Change the contribution of coefficients
to the loss function
Ridge regression:
k
X
Coefficients contribute ✓j2
j=1
LASSO regression:
k
X
Coefficients contribute |✓j |
j=1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 12 / 22
Regularization
Variations on regularization
Change the contribution of coefficients
to the loss function
Ridge regression:
k
X
Coefficients contribute ✓j2
j=1
LASSO regression:
k
X
Coefficients contribute |✓j |
j=1
Elastic net regression:
k
X
2
Coefficients contribute 1 |✓j | + 2 ✓j
j=1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 12 / 22
The non-polynomial case
Percentage of urban
population as a function of
per capita GDP
Not clear what polynomial
would be reasonable
⑧
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 13 / 22
The non-polynomial case
Percentage of urban
population as a function of
⑨
per capita GDP
Not clear what polynomial
would be reasonable
Take log of GDP
Regression we are
computing is
y = ✓0 + ✓1 log x1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 13 / 22
The non-polynomial case
Reverse the relationship
Plot per capita GDP in
terms of percentage of
urbanization
Now we take log of the
output variable
log y = ✓0 + ✓1 x1
Log-linear transformation
Earlier was linear-log
Can also use log-log
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 14 / 22
Regression for classification
Regression line
Set a threshold
Classifier
Output below threshold : 0 (No)
Output above threshold : 1 (Yes)
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 15 / 22
Regression for classification
classific
Regression line ↓
PhHed
Set a threshold
line
Classifier
Output below threshold : 0 (No) Treshold
Output above threshold : 1 (Yes)
Classifier output is a step function
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 15 / 22
Smoothen the step
2
2 1,2
-
-
-> 0
Sigmoid function
(z) =
1 27-
8,1 -
0
1+e z
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 16 / 22
Smoothen the step
Sigmoid function
1
(z) = z
1+e
Input z is output of our
regression
1
(z) =
1 + e 0 1 x1 +···+✓k xk )
(✓ +✓
-
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 16 / 22
Smoothen the step
Sigmoid function
j
1
(z) = z
1+e
Input z is output of our
regression
1
(z) =
1 + e 0 1 x1 +···+✓k xk )
(✓ +✓
Adjust parameters to fix
horizontal position and steepness
of step
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 16 / 22
Logistic regression
Compute the coefficients?
Solve by gradient descent
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 17 / 22
Logistic regression
Compute the coefficients?
Solve by gradient descent
Need derivatives to exist
Hence smooth sigmoid, not
step function
Check that
0
(z) = (z)(1 (z))
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 17 / 22
Logistic regression
Compute the coefficients?
Solve by gradient descent
Need derivatives to exist
Hence smooth sigmoid, not
step function
Check that
0
(z) = (z)(1 (z))
Need a cost function to minimize
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 17 / 22
MSE for logistic regression and gradient descent
Suppose we take mean squared error as the loss function.
n
1X
C= (yi (zi ))2 , where zi = ✓0 + ✓1 xi1 + ✓2 xi2
n
Lacto
i=1
↳preducted
class
class
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 18 / 22
MSE for logistic regression and gradient descent
Suppose we take mean squared error as the loss function.
n
1X
C= (yi (zi ))2 , where zi = ✓0 + ✓1 xi1 + ✓2 xi2
n
i=1
@C @C @C
For gradient descent, we compute , ,
@✓1 @✓2 @✓0
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 18 / 22
MSE for logistic regression and gradient descent
Suppose we take mean squared error as the loss function.
1X
n S
C= (yi (zi ))2 , where zi = ✓0 + ✓1 xi1 + ✓2 xi2
n
i=1
@C @C @C
For gradient descent, we compute , ,
@✓1 @✓2 @✓0
Consider two inputs x = (x1 , x2 )
For j = 1, 2,
O
n
@C 2X @ (zi )
= (yi (zi )) ·
@✓j n @✓j
i=1
-1.80
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 18 / 22
MSE for logistic regression and gradient descent
Suppose we take mean squared error as the loss function.
n
1X
C= (yi (zi ))2 , where zi = ✓0 + ✓1 xi1 + ✓2 xi2
n
i=1
@C @C @C
For gradient descent, we compute , ,
@✓1 @✓2 @✓0
↓v'(z)
Consider two inputs x = (x1 , x2 )
For j = 1, 2,
n n
@C 2X @ (zi ) 2 X @ (zi ) @zi
= (yi (zi )) · = ( (zi ) yi )
@✓j n @✓j n @zi @✓j
=
i=1 i=1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 18 / 22
MSE for logistic regression and gradient descent
Suppose we take mean squared error as the loss function.
n
1X
C= (yi (zi ))2 , where zi = ✓0 + ✓1 xi1 + ✓2 xi2
n
i=1
@C @C @C
For gradient descent, we compute , ,
@✓1 @✓2 @✓0
Consider two inputs x = (x1 , x2 )
For j = 1, 2,
n n
@C 2X @ (zi ) 2X @ (zi ) @zi
= (yi (zi )) · = ( (zi ) yi )
@✓j n @✓j n @zi @✓j
i=1 i=1
n
X
2
= ( (zi ) yi ) 0 (zi )xij
n
i=1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 18 / 22
MSE for logistic regression and gradient descent
Suppose we take mean squared error as the loss function.
n
1X
C= (yi (zi ))2 , where zi = ✓0 + ✓1 xi1 + ✓2 xi2
n
i=1
@C @C @C
For gradient descent, we compute , ,
@✓1 @✓2 @✓0
Consider two inputs x = (x1 , x2 )
For j = 1, 2,
n n
@C 2X @ (zi ) 2X @ (zi ) @zi
= (yi (zi )) · = ( (zi ) yi )
@✓j n @✓j n @zi @✓j
i=1 i=1
n
X
2
= ( (zi ) yi ) 0 (zi )xij
n
i=1
n n
@C 2X @ (zi ) @zi 2X
= ( (zi ) yi ) = ( (zi ) yi ) 0 (zi )
@✓0 n @zi @✓0 n
i=1 i=1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 18 / 22
MSE for logistic regression and gradient descent . . .
n n
@C 2X @C 2X
For j = 1, 2, = ( (zi ) yi ) 0 (zi )xji , and = ( (zi ) yi ) 0 (zi )
@✓j n @✓0 n
i=1 i=1
@C @C @C 0 (z
Each term in , , is proportional to i)
@✓1 @✓2 @✓0
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 19 / 22
MSE for logistic regression and gradient descent . . .
n n
@C 2X @C 2X
For j = 1, 2, = ( (zi ) yi ) 0 (zi )xji , and = ( (zi ) yi ) 0 (zi )
@✓j n @✓0 n
i=1 i=1
@C @C @C 0 (z
Each term in , , is proportional to i)
@✓1 @✓2 @✓0
Ideally, gradient descent should take large steps when (z) y is large
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 19 / 22
MSE for logistic regression and gradient descent . . .
n n
@C 2X @C 2X
For j = 1, 2, = ( (zi ) yi ) 0 (zi )xji , and = ( (zi ) yi ) 0 (zi )
@✓j n @✓0 n
i=1 i=1
@C @C @C 0 (z
Each term in , , is proportional to i)
@✓1 @✓2 @✓0
Ideally, gradient descent should take large steps when (z) y is large
(z) is flat at both extremes
If (z) is completely wrong,
(z) ⇡ (1 y ), we still have
0 (z) ⇡ 0
Learning is slow even when current
model is far from optimal
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 19 / 22
Loss function for logistic regression
Goal is to maximize log likelihood
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 20 / 22
Loss function for logistic regression
Goal is to maximize log likelihood
Let h✓ (xi ) = (zi ).
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 20 / 22
Loss function for logistic regression
Goal is to maximize log likelihood
Let h✓ (xi ) = (zi ). So, P(yi = 1 | xi ; ✓) = h✓ (xi ),
P(yi = 0 | xi ; ✓) = 1 h✓ (xi )
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 20 / 22
Loss function for logistic regression
Goal is to maximize log likelihood
Let h✓ (xi ) = (zi ). So, P(yi = 1 | xi ; ✓) = h✓ (xi ),
P(yi = 0 | xi ; ✓) = 1 h✓ (xi )
&
y1
=
-
Combine as P(yi | xi ; ✓) = h✓ (xi )yi · (1 h✓ (xi ))1 yi
-
↑
y =
0
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 20 / 22
Loss function for logistic regression
Goal is to maximize log likelihood
Let h✓ (xi ) = (zi ). So, P(yi = 1 | xi ; ✓) = h✓ (xi ),
P(yi = 0 | xi ; ✓) = 1 h✓ (xi )
Combine as P(yi | xi ; ✓) = h✓ (xi )yi · (1 h✓ (xi ))1 yi
n
Y
Likelihood: L(✓) = h✓ (xi )yi · (1 h✓ (xi ))1 yi
i=1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 20 / 22
Loss function for logistic regression
Goal is to maximize log likelihood
Let h✓ (xi ) = (zi ). So, P(yi = 1 | xi ; ✓) = h✓ (xi ),
P(yi = 0 | xi ; ✓) = 1 h✓ (xi )
Combine as P(yi | xi ; ✓) = h✓ (xi )yi · (1 h✓ (xi ))1 yi
U(G) log(π -)
n
Y -
h✓ (xi )yi · (1 h✓ (xi ))1 yi
-
Likelihood: L(✓) =
i=1
n
X
=
Elg( -
.)
Log-likelihood: `(✓) = yi log h✓ (xi ) + (1 yi ) log(1 h✓ (xi )) =
i=1
*
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 20 / 22
Loss function for logistic regression
#
Goal is to maximize log likelihood
Let h✓ (xi ) = (zi ). So, P(yi = 1 | xi ; ✓) = h✓ (xi ), vz
P(yi = 0 | xi ; ✓) = 1 h✓ (xi )
Combine as P(yi | xi ; ✓) = h✓ (xi )yi · (1 h✓ (xi ))1 yi
n
Y
Likelihood: L(✓) = h✓ (xi )yi · (1 h✓ (xi ))1 yi
i=1
n
X
Log-likelihood: `(✓) = yi log h✓ (xi ) + (1 yi ) log(1 h✓ (xi ))
-
i=1
n
X
Minimize cross entropy: yi log h✓ (xi ) + (1 yi ) log(1 h✓ (xi ))
i=1
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 20 / 22
Cross entropy and gradient descent
C= [y ln( (z)) + (1 y ) ln(1 (z))]
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 21 / 22
Cross entropy and gradient descent
C= [y ln( (z)) + (1 y ) ln(1 (z))]
@C @C @
=
@✓j @ @✓j
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 21 / 22
Cross entropy and gradient descent
↓0
C= [y ln( (z)) + (1 y ) ln(1 (z))]
@C @C @ y 1 y @
= =
@✓j @ @✓j (z) 1 (z) @✓j
-
~
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 21 / 22
Cross entropy and gradient descent
C= [y ln( (z)) + (1 y ) ln(1 (z))]
@C @C @ y 1 y @
= =
@✓j @ @✓j (z) 1 (z) @✓j
y 1 y @ @z
=
(z) 1 (z) @z @✓j
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 21 / 22
Cross entropy and gradient descent
C= [y ln( (z)) + (1 y ) ln(1 (z))]
@C @C @ y 1 y @
= =
@✓j @ @✓j (z) 1 (z) @✓j
y 1 y @ @z
=
(z) 1 (z) @z @✓j
y 1 y 0
= (z)xj
(z) 1 (z)
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 21 / 22
Cross entropy and gradient descent
C= [y ln( (z)) + (1 y ) ln(1 (z))]
@C @C @ y 1 y @
= =
@✓j @ @✓j (z) 1 (z) @✓j
y 1 y @ @z
=
(z) 1 (z) @z @✓j
y 1 y 0
= (z)xj
(z) 1 (z)
y (1 (z)) (1 y ) (z) 0
= (z)xj
(z)(1 (z))
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 21 / 22
Cross entropy and gradient descent . . .
@C y (1 (z)) (1 y ) (z) 0
= (z)xj
@✓j (z)(1 (z))
- -
Recall that 0 (z) = (z)(1 (z))
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 22 / 22
Cross entropy and gradient descent . . .
@C y (1 (z)) (1 y ) (z) 0
= (z)xj
@✓j (z)(1 (z))
Recall that 0 (z) = (z)(1 (z))
@C
Therefore, = [y (1 (z)) (1 y ) (z)]xj
@✓j
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 22 / 22
Cross entropy and gradient descent . . .
@C y (1 (z)) (1 y ) (z) 0
= (z)xj
@✓j (z)(1 (z))
Recall that 0 (z) = (z)(1 (z))
@C
Therefore, = [y (1 (z)) (1 y ) (z)]xj
@✓j
= [y y (z) (z) + y (z)]xj
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 22 / 22
Cross entropy and gradient descent . . .
@C y (1 (z)) (1 y ) (z) 0
= (z)xj
@✓j (z)(1 (z))
Recall that 0 (z) = (z)(1 (z))
@C
Therefore, = [y (1 (z)) (1 y ) (z)]xj
@✓j
= [y y/ (z) /(z) + y (z)]xj
= ( (z) y )xj
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 22 / 22
Cross entropy and gradient descent . . .
@C y (1 (z)) (1 y ) (z) 0
= (z)xj
@✓j (z)(1 (z))
Recall that 0 (z) = (z)(1 (z))
@C
Therefore, = [y (1 (z)) (1 y ) (z)]xj
@✓j
= [y y (z) (z) + y (z)]xj
= ( (z) y )xj
@C
Similarly, = ( (z) y)
@✓0
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 22 / 22
Cross entropy and gradient descent . . .
@C y (1 (z)) (1 y ) (z) 0
= (z)xj
@✓j (z)(1 (z))
Recall that 0 (z) = (z)(1 (z))
@C
Therefore, = [y (1 (z)) (1 y ) (z)]xj
@✓j
= [y y (z) (z) + y (z)]xj
= ( (z) y )xj
@C
Similarly, = ( (z) y)
@✓0
Thus, as we wanted, the gradient is proportional to (z) y
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 22 / 22
Cross entropy and gradient descent . . .
@C y (1 (z)) (1 y ) (z) 0
= (z)xj
@✓j (z)(1 (z))
Recall that 0 (z) = (z)(1 (z))
@C
Therefore, = [y (1 (z)) (1 y ) (z)]xj
@✓j
= [y y (z) (z) + y (z)]xj
= ( (z) y )xj
@C
Similarly, = ( (z) y)
@✓0
Thus, as we wanted, the gradient is proportional to (z) y
The greater the error, the faster the learning rate
Madhavan Mukund Lecture 8: 2 February, 2023 DMML Jan–Apr 2023 22 / 22
