Dynamic Programming Matrix Chain Multiplication

Dynamic
Dynamic Programming:
Matrix Chain Products
By
Krishnakoumar C
M.Tech DS (I Year)
Puducherry Technological University

What is Dynamic Programming?
• Dynamic Programming refers to simplifying a complicated problem by
breaking it down into simpler sub-problems in a recursive manner.
• Dynamic Programming techniques:
• Simple Subproblems
• Subproblems Optimality
• Subproblem Overlap

Matrix Chain Product
• Matrix chain multiplication is an optimization problem concerning the
most efficient way to multiply a given sequence of matrices
• The problem is not actually to perform the multiplications, but merely
to decide the sequence of the matrix multiplications involved.
• There are many options because matrix multiplication is associative.
In other words, no matter how the product is parenthesized, the
result obtained will remain the same.

Consider 2x3 matrices A, B, and C:
• We, can perform matrix Multiplication in the following ways:
i) (A x B) x C
ii) A x (B x C)
• Note: Matrix Multiplication is Associative.

Matrix Multiplication
• Let us consider two matrices A and B:
𝐴 =
𝑎11 𝑎12 𝑎13
𝑎21 𝑎22 𝑎23
𝐵 =
𝑏11 𝑏12
𝑏21 𝑏22
𝑏31 𝑏32
2 x 3 3 x 2

• Let us consider two matrices A and B of dimensions 2 x 3:
𝐴 =
𝑎11 𝑎12 𝑎13
𝑎21 𝑎22 𝑎23
𝐵 =
𝑏11 𝑏12
𝑏21 𝑏22
𝑏31 𝑏32
2 x 3 3 x 2
Condition for Matrix Multiplication

𝐴 =
𝑎11 𝑎12 𝑎13
𝑎21 𝑎22 𝑎23
𝐵 =
𝑏11 𝑏12
𝑏21 𝑏22
𝑏31 𝑏32
• So, the product of two matrices A and B is given by:
𝑎11𝑏11 + 𝑎12𝑏21 + 𝑎13𝑏31 𝑎11𝑏12 + 𝑎12𝑏22 + 𝑎13𝑏32
𝑎21𝑏11 + 𝑎22𝑏21 + 𝑎23𝑏31 𝑎21𝑏12 + 𝑎22𝑏22 + 𝑎23𝑏32

𝐴 =
𝑎11 𝑎12 𝑎13
𝑎21 𝑎22 𝑎23
𝐵 =
𝑏11 𝑏12
𝑏21 𝑏22
𝑏31 𝑏32
• So, the product of two matrices A and B is given by:
1 2 3 4 5 6
𝑎11𝑏11 + 𝑎12𝑏21 + 𝑎13𝑏31 𝑎11𝑏12 + 𝑎12𝑏22 + 𝑎13𝑏32
𝑎21𝑏11 + 𝑎22𝑏21 + 𝑎23𝑏31 𝑎21𝑏12 + 𝑎22𝑏22 + 𝑎23𝑏32
7 8 9 10 11 12

• To calculate the number of multiplication operations required for a
given matrix, we can use the below technique:
𝐴 =
𝑎11 𝑎12 𝑎13
𝑎21 𝑎22 𝑎23
𝐵 =
𝑏11 𝑏12
𝑏21 𝑏22
𝑏31 𝑏32
2 x 3 3 x 2
Total Number of Multiplication required = 2 x 3 x 2 = 12 operations

Parenthesization
• Let us consider an example:
A1 x A2 x A3
2 3 3 4 4 2
d0 d1 d1 d2 d2 d3

Parenthesization
• Parenthesization of given matrices:
A1 x A2 x A3
2 3 3 4 4 2
d0 d1 d1 d2 d2 d3
(A1 x A2) x A3 A1 x (A2 x A3)

Parenthesization
(A1 x A2) x A3
2 3 3 4 4 2
Number of Operations: C[1, 2] = 2 x 3 x 4 = 24 C[3, 3] = 0
New Dimension: 2 x 4 4 x 2
Number of Operations: 2 x 4 x 2 = 16
Total Number of operations: 24 + 16 = 40 Multiplications
General Notation for this operation: C[1, 2] + C[3, 3] + d0 + d2 + d3 = 40 ------------------------------ (i)

Paranthesization
A1 x (A2 x A3)
2 3 3 4 4 2
Number of Operations: C[1, 1] = 0 C[2, 3] = 3 x 4 x 2 = 24
New Dimension: 2 x 3 3 x 2
Number of Operations: 2 x 3 x 2 = 12
Total Number of operations: 24 + 12 = 38 Multiplications
General Notation for this operation: C[1, 1] + C[2, 3] + d0 + d1 + d3 = 38 ------------------------------ (ii)

Parenthesization
• From Equation (i) and (ii):
C[1, 2] + C[3, 3] + d0 + d2 + d3 = 40 C[1, 1] + C[2, 3] + d0 + d1 + d3 = 38
>

Parenthesization
• From Equation (i) and (ii):
C[1, 2] + C[3, 3] + d0 + d2 + d3 = 40 C[1, 1] + C[2, 3] + d0 + d1 + d3 = 38
>
This is the optimal solution among the two ways

Parenthesization
• From Equation (i) and (ii), we can derive the generalized form
C[1, 2] + C[3, 3] + d0 + d2 + d3 = 40 C[1, 1] + C[2, 3] + d0 + d1 + d3 = 38
>
C[i, j] = min { C[i, k] + C[k + 1, j] + di-1 x dk x dj}
I ≤ k < j

Parenthization
• The possible number of ways the given matrices can be parenthized
using the below formula:
𝐶(𝑛 −1)
2(𝑛 −1)
𝑛
Where:
n – Number of matrices for multiplication

Example Problem:
• Let us consider four matrices namely A1, A2, A3, and A4 of dimensions d0, d1,
d2, d3, and d4 respectively. Find the optimal matrix multiplication.
A1 x A2 x A3 x A4
d0 d1 d1 d2 d2 d3 d3 d4
3 2 2 4 4 2 2 5
The number of possible ways the matrices can be parenthesized:
Here, n = 4
𝐶(𝑛 −1)
2(𝑛 −1)
𝑛 =
𝐶(4 −1)
2(4 −1)
4 =
𝐶3
6
4 =
6 𝑋 5 𝑋 4
3 𝑋 2 𝑋 1
4 = 5. Hence there are 5 five possible ways of parenthesization.

Example Problem
• The following are 5 different ways of parenthesization:
1. A1 X (A2 X (A3 X A4))
2. A1 X ((A2 X A3) X A4)
3. (A1 X A2) X (A3 X A4)
4. (A1 X (A2 X A3)) X A4
5. ((A1 X A2) X A3) X A4

Example Problem
• Let us consider 2 tables to arrange the resultant cost and k-values.
1 2 3 4 1 2 3 4
00
0
0
0
1
2
3
4
1
2
3
4
i
j
k
C
i
j

Example Problem
• Let us find the C[1, 2]: C[i, j] = min { C[i, k] + C[k + 1, j] + di-1 x dk x dj}
I ≤ k < j
C[1, 2] = min { C[1, 1] + C[2, 2] + d0 x d1 x d2}
1 ≤ 1< 2
K = 1
= 0 + 0 + 3 x 2 x 4
= 24
0 24
0
0
0
1
Cost K value
1 2 3 4 1 2 3 4
1
2
3
4
1
2
3
4

Example Problem
I ≤ k < j
C[2, 3] = min { C[2, 2] + C[3, 3] + d1 x d2 x d3}
2 ≤ 2< 3
K = 2
= 0 + 0 + 2 x 4 x 2
= 16
0 24
0 16
0
0
1
2
Cost K value
1 2 3 4 1 2 3 4
1
2
3
4
1
2
3
4

Example Problem
I ≤ k < j
C[3, 4] = min { C[3, 3] + C[4, 4] + d2 x d3 x d4}
3 ≤ 3< 4
K = 3
= 0 + 0 + 4 x 2 x 5
= 40
0 24
0 16
0 40
0
1
2
3
Cost K value
1 2 3 4 1 2 3 4
1
2
3
4
1
2
3
4

Example Problem
I ≤ k < j
C[1, 3] = min C[1, 1] + C[2, 3] + d0 x d1 x d3 = 0 + 16 + 3 x 2 x 2 = 28
C[1, 2] + C[3, 3] + d0 x d2 x d3 = 24 + 0 + 3 x 4 x 2 = 48 When, K = 2
0 24 28
0 16
0 40
0
1 1
2
3
Cost K value
1 2 3 4 1 2 3 4
1
2
3
4
1
2
3
4
When, K = 1
1 ≤ k < 3

Example Problem
I ≤ k < j
C[2, 4] = min C[2, 2] + C[3, 4] + d1 x d2 x d4 = 0 + 40 + 2 x 4 x 5 = 80
C[2, 3] + C[4, 4] + d1 x d3 x d4 = 16 + 0 + 2 x 2 x 5 = 36 When, K = 3
0 24 28
0 16 36
0 40
0
1 1
2 3
3
Cost K value
1 2 3 4 1 2 3 4
1
2
3
4
1
2
3
4
When, K = 2
2 ≤ k < 4

Example Problem
I ≤ k < j
C[1, 4] = min C[1, 1] + C[2, 4] + d0 x d1 x d4 = 0 + 36 + 3 x 2 x 5 =66
C[1, 2] + C[3, 4] + d0 x d2 x d4 = 24 + 40 + 3 x 4 x 5 = 124
C[1, 3] + C[4, 4] + d0 x d3 x d4 = 28 + 0 + 3 x 2 x 5 = 58
When, K = 2
0 24 28 58
0 16 36
0 40
0
1 1 3
2 3
3
Cost K value
1 2 3 4 1 2 3 4
1
2
3
4
1
2
3
4
When, K = 1
When, K = 3
1 ≤ k < 4

Example Problem
0 24 28 58
0 16 36
0 40
0
1 1 3
2 3
3
Cost K value
1 2 3 4 1 2 3 4
1
2
3
4
1
2
3
4
• From the k-value table, we can parenthesize the given matrices:
A1 x A2 x A3 x A4
(A1 x A2 x A3) x A4
(A1 x (A2 x A3)) x A4
((A1) x (A2 x A3)) x A4
A1 A2 A3 A4

Effectiveness of Parenthesization
• Example:
B is 3 x 100
C is 100 x 5
D is 5 x 5
(B x C) x D takes 1500 + 75 = 1575 operations
B x (C x D) takes 1500 + 2500 = 4000 operations

Naïve Approach
• In naïve approach, we try the brute-force method of finding all
possible parenthesization methods and then to choose the optimal
method.
• The number of paranthesizations will be equal to number of binary
trees with n nodes.
• This is Exponential in time
• This is called the Catalan number, and it is almost 4n

Greedy Approach - 1
• Repeatedly select the product that uses the most operations.
A is 10 x 5
B is 5 x 10
C is 10 x 5
D is 5 x 10
(A x B) x (C x D) takes 500 + 1000 + 500 = 2000 operations
A x ((B x C) x D) takes 500 + 250 + 250 = 1000 operations

Greedy Approach - 2
• Repeatedly select the product that uses the fewest operations.
A is 101 x 11
B is 11 x 9
C is 9 x 100
D is 100 x 99
A x ((B x C) x D)) takes 109989 + 9900 + 108900 = 228789 operations.
(A x B) x (C x D) takes 9999 + 89991 + 89100 = 189090 operations

Dynamic Programming Algorithm: Matrix-
Chain Multiplication
Algorithm matrixChain(S):
Input: sequence S of n matrices to be multiplied
Output: number of operations in an optimal parentheization of S
for i <- 1 to n-1 do
Ni,i <- 0
for b <- 1 to n-1 do
for i <- 0 to n-b-1 do
j <- i + b
Ni,j <- +infinity
for k <- i to j-1 do
Ni,j <- min {Ni,j, Ni,k + Nk+1,j + di x dk+1 x dj+1}

Dynamic Programming Algorithms:
• The subproblems here are not independent, the subproblems
overlap.
• Since subproblems overlaps, we don’t use recursion.
• Instead we construct optimal subproblems “bottom-up”.
• The running time is O(n3).

Reference
• Matrix Chain Multiplication - Dynamic Programming – Abdul
Bari. https://youtu.be/prx1psByp7U?si=bvn_xBi78xEl3HQI
• Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction to
Algorithms, Third Edition (3rd. ed.). The MIT Press.
• Steven S. Skiena. 2008. The Algorithm Design Manual (2nd. ed.). Springer Publishing Company,
Incorporated.

Dynamic Programming Matrix Chain Multiplication

More Related Content

What's hot

Similar to Dynamic Programming Matrix Chain Multiplication

Recently uploaded

In this document

Dynamic Programming Matrix Chain Multiplication