Facial Expression
Recognition Using Deep
Learning
By: Anshika Modi (CSE)
(2003330100017)
Abstract Facial
expressions is a
way to express
• Facial expression recognition plays an important role in
emotions.
human-computer interaction.
• It is an important hotspot in CV, IOT and AI fields.
• Our model analyzes the facial expressions by performing
various steps:
i. Pre-processing
ii. Segmentation
iii. Feature extraction
iv. Classification
2023-24 Facial Expression Recognition 2
Scope and Objectives
• The main objective of this project is to recognize the facial expressions from the facial images with low
false positive rate and high accuracy.
• In addition, to increase the facial image quality, preprocessing is initialized by performing normalization to
estimate the pose and angle of the facial image and also reduces the illumination effects to increase the
accuracy of the facial expression.
To extract the features efficiently, cluster-based segmentation is performed to estimate the facial objects
such as eyebrows, eyes, nose, and mouth which reduces the loss of small expressions.
Multi-feature is extracted in the feature extraction process by considering both low-level and high-level
features of the facial images to increase the accuracy of facial expression recognition.
2023-24 Facial Expression Recognition 3
Literature Review
1. A REVIEW ON FACIAL EXPRESSION RECOGNITION
SYSTEMSS. Shaul Hammed1 , Dr. A. Sabanayagam2, E.
Ramakalaivani3 ISSN-2394-5125 Vol 7, Issue 4, 2020
2. Zhang, J., Hu, H., & Feng, S. (2020). Robust Facial Landmark
Detection via Heatmap-Offset Regression. IEEE Transactions
on Image Processing, 29, 5050-5064.
3. D. Ghimire and J. Lee, “Geometric Feature-Based Facial
ExpressionRecognition in Image Sequences Using Multi-Class
AdaBoost andSupport Vector Machines,” Sensor, vol. 13, pp.
7714–7734, 2013.
4. Chen, A., Xing, H., & Wang, F. (2020). A Facial Expression
Recognition Method Using Deep Convolutional Neural
Networks Based on Edge Computing. IEEE Access, 8, 49741-
49751.
2023-24 Facial Expression Recognition 4
Methodology: Any facial image undergoes
via three phases:
Bi-level preprocessing
Clustering-based Segmentation
We have provided Cohn Kanade
Multi-Feature Extraction and Facial Expression
(CK+) dataset to our model. Classification.
Bi-level preprocessing
Removes the contrast from input
image and do angle orientation to
produce well posed image.
Segmentation
Distinguishes various facial features
Feature Extraction
like eyebrows, mouth, nose and Various facial features are detected from
divide the image into segments and landmarked image and Capsule Neural
perform landmark detection. Network do classify the facial expression.
2023-24 Facial Expression Recognition 5
Input Grayscale Gray Scale Pose
Normalisation For Normalised Cluster Based Segmented
image image
removing Contrast Image Segmentation Image
Pose
Geometry based Polar Face
Normalised Landmark
Transformation Landmark
Image Detection
Detected
Bi-Level Preprocessing Segmentation and Landmark Detection
High Level Happy
Landmark
Block Diagram
Graph Features Sad
Feature
GNN Extraction Anger
Surprise
CNaps Neutral
Net Classification
Low Level Disgust
Features Fear
Multi Feature Extraction and Expression
Classification
2023-24 Facial Expression Recognition 6
3. In Bi-Level Processing
Implementation Phase, the selected image
first gets converted to
grayscale image and then
1. We first select 2. The selected image it gets well posed as
any image gets loaded shown:
from the CK+ successfully as shown
dataset. below:
2023-24 Facial Expression Recognition 7
4. Then this image is
segmented and various facial
objects are recognized and
thus the facial landmarks are
detected.
5. The landmark
detected image is
then classified into
low and high level
features extraction.
2023-24 Facial Expression Recognition 8
6. The classification is then
performed by Capsule and Graph
Neural Network and then it
classifies the emotion from one of
the 7 available emotions.
7. Seven important facial
expressions are classified depend
upon the extracted features. They
are neutral, happy, sad, fear,
disgust, surprise, and anger.
The classification result is: “
Sad” as shown:
2023-24 Facial Expression Recognition 9
Performance Metrics
We have evaluated the following performance metrics:
Accuracy, Precision, Recall, Facial landmark detection error, Confusion matrix.
2023-24 Facial Expression Recognition 10
Conclusion
o Deep learning covers various information, measurements which is the example of
human made consciousness.
o Facial Expression Recognition becomes a challenge in the computer vision and deep
learning.
o Researchers are improving the accuracy day by day after doing innovative changes by
using the deep learning techniques.
o Facial Expression Recognition is used for classification of input facial images into
seven different expressions such as Sad, Fear, Anger, Happy etc.
o In this research, we evaluate the performance by using the following Performance
Metrics such as Accuracy, Recall, Precision, Facial Landmark Detection Error,
Confusion Matrix.
2023-24 Facial Expression Recognition 11
References
1. Liu, C., Hirota, K., Ma, J., Jia, Z., & Dai, Y. (2021). Facial Expression Recognition Using Hybrid Features of Pixel
and Geometry. IEEE Access, 9, 18876-18889.
2. Mohan, K., Seal, A., Krejcar, O., & Yazidi, A. (2021). Facial Expression Recognition Using Local Gravitational
Force Descriptor-Based Deep Convolution Neural Networks. IEEE Transactions on Instrumentation and
Measurement, 70, 1-12.
3. Liu, J., Feng, Y., & Wang, H. (2021). Facial Expression Recognition Using Pose-Guided Face Alignment and
Discriminative Features Based on Deep Learning. IEEE Access, 9, 69267-69277.
4. Tsai, K., Tsai, Y., Lee, Y., Ding, J., & Chang, R.Y. (2021). Frontalization and adaptive exponential ensemble rule for
deep-learning-based facial expression recognition system. Signal Process. Image Commun., 96, 116321.
5. Kim, H., Kim, H., Rew, J., & Hwang, E. (2020). FLSNet: Robust Facial Landmark Semantic Segmentation. IEEE
Access, 8, 116163-116175.
2023-24 Facial Expression Recognition 12
Thankyou!
Presented By: Anshika Modi (CSE)
2003330100017
2023-24 Facial Expression Recognition 13
