Confusion Matrix in Machine Learning
The confusion matrix is a matrix used to determine the performance of
the classification models for a given set of test data. It can only be
determined if the true values for test data are known. The matrix itself
can be easily understood, but the related terminologies may be confusing.
Since it shows the errors in the model performance in the form of a
matrix, hence also known as an error matrix. Some features of Confusion
matrix are given below:
o For the 2 prediction classes of classifiers, the matrix is of 2*2 table,
for 3 classes, it is 3*3 table, and so on.
o The matrix is divided into two dimensions, that are predicted
values and actual values along with the total number of
predictions.
o Predicted values are those values, which are predicted by the
model, and actual values are the true values for the given
observations.
o It looks like the below table:
The above table has the following cases:
o True Negative: Model has given prediction No, and the real or
actual value was also No.
o True Positive: The model has predicted yes, and the actual value
was also true.
o False Negative: The model has predicted no, but the actual value
was Yes, it is also called as Type-II error.
o False Positive: The model has predicted Yes, but the actual value
was No. It is also called a Type-I error.
Need for Confusion Matrix in Machine learning
o It evaluates the performance of the classification models, when they
make predictions on test data, and tells how good our classification
model is.
o It not only tells the error made by the classifiers but also the type of
errors such as it is either type-I or type-II error.
o With the help of the confusion matrix, we can calculate the different
parameters for the model, such as accuracy, precision, etc.
Example: We can understand the confusion matrix using an example.
Suppose we are trying to create a model that can predict the result for the
disease that is either a person has that disease or not. So, the confusion
matrix for this is given as:
From the above example, we can conclude that:
o The table is given for the two-class classifier, which has two
predictions "Yes" and "NO." Here, Yes defines that patient has the
disease, and No defines that patient does not has that disease.
o The classifier has made a total of 100 predictions. Out of 100
predictions, 89 are true predictions, and 11 are incorrect
predictions.
o The model has given prediction "yes" for 32 times, and "No" for 68
times. Whereas the actual "Yes" was 27, and actual "No" was 73
times.
Calculations using Confusion Matrix:
We can perform various calculations for the model, such as the model's
accuracy, using this matrix. These calculations are given below:
o Classification Accuracy: It is one of the important parameters to
determine the accuracy of the classification problems. It defines
how often the model predicts the correct output. It can be
calculated as the ratio of the number of correct predictions made by
the classifier to all number of predictions made by the classifiers.
The formula is given below:
o Misclassification rate: It is also termed as Error rate, and it
defines how often the model gives the wrong predictions. The value
of error rate can be calculated as the number of incorrect
predictions to all number of the predictions made by the classifier.
The formula is given below:
o Precision: It can be defined as the number of correct outputs
provided by the model or out of all positive classes that have
predicted correctly by the model, how many of them were actually
true. It can be calculated using the below formula:
o Recall: It is defined as the out of total positive classes, how our
model predicted correctly. The recall must be as high as possible.
o F-measure: If two models have low precision and high recall or vice
versa, it is difficult to compare these models. So, for this purpose,
we can use F-score. This score helps us to evaluate the recall and
precision at the same time. The F-score is maximum if the recall is
equal to the precision. It can be calculated using the below formula:
Other important terms used in Confusion Matrix:
o Null Error rate: It defines how often our model would be incorrect
if it always predicted the majority class. As per the accuracy
paradox, it is said that "the best classifier has a higher error rate
than the null error rate."
o ROC Curve: The ROC is a graph displaying a classifier's
performance for all possible thresholds. The graph is plotted
between the true positive rate (on the Y-axis) and the false Positive
rate (on the x-axis).
Confusion Matrix Metrics
Figure 3: Confusion Matrix for a classifier
Consider a confusion matrix made for a classifier that classifies people based on
whether they speak English or Spanish.
From the above diagram, we can see that:
True Positives (TP) = 86
True Negatives (TN) = 79
False Positives (FP) = 12
False Negatives (FN) = 10
Accuracy: The accuracy is used to find the portion of correctly classified values. It tells us
how often our classifier is right. It is the sum of all true values divided by total values.
Figure 4: Accuracy
In this case:
Accuracy = (86 +79) / (86 + 79 + 12 + 10) = 0.8823 = 88.23%
Precision: Precision is used to calculate the model's ability to classify positive values
correctly. It is the true positives divided by the total number of predicted positive values.
Figure 5: Precision
In this case,
Precision = 86 / (86 + 12) = 0.8775 = 87.75%
Recall: It is used to calculate the model's ability to predict positive values. "How often
does the model predict the correct positive values?". It is the true positives divided by the
total number of actual positive values.
Figure 6: Recall
In this case,
Recall = 86 / (86 + 10) = 0.8983 = 89.83%
F1-Score: It is the harmonic mean of Recall and Precision. It is useful when you need to
take both Precision and Recall into account.
Figure 7: F1-Score
In this case,
F1-Score = (2* 0.8775 * 0.8983) / (0.8775 + 0.8983) = 0.8877 = 88.77%
Scaling a Confusion Matrix
To scale a confusion matrix, increase the number of rows and columns. All the True
Positives will be along the diagonal. The other values will be False Positives or False
Negatives.
Figure 12: Scaling down our dataset
Now that we understand what a confusion matrix is and its inner working, let's
explore how we find the accuracy of a model with a hands-on demo on confusion
matrix with Python.
Confusion Matrix With Python
We'll build a logistic regression model using a heart attack dataset to predict if a
patient is at risk of a heart attack.
Depicted below is the dataset that we'll be using for this demonstration.
Figure 9: Heart Attack Dataset
Let’s import the necessary libraries to create our model.
Figure 10: Importing Confusion Matrix in python
We can import the confusion matrix function from sklearn.metrics. Let’s split our
dataset into the input features and target output dataset.
Figure 11: Splitting data into variables and target dataset
As we can see, our data contains a massive range of values, some are single digits,
and some have three numbers. To make our calculations more straightforward, we
will scale our data and reduce it to a small range of values using the Standard
Scaler.
Figure 12: Scaling down our dataset
Now, let's split our dataset into two: one to train our model and another to test our
model. To do this, we use train_test_split imported from sklearn. Using a Logistic
Regression Model, we will perform Classification on our train data and predict our
test data to check the accuracy.
Figure 13: Performing classification
To find the accuracy of a confusion matrix and all other metrics, we can import
accuracy_score and classification_report from the same library.
Figure 14: Accuracy of classifier
The accuracy_score gives us the accuracy of our classifier
Figure 15: Confusion Matrix for data
Using the predicted values(pred) and our actual values(y_test), we can create a
confusion matrix with the confusion_matrix function.
Then, using the ravel() method of our confusion_matrix function, we can get the True
Positive, True Negative, False Positive, and False Negative values.
Figure 16: Extracting matrix value
Figure 17: Confusion Matrix Metrics
Finally, using the classification_report, we can find the values of various metrics of
our confusion matrix.
