MUTATION
TESTING
Mutation Testing
How to assess the effectiveness of selected
test suite?
Is this test suite adequate for the program?
Mutation Testing
If the test suite is not able to make the program fail, there
may be one of the following reasons:
Test suit is effective but hardly any errors are there in
the program.
Test suite is not effective and could not find any
errors.
Mutation Testing
Mutation and Mutants
Process of changing a program is known as mutation.
Prepare a copy of the program under test and make a change in a statement of
the program.
This changed version of the program is known as a mutant of the original
program.
Mutation Testing
Mutation and Mutants
Mutants generated by making only one change are
known as first order mutants.
Second order mutants may be obtained by making two
changes and so on……….
Mutation Testing
1 void main()
2 {
3 float A,B,C;
4 clrscr();
5 printf("Enter number 1:\n");
6 scanf("%f", &A);
7 printf("Enter number 2:\n");
8 scanf("%f", &B);
9 printf("Enter number 3:\n");
10 scanf("%f", &C); Mutated statement (‘>’
11 if(A>B) if(A=B) is replaced by ‘=‘)
12 if(A>C) {
{ {
13
printf("The largest number is: %f\n",A);
14 }
15 else {
16 printf("The largest number is: %f\n",C);
17 }
18 }
19 else {
20 if(C>B) {
21 printf("The largest number is: %f\n",C);
22 }
23 else {
24 printf("The largest number is: %f\n",B);
25 }
Mutation Testing
Mutation Operators
Mutants are produced by applying mutant operators.
An operator is essentially a grammatical rule that change a single expression to another
expression.
Higher order mutants are difficult to manage, control and trace.
To kill a mutant, we should be able to execute the changed statement of the program.
Mutation Testing
Mutation Operators for OO languages like Java,
C++:
Changing the access modifier, like public to private.
Static modifier change
Argument change
Super keyword change
Operator change
Any operand change by a numeric value
Mutation Testing
Mutation Score
The results of the program were affected by the change and any test
case detects it. If this happens, the mutant is called a killed mutant.
The results of the program are not changed and any test case of test
suit does not detect the mutation. The mutant is called a live mutant.
Mutation Testing
Mutation Score
The score associated with a test suite and its mutants is
calculated as:
Number of mutants killed
Mutation Score
Total number of mutants
Where, total number of mutants is equal to number of killed
mutant plus number of live mutants.
Mutation Testing
Mutation Score
Live mutants are important and should be analyzed
thoroughly.
New test cases should be written for live mutants and all these
mutants should be killed.
Test cases that identify the changed behavior should be
preserved and transferred to original test suite in order to
enhance the capability of the test suite.
Mutation Testing
Consider the program to find largest of three numbers. The
test suite selected by a testing technique is given as:
Test case A B C Expected output
1 6 10 2 10
2 10 6 2 10
3 6 2 10 10
4 6 10 20 20
Mutation Testing
Solution:
Mutant Line no. Original line Modified Line
No.
M1 11 if(A>B) if (A<B)
M2 11 if(A>B) if(A>(B+C))
M3 12 if(A>C) if(A<C)
M4 20 if(C>B) if(C=B)
M5 16 printf(“The Largest printf(“The Largest
number is:%f\n”,C); number is:%f\n”,B);
Mutation Testing
Actual output of mutant 1 and 2 using given test suite
Test case A B C Expected output Actual output
1 6 10 2 10 6
2 10 6 2 10 6
3 6 2 10 10 10
4 6 10 20 20 20
Test case A B C Expected output Actual
output
1 6 10 2 10 10
2 10 6 2 10 10
3 6 2 10 10 10
4 6 10 20 20 20
Mutation Testing
Actual output of mutant 3 and 4 using given test suite
Test case A B C Expected Actual
output output
1 6 10 2 10 10
2 10 6 2 10 2
3 6 2 10 10 6
4 6 10 20 20 20
Test case A B C Expected Actual
output output
1 6 10 2 10 10
2 10 6 2 10 10
3 6 2 10 10 10
4 6 10 20 20 10
Mutation Testing
Actual output of mutant 5 using given test suite
Test case A B C Expected Actual output
output
1 6 10 2 10 10
2 10 6 2 10 10
3 6 2 10 10 2
4 6 10 20 20 20
Number of mutants killed
Mutation Score
Total number of mutants
4
5
0.8
Mutation Testing
Additional test case
Test case A B C Expected
output
5 10 5 6 10
Mutation Testing
Revised Test suite
Test case A B C Expected output
1 6 10 2 10
2 10 6 2 10
3 6 2 10 10
4 6 10 20 20
5 10 5 6 10
Mutation Testing
Consider the program for classification of triangle given in
figure 3.18. The test suite A and B are selected by two different
testing techniques and are given in table 4.31 and table 4.32,
respectively. The five first order mutants and the modified lines
are given in table 4.33. Calculate the mutation score of each test
suite and compare their effectiveness. Also, add any additional
test case, if required.
Mutation Testing
Test case a b c Expected output
1 30 40 90 Invalid triangle
2 30 20 40 Obtuse angled triangle
3 50 40 60 Acute angled triangle
4 30 40 50 Right angled triangle
5 -1 50 40 Input values are out of range
6 50 150 90 Input values are out of range
7 50 40 -1 Input values are out of range
Mutation Testing
Test case a b c Expected output
1 40 90 20 Invalid triangle
2 40 30 60 Obtuse angled triangle
3 40 50 60 Acute angled triangle
4 30 40 50 Right angled triangle
5 -1 50 40 Input values are out of
range
6 30 101 90 Input values are out of
range
7 30 90 0 Input values are out of
range
Mutation Testing
Mut Line Original line Modified Line
ant no.
No.
M1 13 if(a>0&&a<=100&&b>0&&b<=100&&c> if(a>0||
0&&c<=100) { a<=100&&b>0&&b<=100&&c>0&&c<=100
){
M2 14 if((a+b)>c&&(b+c)>a&&(c+a)>b) { if((a+b)>c&&(b+c)>a&&(b+a)>b) {
M3 21 if(valid==1) { if(valid>1) {
M4 23 a2=(b*b+c*c)/(a*a); a2=(b*b+c*c)*(a*a);
M5 25 if(a1<1||a2<1||a3<1) { if(a1>1||a2<1||a3<1) {
