Mutation testing for a safer Future

Mutation Testing,
for a Safer Future
Mathieu Godart – Sêmeia

« Quis custodiet ipsos
custodes? »
« Who will guard the
guards? »
— Juvenal, ﬁrst century AD

Code coverage myth.
int sum(int a, int b) {
return a * b; // Should be a + b
}

Code coverage myth.
test_result test_sum() {
int result = sum(9, 5);
if (result > 0) {
return success;
}
return fail;
}

Code coverage myth.
test_result test_sum() {
int result = sum(9, 5);
if (result > 0) {
return success;
}
return fail;
}
• test_sum() => Success. Coverage: 100%

Code Coverage
is just a way to measure ratio of
executed code

Mutation Testing:
breaking the code and
checking that the failure is
detected by a test

Mutation Example
return a + b;
}
Becomes:
return a * b;
}

History
• 1971: Idea by Richard Lipton.
• 1981: Implementation by Timothy Budd, Yale
Univ.
• 2004: Hardware veriﬁcation by Certess
(Synopsys).

The competent programmer,
most bugs due to small syntax

Coupling effect,
faults can cascade

Some Mutation Operators
• Statement deletion
• Statement duplication or insertion, 
e.g. goto fail;
• Replacement of boolean subexpressions with true and false
• Replacement of some operations with others, 
e.g. + with *, - with /, > with >=, == and <=
• Replacement of variables with others from the same scope

Who uses this today?
Java, Dot Net… Swift?

Fuzzing
can be seen as
Mutation Testing.

Fuzzing:
using invalid, unexpected,
or random data
as input to a program

1950
taking punched cards
from the trash as
input to programs

1983
« The Monkey » by Steve
Capps:
testing Mac OS with
random user inputs

But remember to write tests!
(before you write code)
It will save you time!

References
• LLVM-Based Mutation Testing System. RFC. 
https://lowlevelbits.org/llvm-based-mutation-testing-system.-request-for-comments
• Wikipedia 
https://en.wikipedia.org/wiki/Mutation_testing 
https://en.wikipedia.org/wiki/Fuzzing
• The blog from our friends 
https://blog.xebia.com/mutation-testing-how-good-are-your-unit-tests/ 
https://blog.octo.com/mutation-testing-un-pas-de-plus-vers-la-perfection/

Thank You
Mathieu Godart – Sêmeia

Mutation testing for a safer Future

More Related Content

Similar to Mutation testing for a safer Future

More from CocoaHeads France

Recently uploaded

Mutation testing for a safer Future