Mutation Testing: Testing your tests

z
Mutation Testing
Testing your tests to test that they test what you think
they test
Stephen Leigh
Test Manager
Sky Betting and Gaming

z
Why do we write unit tests?
- Fast feedback if code is not behaving in the expected way
- Good for debugging, reducing cost of defects
- A form of documentation
- Encourages better design principles
The most relevant reason for the purposes of mutation testing:
- To ensure that any changes we make don’t accidentally
break the functionality of the existing codebase

z
Why do we write unit tests?
“To ensure that any changes we make don’t accidentally break the
functionality of the existing codebase”
To make sure our changes don’t alter any existing functionality, we
need to be sure our unit tests are exercising this functionality
appropriately.
This is what mutation testing can help with.

z
What is mutation testing?
A mutant is a small change in an application’s source code, e.g.:
Source Code
const isPositive = num => num > 0;
Mutant Source Code
const isPositive = num => num >= 0;
Mutation test tools create a large number of these mutants,
then run unit tests against the edited code

z
Mutation examples
Code Possible mutations
if (x === 3) if (x >= 3)
if (x <= 3)
if (x !== 3)
if (true)
if (false)
if (x === 3) {
k++
}
if (x === 3) {}
if (x === 3) k--
return {token: “c38bf32”} return {}
return null
return {token: “”}

z
Killing mutants
When unit tests are run against the mutant, one of three things
happens:
- At least one unit test fails. This means the mutant has been ‘killed’
and therefore the part of the code that has been changed is
properly covered.
- All unit tests pass. This means the mutant has ‘survived’, and the
changed functionality is not covered by tests.
- Infinite loop/runtime error. This usually means that the mutation
isn’t something that could actually happen, and counts as a kill.
As you might expect, generating hundreds of mutants is very
computationally expensive. Fortunately, it’s all automatically handled
by the mutation testing framework.

Simple example
Source Code
function isPositive(num) {
if (num > 0) {
return true;
} else {
return false;
}
}
Pseudo-unit tests
testPositive() {
assertTrue(isPositive(6))
}
testNegative() {
assertFalse(isPositive(-3))
}

Simple example
Mutated Source Code
if (num >= 0) {
return true;
} else {
return false;
}
}
Pseudo-unit tests
testPositive() {
}
testNegative() {
}

Simple example
Mutated Source Code
if (num >= 0) {
return true;
} else {
return false;
}
}
Pseudo-unit tests
testPositive() {
}
testNegative() {
}
testZero() {
assertFalse(isPositive(0))
}
Mutant killed!

More complex example
Source Code
function handleLogin(request, response)
const {username, password} =
if (!username) {
return response.status(400)
.json({reason: ‘ERR_NO_USERNAME’})
}
if (!password) {
.json({reason: ‘ERR_NO_PASSWORD’})
}
}
Pseudo-unit tests
const testRequest = {
body: {} // no username or password
fields
}
const mockResponse = () => ...
testNoUsername() {
handleLogin(testRequest, mockResponse);
expect(mockResponse.calls.single.toBe(400)
);
}
testNoPassword() {
);
}

Mutated Source Code
if (!username) {
}
if (false) {
}
}
Pseudo-unit tests
fields
}
testNoUsername() {
);
}
testNoPassword() {
);
}

Mutated Source Code
if (!username) {
}
if (false) {
}
}
Pseudo-unit tests
fields
}
testNoUsername() {
expect(mockResponse.calls[0].toBe(400));
}
testNoPassword() {
const testRequestUsername = {
body: {
username: “abc”
}
};
handleLogin(testRequestUsername,
mockResponse);Mutant killed!

z
Mutation testing: advantages and disadvantages
Advantages
• More reliable metric than line coverage – actually ensures your unit tests are testing
what they should be, and that they follow all possible lines of logic (cyclomatic
complexity)
• Catches many small and easy-to-miss programming errors, as well as holes in unit
tests that would otherwise go unnoticed
Disadvantages
• Extremely computationally expensive – runs can take several hours, making it
unsuitable to run as part of a standard release process
• Requires brainpower to sort ‘junk’ mutations or unimportant survivors from useful
catches. May be an unfavourable signal-to-noise ratio. In these cases it’s potentially
a more useful process to compare over a period of time, to ensure surviving
mutations don’t start going up dramatically.

z
Suggested Tools
- JavaScript: Stryker
- Java/Kotlin: PITest
Questions?

Mutation Testing: Testing your tests

More Related Content

What's hot

Similar to Mutation Testing: Testing your tests

Recently uploaded

In this document

Mutation Testing: Testing your tests