Mutation Testing - Ruby Edition

Mutation Testing

Chris Sinjakli

Testing is a good thing

But how do we know our tests are
good?

Code coverage is a start

But it can give a “good” score with
really dreadful tests

Really dreadful tests
class Adder
def self.add (x, y)
return x - y
end
end

describe Adder do
it "should add the two arguments" do
Adder.add(1, 1)
end
end
Coverage: 100%
Usefulness: 0

A contrived example

But how could we detect it?

Mutation Testing!

“Who watches the watchmen?”

If you can change the code, and a
test doesn’t fail, either the code is
never run or the tests are wrong.

How?
1. Run test suite
2. Change code (mutate)
3. Run test suite again

If tests now fail, mutant dies. Otherwise it
survives.

Going with our previous example
class Adder
def self.add (x, y)
return x - y
end
end Let’s change something

describe Adder do
Adder.add(1, 1)
end
end

Going with our previous example
class Adder
def self.add (x, y)
return x + y
end
end
This still passes

describe Adder do
Adder.add(1, 1)
end
end

Success

We know something is wrong

So what? It caught a really
rubbish test
How about something slightly less
obvious?

Slightly less obvious (and I mean slightly)
class ConditionChecker
def self.check(a, b)
if a && b
return 42
else
return 0
end
end
end

describe ConditionChecker do
it "should return 42 when both arguments are true" do
ConditionChecker.check(true, true).should == 42
end
it "should return 0 when both arguments are false" do
ConditionChecker.check(false, false).should == 0
end
end Coverage: 100%
Usefulness: >0
But still wrong

if a && b
return 42
else Mutate
return 0
end
end
end

describe ConditionChecker do
end
end
end

if a || b
return 42
else
return 0
end
end
end

describe ConditionChecker do Passing tests
end
end
end

Mutation testing caught our
mistake
:D

Useful technique

But still has its flaws

The downfall of mutation
(Equivalent Mutants)
index = 0

while index != 100 do
doStuff()
index += 1
end
Mutates to
index = 0

while index < 100 do
doStuff()
index += 1
end
But the programs are equivalent, so no test will fail

There is no possible test which
can “kill” the mutant
The programs are equivalent

Also (potentially)
• Infinite loops
• More memory used
• Compile/run time errors – tools should
minimise these

How bad is it?
• Good paper assessing the problem [SZ10]
• Took 7 widely used, “large” projects
• Found:
– 15 mins to assess one mutation
– 45% uncaught mutations are equivalent
– Better tested project -> worse signal-to-noise ratio

Can we detect the equivalents?
• Not in the general case [BA82]
• Some specific cases can be detected
– Using compiler optimisation techniques [BS79]
– Using mathematical constraints [DO91]
– Line coverage changes [SZ10]
• All heuristic algorithms – not seen any
claiming to kill all equivalent mutants

Tools

Some Ruby, then a Java one I liked

Ruby
• Looked into Heckle
• Seemed unmaintained (nothing since 2009)
• Then I saw...

Ruby
• Mutant seems to be the new favourite
• Runs in Rubinius (1.8 or 1.9 mode)
• Only supports RSpec
• Easy to set up
rvm install rbx-head
rvm use rbx-head
gem install mutant
• And easy to use
mutate “ClassName#method_to_test” spec

Java
• Loads of tools to choose from
• Bytecode vs source mutation
• Will look at PIT (seems like one of the better
ones)

PIT - pitest.org
• Works with “everything”
– Command line
– Ant
– Maven
• Bytecode level mutations (faster)
• Very customisable
– Exclude classes/packages from mutation
– Choose which mutations you want
– Timeouts
• Makes pretty HTML reports (line/mutation coverage)

Summary
• Can point at weak areas in your tests
• At the same time, can be prohibitively noisy
• Try it and see

References
• [BA82] - T. A. Budd and D. Angluin. Two notions of correctness and
their relation to testing. Acta Informatica, 18(1):31-45, November
1982.
• [BS79] - D. Baldwin and F. Sayward. Heuristics for determining
equivalence of program mutations. Research report 276,
Department of Computer Science, Yale University, 1979.
• [DO91] - R. A. DeMillo and A. J. O
utt. Constraint-based automatic test data generation. IEEE
Transactions on Software Engineering, 17(9):900-910, September
1991.
• [SZ10] - D. Schuler and A. Zeller. (Un-)Covering Equivalent Mutants.
Third International Conference on Software Testing, Verification and
Validation (ICST), pages 45-54. April 2010.

Also interesting
• [AHH04] – K. Adamopoulos, M. Harman and R. M. Hierons. How to
Overcome the Equivalent Mutant Problem and Achieve Tailored
Selective Mutation Using Co-evolution. Genetic and Evolutionary
Computation -- GECCO 2004, pages 1338-1349. 2004.

Mutation Testing - Ruby Edition

More Related Content

What's hot

Similar to Mutation Testing - Ruby Edition

Recently uploaded

Mutation Testing - Ruby Edition

Editor's Notes