Mutation Testing and MuJava

1
Mutation testing of object orientedMutation testing of object oriented
programs of Java : µJavaprograms of Java : µJava
Krunal Parmar
(15CS60R13)
Guided By :
Prof. Rajib Mall
1

2
OutlineOutline
• Introduction
• Overview of Object Oriented Testing.
• Fault hypothesis for object-oriented software
• Mutation testing.
• Mutation testing tool for java (µJava).
• µJava operators and example.
• Demo of testing program using µJava.
• Conclusion
2

• Software testing is necessary to produce highly
reliable systems.
• More than 50% of development effort is being
spent on testing.
• Quality and effective test case design is equally
important.
IntroductionIntroduction
3

4
Object Oriented TestingObject Oriented Testing
• Research confirms that testing methods proposed
for procedural approach are not adequate for
Object Oriented approach. Ex. Statement
coverage
• Object Oriented software testing poses
additional problems due to Object Oriented
fetchers. Ex. Inheritance ,encapsulation and
polymorphism
• Testing efforts for OO software found to be
increased compared to testing procedural
software.
4

• Object Oriented software testing is performed
at different levels.
• Object oriented testing can be classified into
three levels:
1) Class level.
2) Cluster level.
3) System level.
Object Oriented Testing (contd.)Object Oriented Testing (contd.)
55

1 ) Class level:
– Testing interactions between attributes and
methods of class must be addressed.
2 ) Cluster level:
– Tests the interactions among a group of
cooperating classes.
– A sequence of interactions is typically
required to implement a visible behaviour
(i.e. a use case).
3 ) System level:
– Tests an entire operational system.
Object Oriented Testing (contd.)Object Oriented Testing (contd.)
6

7
fault hypothesis for object-orientedfault hypothesis for object-oriented
softwaresoftware
fault hypothesis :
an assumption about where faults are likely to
be found.
Because number of possible tests is infinite
for practical purposes, Rational testing must be
based on fault hypothesis.
7

8
fault hypothesis (Cont.)fault hypothesis (Cont.)
There are two general fault hypothesis which
correspond to two basic testing strategies:
( 1 ) conformance-directed testing, which seeks to
establish conformance to requirements or
specifications.
( 2 ) fault-directed testing, which seeks to reveal
implementation faults.
8

9
fault-directedfault-directed testingtesting
• Fault-directed testing is motivated by the
observation that conformance can be demonstrated
for an implementation that contains faults.
• Searching for faults is a practical and prudent
alternative to conformance (Myers, 1979).
• Since the combinations of input, state, output and
paths are astronomically large, efficient probing of
an implementation requires a specific fault
hypothesis
to direct the search for faults. 9

10
Testing is a good thingTesting is a good thing
But how do we know our tests are good?
10

Mutation TestingMutation Testing
• Mutation testing is a fault-based testing technique
that measures the effectiveness of test cases.
• Proposed by Richard J. Lipton in 1971
(winner of 2014 Knuth Prize)
• A better way to measure the quality of your tests
• Surge of interest in the 1980s
“Who watches the watchmen?”
1111

TERMINOLOGY: MutationTERMINOLOGY: Mutation
• A mutation is a (small) change in your codebase,
for example:
1212

TERMINOLOGY: MUTANTTERMINOLOGY: MUTANT
DebitCard >= anotherDebitCard
^(type = anotherDebitCard type)
and: [ number = anotherDebitCard number ]
DebitCard >= anotherDebitCard
^(type = anotherDebitCard type)
or: [ number = anotherDebitCard number ]
Operator: Change #and: by #or:
• A mutant is a mutated version of your class or
method.
13

Process of creating the MutantProcess of creating the Mutant
The Source
Code
The Mutation “Operator”
Mutation
Process
The “Mutant”
1414

Mutation Testing (contd.)Mutation Testing (contd.)
• Mutation testing is based on the assumption
that a program will be well tested if a majority
of simple faults are detected and removed.
• Simple faults are introduced into the program
by creating a set of mutants.
• These mutants are created from the original
program by applying mutation operators,
which describe syntactic changes to the
programming language.
1515

• Measure the effectiveness of a test set in
terms of its ability to detect faults
17/20
Objective of mutation testingObjective of mutation testing
1717

Steps : Mutation testingSteps : Mutation testing
1. Run test suite
2. Change code (mutate)
3. Run test suite again
4. Observe outcomes 18
Test cases are used to execute these mutants with
the goal of causing each mutant to produce
incorrect output.
A test case that distinguishes the program from
one or more mutants is considered to be effective
at finding faults in the program.
Steps :
18

OUTCOME #1:OUTCOME #1: KILLEDKILLED
19
• If a test set can distinguish a mutant from the
original program i.e., it produces different
execution result, the mutant is said to be killed.
• In short, mutant is killed if a test fails (detecting
the mutated code)
OUTCOME #2:OUTCOME #2: LIVEDLIVED
• A mutant didn’t trigger a failing test , mutant
remains live because it is equivalent to the
original program i.e. it is functionally
identical to the original program or the test
data is inadequate to kill the mutant.
19

OUTCOME #3:OUTCOME #3: TIMED OUTTIMED OUT
• The mutant caused the program loop,
get stuck
20
Other outcomesOther outcomes
1) NON-VIABLE
JVM could not load the mutant byte-code
2)MEMORY ERROR
JVM ran out of memory during test
3)RUN ERROR
An error but none of the above.
20

If process is not
error-free, fix it
Test
Mutant
s
Process
MutationMutationMutation Tests
Test
Process
Create
Mutants
Yes
Test
CompleteNo
Any Live
Mutants?
Problem with
Tests?
Any Mutations
that are caught by
tests are killed
New Test
Data
The Mutation ProcessThe Mutation Process
21

Equivalent MutantsEquivalent Mutants
• There may be surviving mutants that cannot be
killed, these are called Equivalent Mutants
• Although syntactically different, these mutants
are indistinguishable through testing.
• They therefore have to be checked ‘by hand’
while...
...
i++
if(i>=5)
break;
while...
...
i++
if(i==5)
break;
Original Code Mutant code 22

24
• If a test data is inadequate, it can be improved by
adding test cases to kill the live mutant.
• A test set which can kill all non-equivalent
mutants is said to be adequate.
How to measure the adequacy of a test set?
Adequacy of test set :
(No. of killed mutants)
(No. of non-equivalent mutants)
Adequacy of test setAdequacy of test set
24

25
Advantages of Mutation Testing
1) It can show the ambiguities in test cases.
2) It leads to more reliable product.
Disadvantages of Mutation Testing
1) It is time consuming technique, hence requires
automated tools.
2) Each mutation will have the same size as that of
the original program. So, a large number of mutant
programs may need to be tested against the
candidate test suite.
25

26
TOOLINGTOOLING
There exist many mutation tools for different
languages and technologies.
•µJava: http://cs.gmu.edu/~offutt/mujava/
•Jester: http://jester.sourceforge.net/
•Jumble: http://jumble.sourceforge.net/
•javaLanche:
http://www.st.cs.unisaarland.de/mutation/ 26

27
µjavaµjava
• µJava is a automated mutation system for Java
programs. It automatically generates mutants for
mutation testing.
• µJava can test individual classes and packages of
multiple classes.
• Tests are supplied by the users as sequences of
method calls to the classes under test encapsulated
in methods in separate classes.
27

28
Input
output
28
µjava (Contd.)µjava (Contd.)

29
• µµJava implements both inter- and intra-class
mutation operators.
• Class mutation operators are classified into four
groups, based on the language features that are
affected
Mutation OperatorsMutation Operators
1. Encapsulation
2. Inheritance
3. Polymorphism
4. Java-Specific Features
29

30
Mutation Operator for EncapsulationMutation Operator for Encapsulation
AMC – Access modifier change:
•The AMC operator changes the access level for
instance variables and methods to other access
levels.
•The purpose of the AMC operator is to guide
testers to generate test cases that ensure that
accessibility is correct.
for example,
30

31
Mutation Operators for InheritanceMutation Operators for Inheritance
• Inheritance is a powerful and useful abstraction
mechanism, but incorrect use of inheritance can
lead to a number of faults.
• We define five mutation operators to try to test
the various aspects of using inheritance,
covering
variable shadowing, method overriding, the use
of super and definition of constructors.
31

32
Operators for Inheritance(Contd.)Operators for Inheritance(Contd.)
• IHD – Hiding variable deletion
• IHI – Hiding variable Insertion
32

33
Operators for Inheritance(Contd.)Operators for Inheritance(Contd.)
• IOD – Overriding method deletion
• IOP – Overriding method calling Position change
33

34
Mutation Operators for polymorphismMutation Operators for polymorphism
• Polymorphism allows the behaviour of an object
reference to be different depending the actual
type. Therefore, it is important to identify and
exercise the program with all possible type
bindings.
• The polymorphism mutation operators are
designed to ensure this type of testing.
34

35
Mutation Operators for Java SpecificMutation Operators for Java Specific
featuresfeatures
• Some object-oriented features are not common
to all object-oriented languages. This group of
operators attempt to ensure correct use of such
features supported in Java
35

36
How µJava does it ?How µJava does it ?
• µµJava uses a reflection technique to satisfy
those requirements, specifically, to generate and
run mutants.
• Reflection is the ability of a program to observe
and possibly modify its high level structure.
reflection is a natural way to implement mutation
analysis for several reasons.
1 : it provides an API to easily change the behaviour of a
program during execution.
2 : it lets programmers extract OO-related Information about
a class by providing an object that represents a logical 36

37
Example of Mutant generationExample of Mutant generation
( IOD operator)( IOD operator)
• Parent class • Child class
• Mutant generation code
37

38
Generating mutants with µJavaGenerating mutants with µJava
Mutant generator screen
38

39
Generating mutants with µJava (contd.)Generating mutants with µJava (contd.)
Class mutant viewer screen
39

40
Generating mutants with µJava (contd.)Generating mutants with µJava (contd.)
Test case runner screen
40

41
ConclusionConclusion
41
• Mutation testing is the powerful technique for the
assessment and enhancement of tests.
• The effectiveness of mutation testing depends
heavily on the types of faults that the mutation
operators are designed to represent. Therefore,
the quality of the mutation operators is key to
mutation testing.
• µJava allows the tester to enter and run tests,
and evaluates the mutation coverage of the tests.

42
ReferencesReferences
42
• Testing Object Oriented Software : a Survey.
ROBERT V. BINDER , RBSC Corporation, 3 First National Plaza,
Suite 1400,Chicago,IL 6060 24205,U.S.A.
• Inter-Class Mutation Operators for Java
Yu-Seung Ma, Yong-Rae Kwon and Jeff Offutt. Proceedings of the
13th International Symposium on Software Reliability Engineering,
IEEE Computer Society Press, Annapolis MD, November 2002, pp.
352-363.
• Wikipedia : Mutation Testing
https://en.wikipedia.org/wiki/Mutation_testing
• µJava Home Page
https://cs.gmu.edu/~offutt/mujava/

Mutation Testing and MuJava

In this document

More Related Content

What's hot

Similar to Mutation Testing and MuJava

Recently uploaded

Mutation Testing and MuJava