Overview Regression test selection Automation and tools

Regression Testing

Meenakshi D’Souza

International Institute of Information Technology Bangalore.

Overview Regression test selection Automation and tools

Overview

Regression testing: Overview

Regression test selection techniques
Automation and tools
Overview Regression test selection Automation and tools

Regression Testing

Post release, a software is maintained by developers.

Maintenance involves making modifications to code in use,
adding new code, removing code etc.
These changes are done in response to change requests by
users/customers.
Regression testing is the process of validating modified
software to detect whether new errors have been introduced
into previously tested code and to provide confidence that
modifications are correct.
It is a black-box testing technique.
Typically very expensive.
Overview Regression test selection Automation and tools

Typical regression testing process

Let P be a procedure or program. Let P 0 be a modified version of

P and let T be a test suite for P. A typical regression test
proceeds as follows:
1 Select T 0 ⊆ T , a set of test cases to execute on P 0 .
2 Test P 0 with T 0 , establishing P 0 s correctness with respect to
T 0.
3 If necessary, create T 00 , a set of new functional or structural
test cases for P 0 .
4 Test P 0 with T 00 , establishing P 0 s correctness with respect to
T 00 .
5 Create T 000 , a new test suite and test execution profile for P 0 ,
from T , T 0 and T 00 .
Overview Regression test selection Automation and tools

Regression testing process: Corresponding problems

Each of the steps in regression testing leads to an important

problem to solve.
Step 1 involves the regression test selection problem: the
problem of selecting a subset T 0 of T with which to test P 0 .
Step 3 addresses the coverage identification problem: the
problem of identifying portions of P 0 or its specification that
require additional testing.
Steps 2 and 4 address the test suite execution problem: the
problem of efficiently executing test suites and checking test
results for correctness.
Step 5 addresses the test suite maintenance problem: the
problem of updating and storing test information.
Overview Regression test selection Automation and tools

Regression vs. specifications

Regression test selection is applicable both in cases where the

specifications have not changed, and where they have
changed.
In the latter case, it is necessary to identify the test cases in
T that are obsolete for P 0 prior to performing test case
selection.
Test case t is obsolete for program P 0 iff t specifies an input to
P 0 that is invalid for P 0 , or tt specifies an invalid input-output
relation for P 0 .
Having identified these test cases and removed them from T ,
regression test selection can be performed on the remaining
test cases.
Overview Regression test selection Automation and tools

Selecting regression tests

Selecting regression tests

Requires knowledge about the system and how it affects by
the existing functionalities.
Tests are selected based on the area of frequent defects.
Tests are selected to include the area, which has undergone
code changes many a times.
Tests are selected based on the criticality of the features.
Overview Regression test selection Automation and tools

Minimization techniques

Minimization test selection techniques for regression testing

attempt to select minimal sets of test cases from T that yield
coverage of modified or affected portions of P.
The problem, in its generality, is undecidable.
0-1 integer programming based techniques are available for
procedures that are basically a sequence of statements with
single entry and single exit (no branching, looping etc.).
Overview Regression test selection Automation and tools

Dataflow techniques

Data-flow coverage based regression test selection techniques

select test cases that exercise data interactions that have been
affected by modifications.
For example, the technique of Harrold and Soffa [1988]
requires that every definition-use pair that is deleted from P,
new in P 0 , or modified for P 0 be tested.
The technique selects every test case in T that, when
executed on P, exercised, deleted or modified definition-use
pairs, or executed a statement containing a modified
predicate.
Overview Regression test selection Automation and tools

Safe techniques

Techniques that are not safe can fail to select a test case that
would have revealed a fault in the modified program.
In contrast, when an explicit set of safety conditions can be
satisfied, safe regression test selection techniques guarantee
that the selected subset, T 0 , contains all test cases in the
original test suite T that can reveal faults in P 0 .
For example, the technique of Rothermel and Harrold [1997]
uses CFG representations of P and P 0 , and test execution
profiles gathered on P, to select every test case in T that,
when executed on P, exercised at least one statement that
has been deleted from P, or that, when executed on P 0 , will
exercise at least one statement that is new or modified in P 0
(when P is executed, a statement that does not exist in P
cannot be exercised).
Overview Regression test selection Automation and tools

Random techniques

In this case, a randomly selected set of test cases for P are

executed on P 0 .
This is often useless and done only to show that some
regression testing is done.
Overview Regression test selection Automation and tools

Retest-All techniques

The retest-all technique simply reuses all existing test cases.

To test P 0 , the technique effectively selects all test cases in T .
Overview Regression test selection Automation and tools

Regression test selection

Different regression test selection techniques create different

trade-offs between the costs of selecting and executing test
cases, and the need to achieve sufficient fault detection ability.
It has been found through empirical studies that the choice of
regression test selection algorithm significantly affects the
cost-effectiveness of regression testing.
Overview Regression test selection Automation and tools

Automation of regression testing

Most of the testing efforts in regression testing can be

automated.
Typically the previously executed test cases or the selected
test cases are again run on a new build.
Most of the regression test tools are record and playback type.
The test cases are recorded by navigating through the
program under test and verified whether expected results are
coming or not.
Overview Regression test selection Automation and tools

Some regression testing tools

Regression Tester: Available from

http://www.regressiontester.com/.
TimeShiftX: Available from https://www.vornexinc.com/.
Tools from Micro Focus: Available from
https://www.microfocus.com.
Overview Regression test selection Automation and tools

Some references

Tools: Links already provided, Last access October 2017.

A nice survey paper on regression test selection:
T. L. Graves, M. J. Harrold, J-M. Kim, A. Porter and G.
Rothermel, An Empirical Study of Regression Test Selection
Techniques, in ACM Transactions on Software Engineering
and Methodology, 30(2), April 2001.