Software Testing -

Introduction
DR ATUL GUPTA
Why Software Testing?
Errors, defects, faults & Failures

 Error: A mistake made by a programmer

Example: Misunderstood the requirements.
 Defect/fault/bug: Manifestation of an error
in a program.
Example:
Incorrect code: if (a<b) {foo(a,b);}
Correct code: if (a>b) {foo(a,b);}

 Failure: Manifestation of one or more faults

in the observed program behavior
Failure
 Incorrect program behavior due to a fault in
the program.
 Failure can be determined only with respect
to a set of requirement specifications.
 A necessary condition for a failure to occur
is that execution of the program force the
erroneous portion of the program to be
executed.
 Is it Sufficient condition as well?
 NO
Testing Problem

Error-revealing
Inputs inputs cause
failure

Program
Erroneous
Outputs outputs indicate
failure
Testing Activities in the SDLC
Requirement Acceptance
Specification Testing

SRS Tested Software

System Design System Testing

System Design Subsystem

Detailed Design Integration Testing

Detailed Design Tested Units

Coding Unit Testing

Code
Levels of Testing

Type of Testing Performed By

 Low-level testing
 Unit (module) testing Programmer/Developer
 Integration testing Development team
 High-level testing
 System testing Independent Test Group
 Acceptance testing Customer
Testing Terms
 Test case – A set of Inputs, execution preconditions, and
expected outcomes for testing an specific aspect of CUT
 Test Suite - A collection of test cases for the CUT
 Test Criterion – A set of test requirements
 Failure – An observation of the behavior of CUT different
from expected one
 Fault – A condition in CUT that may cause failure
 Effectiveness – Fault detection capability
 Efficiency – The average testing cost (i.e. effort) to identify a
fault in the program
A Testing Process
The Code Under Test (and
its Specifications)

Procedural Object-Oriented
Systems Systems
Test Case Selection Test Case Selection
(Test Input Selection) (Test Sequence
Selection + Test Input
Selection)

Test Case Execution

Comparison with
Expected Outcomes
(Test Oracle)

NO
Test Execution Results Stop?
(Pass/Fail Information)
YES

Stop
A Testing Process
The Code Under Test (and
its Specifications)

Procedural Object-Oriented
Systems Systems

Test Case Selection Test Case Selection

(Test Input Selection) (Test Sequence
Selection + Test Input
Selection)

Test Case Execution

Comparison with
Expected Outcomes
(Test Oracle)

YES
NO
Debugging/ Fail Test Execution Results
Stop?
Bug Fixing ? (Pass/Fail Information)
YES

Stop
A Testing Process
The Code Under Test (and
its Specification)

Procedural Object-Oriented
Systems Systems

Test Case Selection Test Case Selection

(Test Input Selection) (Test Sequence
Selection + Test Input
Selection)

Test Case Execution

Comparison with
Expected Outcomes
(Test Oracle)

YES
Debugging/ Test Execution Results NO
Fail
(Pass/Fail Information) Stop?
Bug Fixing ?
NO
YES
Stop
Stop
Unit Testing
 Done on individual units (Class or small
cluster classes in OO System and a
function or modules in Procedural Systems)
 Test unit w.r.t unit specification
 Mostly done by developers/programmers
 requires stubs and drivers
 Further Popularized by the availability of
Unit Testing Frameworks
Stubs and Drivers
e.g. unit eg. to unit test B
 Stub relationships in isolation
 dummy units
which simulates A Stub for A
the function of Driver
other units
required to test
unit under test B B
 Driver
 Code that
executes unit C Stub for C
under test
Integration Testing

 Tests
a group of units, modules, or a
subsystem
 Test
subsystem structure w.r.t design,
subsystem functions
 Focuses on unit interfaces
 Done by one/group of developers
 Undertaken based on dependency
considerations
Integration Test Approaches
 Non-incremental (Big-Bang integration )
 unit test each module independently
 combine all the units to form the subsystem in one step, and
test the combination
 Incremental
 instead of testing each unit in isolation, the next unit to be
tested is first combined with the set of units that have
already been tested
 integration approaches:-
 Functional System: Top-down, Bottom-up

 OO System: Based on dependency considerations

Comparison
Non-Incremental Incremental
 requires less stubs,  requires more stubs,
drivers drivers
 module interfacing  module interfacing
errors detected late errors detected early
 debugging errors is  debugging errors is
difficult easier
 results in more
thorough testing of
modules
Example: Integration Testing in Functional
Systems

B C D

E F H
Example: Integration Testing in OO Systems

B C D

E F H
Top-down Integration

 Begin with the top module in the module call hierarchy

 Stub modules are produced
 Stubs are often complicated
 The next module to be tested is any module with at least
one previously tested superordinate (calling) module
 After a module has been tested, one of its stubs is
replaced by an actual module (the next one to be tested)
and its required stubs
 Top-down Integration Testing

Example:
A

Stub B Stub C Stub D

 Top-down Integration Testing

Example: A

B Stub C Stub D

Stub E Stub F
Bottom-Up Integration
 Begin with the terminal modules (those that do
not call other modules) of the modules call
hierarchy
 A driver module is produced for every module
 The next module to be tested is any module
whose subordinate modules (the modules it
calls) have all been tested
 After a module has been tested, its driver is
replaced by an actual module (the next one to
be tested) and its driver
Example: Module Hierarchy

B C D

E F H
 Bottom-Up Integration Testing
Example:

Driver E Driver F

E F
Bottom-Up Integration Testing

Example:
Driver A

E F
Comparison

Top-down Integration Bottom-up Integration

 Advantage  Advantage
 a skeletal version of the  Stubs are NOT required
program can exist early hence less expensive
 Disadvantage  Disadvantage
 required stubs could be
 the program as a whole
expensive
does not exist until the
last module is added

No clear winner
System Testing
 Process of attempting to demonstrate that the
program or system does not meet its original
requirements and objectives as stated in the
requirements specification
 Test cases derived from
 Requirements specification
 System objectives, user documentation
System Testing: An Example
Types of System Tests
 Functional Testing
 Testing Functionality to be delivered by the system

Specialized Testing (Testing NFRs)

 Volume testing
 to determine whether the program can handle the required volumes of data,
requests, etc.
 Load/Stress testing
 to identify peak load conditions at which the program will fail to handle required
processing loads within required time spans
 Usability (human factors) testing
 to identify discrepancies between the user interfaces of a product and the
human engineering requirements of its potential users.
 Security Testing
 to show that the program’s security requirements can be subverted
Types of System Tests
Specialized Testing
 Performance testing
 to determine whether the program meets its performance
requirements (eg. response times, throughput rates, etc.)
 Recovery testing
 to determine whether the system or program meets its
requirements for recovery after a failure
 Installability testing
 to identify ways in which the installation procedures lead to
incorrect results
 Configuration Testing
 to determine whether the program operates properly when
the software or hardware is configured in a required manner
Types of System Tests

Specialized Testing
 Compatibility/conversion testing
 to determine whether the compatibility objectives of the
program have been met and whether the conversion
procedures work
 Reliability/availability testing
 to determine whether the system meets its reliability and
availability requirements
 Resource usage testing
 to determine whether the program uses resources (memory,
disk space, etc.) at levels which exceed requirements
Acceptance Testing

 Performed by the customer or end user

 Compare the software to its initial requirements and needs of its
end users
Acceptance Testing
Tests performed on a SW Product before its released to a wide user
community.
 Alpha testing
 conducted at the developer’s site by a User
 tests conducted in a controlled environment
 Beta testing
 conducted at one or more User sites by the end user of the SW
 it is a “live” use of the SW in an environment over which the developer
has no control
Regression Testing
 Testing changes made to the code
 To ensure that no new bugs are introduced in the system
 Can be performed at any level – unit, integration, and system level
 In practice, mostly performed at unit and system level
 Challenges include
Reducing Regression Test Suite size
Identify Invalid test cases
Write additional test cases
Automation
When to Stop Testing ?

 Stop when the scheduled time for testing expires

 Stop when all the test cases execute without
detecting errors
-- both criteria are not good !
Better Test Completion Criteria
Make use of Coverage based Testing

Example: Test cases derived from

1) satisfying multi-condition coverage and
2) boundary-value analysis and
3) cause-effect graphing
Better Test Completion Criteria
State the completion criteria in terms of
number of errors to be found. This requires:
 An estimate of total number of errors in the
program
 An estimate of the % of errors that can be
found through testing
 An estimates of what fraction of errors
originate in particular design processes, and
during what phases of testing they get
detected.
 Better Test Completion Criteria
 Plot the number of errors found per unit time during a test
phase.
 The rate of error detection falls below a specified threshold

40- 40-

# Errors found
# Errors found

30- 30-
20- 20-
10- 10-
1 2 3 4 5 6 1 2 3 4 5 6
Week Week
Summary (Part-I)
 Testing is performed at different levels
 Unit Testing is the first formal test activity
 Unit Testing can be highly efficient and cost
effective
 Integration Testing is incremental
 Regression testing is about testing changes
made to the code
 Tool support is highly desirable in software
testing
 How can the effectiveness and efficiency of
software testing can be enhanced further?
References
 Beizer, Boris (1990). Software Testing Techniques (Second
Edition ed.)
 Myers, Glenford J. (1979). The Art of Software Testing. John
Wiley and Sons
 Wikipedia http://en.wikipedia.org/wiki/Software_testing