Software testing and analysis

SOFTWARE TESTING AND
ANALYSIS BY 360LOGICA

ULTIMATE GOAL FOR SOFTWARE
TESTING

Quality
Assurance
A L E A D I N G S O F T WA R E T E S T I N G -
360LOGICA

V & V GOALS

Verification and validation should establish confidence that the software is fit for
purpose
This does NOT mean completely free of defects
Rather, it must be good enough for its intended use and the type of use will
determine the degree of confidence that is needed

A LEADING SOFTWARE TESTING - 360LOGICA

VERIFICATION VS. VALIDATION

• Verification: The software should
conform to its specification (Are we
building the product right?)

• Validation: The software should do
what the user really requires (Are we
building the right product?)


“CLASSICAL” LIFECYCLE MODEL

Requirements Phase
Specification Phase (Analysis)
Planning Phase
Design Phase
Implementation Phase
Integration and Testing
Maintenance
Retirement


COST TO FIX FAULTS

60* to 100*

1.5* to 6*

Cost 1*

Definition Development Post Release


THE V & V PROCESS

• Is a whole life-cycle process - V & V
must be applied at each stage in the
software process.
• Has two principal objectives
– The discovery of defects in a system
– The assessment of whether or not the
system is usable in an operational situation.


STATIC AND DYNAMIC VERIFICATION

• Software inspections and
walkthroughs - Concerned with
analysis of the static system
representation to discover problems
(static verification)
• Software testing - Concerned with
exercising and observing product
behaviour (dynamic verification)
– The system is executed with test data
and its operational behaviour is observed

STATIC AND DYNAMIC V&V

Static
verification

Requirements High-level Formal Detailed
specification Program
specification design design

Dynamic
Prototype
validation


V & V PLANNING

• Careful planning is required to get the most
out of testing and inspection processes
• Planning should start early in the
development process
• The plan should identify the balance
between static verification and testing
• Test planning is about defining standards
for the testing process rather than
describing product tests


THE V-MODEL OF DEVELOPMENT

Requir ements System System Detailed
specification specification design design

System Sub-system Module and
Acceptance
integration integration unit code
test plan
test plan test plan and tess

Acceptance System Sub-system
Service
test integration test integration test


THE STRUCTURE OF A SOFTWARE TEST
PLAN

The testing process
Requirements traceability
Tested items
Testing schedule
Test recording procedures
Hardware and software requirements
Constraints


WALKTHROUGHS

Informal examination of a product
(document)
Made up of:
 developers
 client
 next phase developers
 Software Quality Assurance group leader

Produces:
 list of items not understood
 list of items thought to be incorrect F T W A R E T E S T I N G - 3 6 0 L O G I C A
A LEADING SO

SOFTWARE INSPECTIONS

Involve people examining the source
representation with the aim of discovering
anomalies and defects
Do not require execution of a system so may
be used before implementation
May be applied to any representation of the
system (requirements, design, test data,
etc.)
Very effective technique for discovering errors


INSPECTION SUCCESS
Many different defects may be discovered in a single inspection. In testing, one
defect may mask another so several executions are required
The reuse domain and programming knowledge so reviewers are likely to have
seen the types of error that commonly arise


INSPECTIONS AND TESTING

Inspections and testing are complementary
and not opposing verification techniques
Both should be used during the V & V process
Inspections can check conformance with a
specification but not conformance with the
customer‟s real requirements
Inspections cannot check non-functional
characteristics such as performance,
usability, etc.


PROGRAM INSPECTIONS
Formalised approach to document reviews
Intended explicitly for defect DETECTION (not correction)
Defects may be logical errors, anomalies in the code that might indicate an
erroneous condition (e.g. an un-initialised variable) or non-compliance with
standards


INSPECTION PRE-CONDITIONS

A precise specification must be available
Team members must be familiar with the
organisation standards
Syntactically correct code must be available
An error checklist should be prepared
Management must accept that inspection
will
increase costs early in the software
process
Management must not use inspections for

staff

INSPECTION PROCEDURE

System overview presented to inspection team
Code and associated documents are
distributed to inspection team in advance
Inspection takes place and discovered errors
are noted
Modifications are made to repair discovered
errors
Re-inspection may or may not be required


INSPECTION TEAMS

Made up of at least 4 members
Author of the code being inspected
Inspector who finds errors, omissions and
inconsistencies
Reader who reads the code to the team
Moderator who chairs the meeting and notes
discovered errors
Other roles are Scribe and Chief moderator


INSPECTION CHECKLISTS

Checklist of common errors should be used to
drive the inspection
Error checklist is programming language
dependent
The 'weaker' the type checking, the larger the
checklist
Examples: Initialization, Constant naming, loop
termination, array bounds, etc.


INSPECTION RATE

500 statements/hour during overview
125 source statement/hour during individual
preparation
90-125 statements/hour can be inspected
Inspection is therefore an expensive process
Inspecting 500 lines costs about 40 man/hours
effort (@ $50/hr = $2000!!!)


PROGRAM TESTING

• Can reveal the presence of errors NOT their
absence
• A successful test is a test which discovers
one or more errors
• The only validation technique for non-
functional requirements
• Should be used in conjunction with static
verification to provide full V&V coverage


EXECUTION BASED TESTING

“Program testing can be a very effective way
to show the presents of bugs but is
hopelessly inadequate for showing their
absence” [Dijkstra]

Fault: “bug” incorrect piece of code
Failure: result of a fault
Error: mistake made by the
programmer/developer

TESTING AND DEBUGGING

• Defect testing and debugging are distinct
processes
• Verification and validation is concerned with
establishing the existence of defects in a
program
• Debugging is concerned with locating and
repairing these errors
• Debugging involves formulating a
hypothesis
about program behaviour then testing these
hypotheses to find the system error

THE DEBUGGING PROCESS

Test Test
results Specification
cases

Locate Design Repair Re-test
error error repair error program


TESTING PHASES

Component Integration
testing testing

Software developer Independent testing team


TESTING PHASES
Component testing
 Testing of individual program components
 Usually the responsibility of the component
developer (except sometimes for critical
systems)
 Tests are derived from the developer’s
experience
Integration testing
 Testing of groups of components integrated to
create a system or sub-system
 The responsibility of an independent testing
team
 Tests are based onA aEsystemT W A R E T E S T I N G - 3 6 0 L O G I C A
L A D I N G S O F specification

TESTING PRIORITIES
• Only exhaustive testing can show a
program is free from defects. However,
exhaustive testing is impossible
• Tests should exercise a system's
capabilities rather than its components
• Testing old capabilities is more important
than testing new capabilities
• Testing typical situations is more
important than boundary value cases


TEST DATA AND TEST CASES

• Test data Inputs which have been
devised to test the system

• Test cases Inputs to test the system
and the predicted outputs from these
inputs if the system operates
according to its specification


DEVELOPMENT OF TEST CASES

Test cases and test scenarios comprise
much of a software systems testware.

Black box test cases are developed by
domain analysis and examination of the
system requirements and specification.

Glass box test cases are developed by
examining the behavior of the source
code.

THE DEFECT TESTING PROCESS

Test Test Test Test
cases data results reports

Design test Prepare test Run program Compare results
cases data with test data to test cases


METHODS OF TESTING

Test to specification:
 Black box,
 Data driven
 Functional testing
 Code is ignored: only use specification document to
develop test cases
Test to code:
 Glass box/White box
 Logic driven testing
 Ignore specification and only examine the code.


CAN YOU GUARANTEE A PROGRAM IS
CORRECT?
This is called the Halting Problem

Write a program to test if any given program
is correct. The output is correct or
incorrect.
Test this program on itself.
If output is incorrect, then how do you know
the output is correct?

Conundrum, Dilemma, or Contradiction?

BLACK-BOX TESTING

An approach to testing where the program is considered as a „black-box‟
The program test cases are based on the system specification
Test planning can begin early in the software process


BLACK-BOX TESTING
I n p u t s c a u s i n g

a n o m a l o u s

Input test data I b e h a v i o u r

e

S y s t e m

O u t p u t s w h i c h r e v e a l

t h e p r e s e n c e o f

Output test results Oe d e f e c t s


PAIRING DOWN TEST CASES

Use methods that take advantage of
symmetries, data equivalencies, and
independencies to reduce the number of
necessary test cases.
 Equivalence Testing
 Boundary Value Analysis

Determine the ranges of working system
Develop equivalence classes of test cases
Examine the boundaries of these classes
carefully A LEADING SOFTWARE TESTING - 360LOGICA

EQUIVALENCE PARTITIONING

Input data and output results often fall into different classes where all members of
a class are related
Each of these classes is an equivalence partition where the program behaves in an
equivalent way for each class member
Test cases should be chosen from each partition


EQUIVALENCE PARTITIONING

Invalid inputs Valid inputs

System

Outputs


BOUNDARY VALUE TESTING

• Partition system inputs and outputs into
„equivalence sets‟
– If input is a 5-digit integer between 10,000 and
99,999, equivalence partitions are < 10,000,
10,000 - 99, 999 and > 10, 000
• Choose test cases at the boundary of these
sets
– 00000, 09999, 10000, 99999, 10001


EQUIVALENCE PARTITIONS

3 11
4 7 10

Less than 4 Between 4 and 10 More than 10

Number of input values

9999 100000
10000 50000 99999

Less than 10000 Between 10000 and 99999 More than 99999

Input values

SEARCH ROUTINE SPECIFICATION

procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : in out BOOLEAN; L: in out ELEM_INDEX) ;

Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key)
or
-- the element is not in the array
( not Found and
not (exists i, T’FIRSTI N>=O iF T<=R T’LAST,3 6 0 L (i)I C= Key ))
A LEAD G S WA E TESTING -
T OG A

SEARCH ROUTINE - INPUT PARTITIONS

• Inputs which conform to the pre-conditions
• Inputs where a pre-condition does not hold

• Inputs where the key element is a member of
the array

• Inputs where the key element is not a
member
of the array

TESTING GUIDELINES - SEQUENCES

Test software with sequences which have only a single value
Use sequences of different sizes in different tests
Derive tests so that the first, middle and last elements of the sequence are
accessed
Test with sequences of zero length


SEARCH ROUTINE - INPUT PARTITIONS

Array Element
Single value In sequence
Single value Not in sequence
More than 1 value First element in sequence
More than 1 value Last element in sequence
More than 1 value Middle element in sequence
More than 1 value Not in sequence

Input sequence (T) Key (Key) Output (Found, L)
17 17 true, 1
17 0 false, ??
17, 29, 21, 23 17 true, 1
41, 18, 9, 31, 30, 16, 45 45 true, 7
17, 18, 21, 23, 29, 41, 38 23 true, 4
21, 23, 29, 33, 38 25 false, ??

SORTING EXAMPLE

Example: sort (lst, n)
 Sort a list of numbers
 The list is between 2 and 1000 elements

Domains:
 The list has some item type (of little concern)
 n is an integer value (sub-range)
Equivalence classes;
 n<2
 n > 1000
 2 <= n <= 1000


SORTING EXAMPLE

What do you test?
Not all cases of integers
Not all cases of positive integers
Not all cases between 1 and 1001

Highest payoff for detecting faults is to test around
the boundaries of equivalence classes.

Test n=1, n=2, n=1000, n=1001, and say n= 10
Five tests versus 1000.


WHITE-BOX TESTING

• Sometime called structural testing or glass-
box testing
• Derivation of test cases according to
program structure
• Knowledge of the program is used to
identify additional test cases
• Objective is to exercise all program
statements (not all path combinations)


TYPES OF STRUCTURAL TESTING

Statement coverage -
 Test cases which will execute every statement at least once.
 Tools exist for help
 No guarantee that all branches are properly tested. Loop exit?
Branch coverage
 All branches are tested once
Path coverage - Restriction of type of paths:
 Linear code sequences
 Definition/Use checking (all definition/use paths)
 Can locate dead code


WHITE-BOX TESTING

Test data

Tests Derives

Component Test
code outputs


WHITE BOX TESTING - BINARY SEARCH
int EXAMPLE
search ( int key, int [] elemArray)
{
int bottom = 0;
int top = elemArray.length - 1;
int mid;
int result = -1;
while ( bottom <= top )
{
mid = (top + bottom) / 2;
if (elemArray [mid] == key)
{
result = mid;
return result;
} // if part
else
{
if (elemArray [mid] < key)
bottom = mid + 1;
else
top = mid - 1;
}
} //while loop
return result;
} // search


BINARY SEARCH EQUIVALENCE
PARTITIONS

• Pre-conditions satisfied, key element in
array
• Pre-conditions satisfied, key element not in
array
• Pre-conditions unsatisfied, key element in
array
• Pre-conditions unsatisfied, key element not
in array
• Input array has a single value
• Input array has an even number of values

• Input array has an odd number of values

BINARY SEARCH EQUIVALENCE
PARTITIONS

Equivalence class boundaries

Elements < Mid Elements > Mid

Mid-point


BINARY SEARCH - TEST CASES

Input array (T) Key (Key) Output ( Found, L )
17 17 true, 1
17 0 false, ??
17, 21, 23, 29 17 true, 1
9, 16, 18, 30, 31, 41, 45 45 true, 7
17, 18, 21, 23, 29, 38, 41 23 true, 4
17, 18, 21, 23, 29, 33, 38 21 true, 3
12, 18, 21, 23, 32 23 true, 4
21, 23, 29, 33, 38 25 false, ??


PATH TESTING

The objective of path testing is to ensure that
the set of test cases is such that each path
through the program is executed at least
once
The starting point for path testing is a program
flow graph that shows nodes representing
program decisions and arcs representing
the flow of control
Statements with conditions are therefore
nodes in the flow graph

PROGRAM FLOW GRAPHS

• Describes the program control flow. Each
branch is shown as a separate path and
loops are shown by arrows looping back to
the loop condition node
• Used as a basis for computing the
cyclomatic
complexity
• Cyclomatic complexity = Number of edges -
Number of nodes +2


CYCLOMATIC COMPLEXITY

• The number of tests to test all control
statements equals the cyclomatic
complexity
• Cyclomatic complexity equals number of
conditions in a program
• Useful if used with care. Does not imply
adequacy of testing
• Although all paths are executed, all
combinations of paths are not executed


BINARY SEARCH FLOW
1 GRAPH
while bottom <= top
bottom > top
2

3 if (elemArray [mid] == key

8 4
(if (elemArray [mid]< key

5 6
9

7


INDEPENDENT PATHS

• 1, 2, 3, 8, 9
• 1, 2, 3, 4, 6, 7, 2
• 1, 2, 3, 4, 5, 7, 2
• 1, 2, 3, 4, 6, 7, 2, 8, 9
• Test cases should be derived so that
all of these paths are executed
• A dynamic program analyser may be
used to check that paths have been
executed A LEADING SOFTWARE TESTING - 360LOGICA

FEASIBILITY

Pure black box testing (specification) is
realistically impossible because there are
(in general) too many test cases to
consider.
Pure testing to code requires a test of every
possible path in a flow chart. This is also
(in general) infeasible. Also every path
does not guarantee correctness.
Normally, a combination of Black box and
Glass box testing is done.


INTEGRATION TESTING

Tests complete systems or subsystems
composed of integrated components
Integration testing should be black-box testing
with tests derived from the specification
Main difficulty is localising errors
Incremental integration testing reduces this
problem


INCREMENTAL INTEGRATION TESTING

A T1
T1
A
T1 T2
A B
T2
T2 B T3
T3
B C
T3 T4
C
T4
D T5

Test sequence Test sequence Test sequence
1 2A L E A D I N G SOFTWARE
3
TESTING - 360LOGICA

APPROACHES TO INTEGRATION TESTING

Top-down testing
 Start with high-level system and integrate from the
top-down replacing individual components by stubs
where appropriate
Bottom-up testing
 Integrate individual components in levels until the
complete system is created
In practice, most integration involves a
combination of these strategies


TOP-DOWN TESTING

Testing
Level 1 Level 1 . ..
sequence

Level 2 Level 2 Le vel 2 Level 2

Le vel 2
stubs

Le vel 3
stubs


BOTTOM-UP TESTING

Test
drivers
Testing
Level N Level N Le vel N Level N Level N
sequence

Test
drivers
Level N–1 Level N–1 Level N–1


SOFTWARE TESTING METRICS

Defects rates
Errors rates
Number of errors
Number of errors found per person hours
expended
Measured by:
 individual
 module
 during development
Errors should be categorized by origin, type,
cost A LEADING SOFTWARE TESTING - 360LOGICA

MORE METRICS

Direct measures - cost, effort, LOC, etc.
Indirect Measures - functionality, quality,
complexity, reliability, maintainability

Size Oriented:
 Lines of code - LOC
 Effort - person months
 errors/KLOC
 defects/KLOC
 cost/KLOC


Software testing and analysis

More Related Content

What's hot

Viewers also liked

Similar to Software testing and analysis

More from 360logica Software Testing Services (A Saksoft Company)

Recently uploaded

Software testing and analysis