Testing concepts [3] - Software Testing Techniques (CIS640)

Testing Concepts
Venkatesh-Prasad Ranganath
Kansas State University
Slides with * in title capture whiteboard content

Basics
• What is testing?
• What is the purpose of testing?
• How do you accomplish testing?

Basics*
• Ensuring a program runs correctly
• Handle correct input by producing correct output
• Handle invalid input by producing correct output
• Correctness guided by Speciﬁcation / Requirements
• Checking for faults
• Handling invalid inputs
• Meeting the criteria of requirements
• Checking for bugs
• Making sure it fails properly

Basics*
• Checking for unexpected behaviors
• Minimizing bad behaviors
• Checking expected output for variety of inputs
• Assurance from faulty code from not running how it is
supposed to
• Checking requirements have been met
• Checking invalid inputs are handled appropriately

Basics*
• Checking if requirements are met
• Ensuring (checking?) quality of software
• correctness, performance, reliability, …
• Reducing maintenance cost of software
• Way of guaranteeing the client software behaves as
intended

Basics*
• Companies are forced to :)
• Meeting regulations :)
• You want the product to be used
• We use buggy products all the time :)
• Code is not faulty / buggy
• Ensuring user satisfaction (??)

Basics
• Checking if actual outcome is the expected
outcome
• Detect failures/errors/deviations (this is fuzzy)
• Prove the presence of bugs (faults)
• What about proving the absence of bugs?
• How do we accomplish testing?
• Often by comparing two entities for equality

Fault, Error, and Failure*
1. Error: Exception thrown to handle certain behavior
2. Fault/Failure: Function does not work as expected
3. Error: Result of an external module not working as
desired
4. Failure: An issue that causes unexpected stoppage
5. Error: State of program that results due to fault/failure
6. Fault: Mistake caused by programmer
7. Fault: Program operating sub-optimally

Fault, Error, and Failure
compute(s) = (s + 1)2 mod 3 + 12

compute(s) = (s + 1)2 mod 3 + 12
def compute(s):
s = s + 1 #1
s = s * 2 #2
s = s % 3 #3
s = s + 12 #4
return s
- Software Testing: Concepts and Operations by Ali Mili and Fairouz Tchier (Section 6.1)

Steps
Correct /
Incorrect
Correct /
Incorrect
Correct /
Incorrect
input s ->1 / s -> 1 s -> 2 / s -> 2 s -> 3 / s -> 3
#1 s -> 2 / s -> 2 s -> 3 / s -> 3 s -> 4 / s -> 4
#2 s -> 4 / s -> 4 s -> 9 / s -> 6 s -> 16 / s -> 8
#3 s -> 1 / s -> 1 s -> 0 / s -> 0 s -> 1 / s -> 2
#4 s -> 13 / s -> 13 s -> 12 / s -> 12 s -> 13 / s -> 14
• Fault desensitized • Fault sensitized
• Error masked
• Fault sensitized
• Error propagated
• Failure observed

compute(s) = (s + 1)2 mod 3 + 12
def compute(s):
s = s + 1 #1
s = s * 2 #2
s = s % 3 #3
s = s + 12 #4
return s
• Fault is the mistake (or cause of error).
• Executions may sensitize faults.
• Error is the impact of the fault on
program states; Error is the difference
between correct and incorrect states.
• Executions may mask or propagate
errors.
• Failure is the event when the program
behaves incorrectly, i.e., violates the
speciﬁcation (observed outcome
differs from the expected outcome).
- Basic Concepts and Taxonomy of Dependable and Secure Computing by Algirdas Avizienis, Jean-Claude Laprie, Brian Randell, and Carl Landwehr. (Section 2.2)
Input
Fault desensitized
Fault
sensitized Error masked
Error propagated
Success
Failure
System

Specification, Program, and
Test
• Specification (S) describes what is to be
accomplished.
• sort an array of ints
• check if a given number exists in an array of ints
• Program (P) describes/embodies how something is
accomplished.
• an implementation using quick sort
• an implementation using binary search
• Test (T) checks parity between specification and
implementation.
- Software Testing: A Craftsman’s Approach, 4ed, by Paul C Jorgensen. (Chapter 1)

Speciﬁcation, Program and
Test: How are they related?
5 6
7
1
2
34
S P
T
8

Speciﬁcation and Program:
How are they related?
S
P
• Program P exhibits none of
the behaviors expected
according to speciﬁcation S

S
P
P
S
• Program P exhibits all of the
behaviors expected
• Some behaviors of program P
are not expected behaviors
• Program P exhibits some of
the behaviors expected
• All behaviors of program P
are expected behaviors

S
P
S
P
are expected according to
speciﬁcation S
• Most behaviors of program P
• Most behaviors of program P
are expected according to
speciﬁcation S

S
PT1
S
P
T2
• Test T1 tests for most of the
behaviors expected
• Test T1 tests for some of the
behaviors exhibited by
program P
behaviors expected
• Test T2 tests for most of the
program P

S
PT1
S
P
T2
• Test T1 tests for all of the
behaviors expected
and much more
program P
behaviors expected
• Test T2 tests for all of the
program P and much more

• Can we prove the presence or absence of bugs
with testing?
• What about fault/failure by commission and fault/
failure by omission?
• Can one fault lead to more than one error?
• Can one fault lead to more than one failure?
• Can the same error stem from different faults?
• Can the same failure stem from different faults?
Few Questions to Consider

• Can we prove the presence or absence of bugs with testing?
• Testing can prove the presence of bugs — a failing test is
evidence that the program can fail.
• In general, testing cannot prove the absence of bugs — a
passing test is not evidence that the program will not fail.
• What about fault/failure by commission and fault/failure by
omission?
• Fault/Failure by commission stems from an incorrect
implementation, e.g., using x*1 instead of x+1.
• Fault/Failure by omission stems from the absence of correct
implementation, e.g., absence of line 3 in compute().
• What about different kind of faults?
Few Questions to Consider

A detour into costs
Lifetime Cost Software Engg % Other Engg %
Design >99 <1
Manufacturing <1 >99
Lifetime Cost Software Engg % Other Engg %
Develolpment ~50 >99
Testing ~50 <1
Maintenance Cost Software Engg % Other Engg %
Corrective ~20 >99
Adaptive ~80 <1
(Corrective) Mntn Cost Software Engg % Other Engg %
Design ~100 ~1
Wear and Tear ~0 ~99

A detour into costs
• Possible reasons for the differences
• Extent of reuse
• Extent of repeatability
• Extent of automation
• Limited quality control
• Digital vs Physical
• Build one instance vs build multiple instances

Testing Taxonomy
• Black-box Testing
• White-box Testing
What is the basis of Testing?

Testing Taxonomy
• Tests are based on speciﬁcation
• Tests are based on implementation

Testing Taxonomy
• Tests are based on speciﬁcation
• Independent of implementation
• Cannot deal with all exhibited (observable)
behaviors
• Tests are based on implementation
• Independent of speciﬁcation
• Cannot deal with unsupported behaviors
• Leads to brittle tests

Testing Taxonomy
• Unit Testing
• Integration Testing
• System Testing
• Acceptance Testing
What is the granularity of Testing?

Testing Taxonomy
• Unit Testing (granularity)
• Test a unit (e.g., function, class) of the system
• A sorting function
• Integration Testing (granularity)
• Test the modules can interact as intended
• Mars Climate Rover mission in 1999: English units vs Metric units
• System Testing (granularity)
• Test the system as a whole to ﬁnd as many faults before
releasing it to the user (development team-centric)
• Acceptance Testing (granularity)
• Test the system as a whole to ensure it satisﬁes the
requirements (user-centric)

Testing Taxonomy
• Reliability Testing
• Safety Testing
• Security Testing
• Performance Testing
• Stress Testing
• Interoperability Testing
• Compatibility Testing
• Installation Testing
• Serviceability Testing
• Usability Testing
• ….
- Software Testing: Concepts and Operations by Ali Mili and Fairouz Tchier (Chapter 7)
- The Art of Software Testing by Glenford J Myers
What aspect is being tested?

Realizing Testing
What do we need to realize any kind of testing?

Realizing Testing
• Expected outcome
• Ability to observe concerned actual outcome
• Ability to compare outcomes for deviations
• Compare expected outcome and observed
outcome for deviations
What else can make it better?

Realizing Testing
• Expected outcome
• Ability to observe concerned actual outcome
• Ability to compare outcomes for deviations
• Compare expected outcome and observed
outcome for deviations
What else can make it better?
• Automatic test execution
• Automatic test data generation
• Automatic test case generation

Testing concepts [3] - Software Testing Techniques (CIS640)

More Related Content

What's hot

Similar to Testing concepts [3] - Software Testing Techniques (CIS640)

More from Venkatesh Prasad Ranganath

Recently uploaded

In this document

Testing concepts [3] - Software Testing Techniques (CIS640)