Software Testing
ER/CORP/CRS/SE04/003 E&R Infosys 1
Objectives
Quality – a culture
Software Testing Life Cycle
Software Testing Techniques
Types of testing
Introduction to Test automation and Test tools
Test Planning and Strategizing
ER/CORP/CRS/SE04/003 E&R Infosys 2
Agenda
What is quality?
What is testing?
Testing life cycle
Testing Techniques
Types of Testing
Test Automation
Test Tools
ER/CORP/CRS/SE04/003 E&R Infosys 3
What is Quality?
“The degree to which a system, component or
process meets requirements”
or
“The degree to which a system, component or
process meets customer or user needs or
expectations”
-I.E.E.E.
ER/CORP/CRS/SE04/003 E&R Infosys 4
Quality is…
What the customer says it is…
Features
Cost
Time
Not a goal, but a means for continuous
improvement
ER/CORP/CRS/SE04/003 E&R Infosys 5
Dimensions of Software Quality
Functionality
Completeness
Correctness
Compatibility
Performance
Time
Resources
Maintainability
Correctability
Expandability
Testability
ER/CORP/CRS/SE04/003 E&R Infosys 6
Dimensions of Software Quality (contd.)
Portability
Hardware independence
Interoperability
Reliability
Error tolerance
Availability
Usability
Understandability
Learnability
Operability
Communicativeness
ER/CORP/CRS/SE04/003 E&R Infosys 7
Software Quality Issues
Challenges
Challenges Problem
Problem Symptoms
Symptoms
Scope User
User or
or Business
Business needs
needs not
not
ScopeCreep
Creep//Churning
Churning
requirements Met
Met
requirements
Fuzzy
Fuzzy Requirements
Requirements Poor
Poor End-user
End-user Experience
Experience
Lack
Lack of
of Skilled
Skilled Resources
Resources Difficult-to-Maintain
Difficult-to-Maintain
Systems
Systems
Lack
Lack of
of aa most
most suitable
suitable process
process
Build
Build and
and Release
Release Issues
Issues
Flaws
Flaws in
in Design
Design
Modules
Modules Don’t
Don’t Integrate
Integrate
Incomplete
Incomplete Test
Test Planning
Planning
Poor
Poor Performance
Performance under
under
Late
Late Testing
Testing Load
Load
Unreliable
Unreliable Data
Data for
for Testing
Testing Flaws
Flaws Discovered
Discovered Very
Very Late
Late
Improper
Improper Defect
Defect Tracking
Tracking Defect
Defect Leakage
Leakage
ER/CORP/CRS/SE04/003 E&R Infosys 8
Testing – What Is it?
A process of demonstrating that errors are not
present?
OR
A way of establishing confidence that a program
does what it is supposed to do?
OR
A means of achieving an error-free program by
finding all errors?
ER/CORP/CRS/SE04/003 E&R Infosys 9
Testing is…
A process of executing a program with the intent
of finding errors
A “DESTRUCTIVE”, yet creative process
ER/CORP/CRS/SE04/003 E&R Infosys 10
Why Testing?
Verifies that all requirements are implemented
correctly (both for positive and negative
conditions)
Identifies defects before software deployment
Helps improve quality and reliability
Makes software predictable in behavior
Reduces incompatibility and interoperability
issues
Helps marketability and retention of customers
ER/CORP/CRS/SE04/003 E&R Infosys 11
Cost of Ineffective Testing
Time
Late Releases
Projects need to be reworked or abandoned
Money
Budget over-runs
Defects are 100 to 1000 times more costly to find and
repair after deployment
Quality
Developers unsure of product quality
Products released with undiscovered or unresolved
defects
ER/CORP/CRS/SE04/003 E&R Infosys 12
Testing Lifecycle
Development Lifecycle Iteration n
Project Requirements Analysis & Implementation
Planning Capture Design
Build 0 Build 1 Build 2
Test Lifecycle
Execute
Plan Design Setup
Test
Test Test Test Bed
Evaluate
Test
Defect Tracking/Enhancement Request
ER/CORP/CRS/SE04/003 E&R Infosys 13
Testing Lifecycle
Requirements
Capture
Analysis The Scenarios Design and the Test Case
Development could normally start in
parallel with the Development Cycle.
Test planning & Test Execution Synchs up with the
Scenario Design Development Cycle during the
functional Testing Phases
New Version Test Case
Development
Defect Fixing
Test Execution
Cycle
Defects Test Cycle
Closure
Test Result
Analysis
ER/CORP/CRS/SE04/003 E&R Infosys 14
Testing Lifecycle
Test requirements identification
Validate for testability
Test Planning and Scenario design
Develop Test Objectives
Identify Test Items
Resources and Schedules
Test Case Development
Test Case Specification
Pre-requisites, post-requisites and Acceptance criteria
ER/CORP/CRS/SE04/003 E&R Infosys 15
Testing Lifecycle (contd.)
Prepare Test Bed
Test Data Preparation
Test Environment Setup
Test Execution
Run Tests and Validate Results
Bug Reporting
Bug fixes and retesting
Test Result Analysis
Defect Analysis
Determining Test Set Coverage and Effectiveness
ER/CORP/CRS/SE04/003 E&R Infosys 16
Start Testing Early
There are two basic forms of testing
Verification Testing
Validation Testing
Verification: “Are we building it right?”
Validation: “Are we building the right thing?”
ER/CORP/CRS/SE04/003 E&R Infosys 17
Testing Stages – the V-Model
Requirement Analysis User Acceptance Testing
Functional Specifications System Testing
High-Level Design Integration Testing
Reviews
Verification Validation
Detailed Design Unit Testing
Code
ER/CORP/CRS/SE04/003 E&R Infosys 18
Testing Techniques
Static Testing
Dynamic Testing
White Box Testing
Black Box Testing
ER/CORP/CRS/SE04/003 E&R Infosys 19
Testing Techniques (contd.)
Static Testing/Reviews
Scope
Requirements Feasibility
Technical Architecture
Design
Program specifications
Test Plans
Test Cases and Data
User Documents
ER/CORP/CRS/SE04/003 E&R Infosys 20
Testing Techniques (contd.)
Static Testing
Advantages
Capture defects early, so saves rework cost
Checklist-based approach
Focuses on coverage
Group perspective
Highest probability of finding defects
Disadvantages
Time-consuming
Cannot test data dependencies
High skill levels required
ER/CORP/CRS/SE04/003 E&R Infosys 21
Testing Techniques (contd.)
Dynamic Testing
White-box/Glass-box Testing
Unit Testing
Integration/Interface Testing
Black-box Testing
System Testing
Acceptance testing
ER/CORP/CRS/SE04/003 E&R Infosys 22
White-Box/Glass-box Testing
Structural Testing process
Program Logic-driven Testing
Design-based Testing
Examines the internal structure of program
ER/CORP/CRS/SE04/003 E&R Infosys 23
White-Box Testing – Advantages
High Code coverage – Exhaustive (thorough)
path testing
Program logic is tested
Internal boundaries are tested
Testing is algorithm specific
ER/CORP/CRS/SE04/003 E&R Infosys 24
Black-box Testing
Specifications-based Testing
Business Transaction-driven Testing
No concern to internal behavior and program
structure
ER/CORP/CRS/SE04/003 E&R Infosys 25
Black-box Testing – Advantages
User’s perspective
Focus on features, not implementation
Big-picture approach
ER/CORP/CRS/SE04/003 E&R Infosys 26
Levels of Testing
Unit Testing
Integration Testing
}
Function Testing
Performance Testing System
Acceptance Testing Testing
Installation Testing
ER/CORP/CRS/SE04/003 E&R Infosys 27
Unit Testing
E&R Infosys 28
Testing Stages – the V-model
Requirement Analysis User Acceptance Testing
Functional Specifications System Testing
High-Level Design Integration Testing
Reviews
Verification Validation
Detailed Design Unit Testing
Code
ER/CORP/CRS/SE04/003 E&R Infosys 29
What is Unit Testing?
Testing of individual units or groups of related
code is known as Unit Testing
Unit tested with a Unit Test Plan
ER/CORP/CRS/SE04/003 E&R Infosys 30
Unit Test Plan should test for:
Standards specified (Ex. GUI related)
Field validations, boundary values
Basic functionality of the component
(add/delete/modify/query activities)
Code coverage and exception handling
Negative or destructive testing
Memory leaks
Data computation/integrity
ER/CORP/CRS/SE04/003 E&R Infosys 31
Unit Test Plan should test for:
Usability
Standards specified (Ex. GUI related)
Field validations, boundary values
Basic functionality of the component
Functionality
(add/delete/modify/query activities)
Code coverage and exception handling
Negative or destructive testing
Memory leaks Performance
Data computation/integrity
ER/CORP/CRS/SE04/003 E&R Infosys 32
Unit Testing – White Box Technique
Activities
Identification of Testable units/modules
Focus on code and logic
Design logic-based Test cases and prepare Test data
Run tests and verify output through path traversing
ER/CORP/CRS/SE04/003 E&R Infosys 33
Unit Testing – White Box Technique
Logic Coverage method includes:
Statement coverage
Decision (branch) coverage
Condition Coverage
Path Coverage
ER/CORP/CRS/SE04/003 E&R Infosys 34
Unit Testing - Black Box Technique
Commonly used:
Equivalence partitioning
Boundary value analysis
Error Guessing
Less commonly used:
Cause effect graphing
State-transition testing
ER/CORP/CRS/SE04/003 E&R Infosys 35
Equivalence Partitioning
Systematic process of identifying a set of input
conditions to be tested
The two distinct steps involved:
Identify equivalence classes
Identify test cases
ER/CORP/CRS/SE04/003 E&R Infosys 36
Equivalence Partitioning
Equivalence Classes – Valid Data
User supplied command
Responses to system prompts
File names
Computational data
Physical parameters, bounding values, initialization
values
Output data formatting commands
Responses to error messages
Graphical data (e.g. mouse picks)
ER/CORP/CRS/SE04/003 E&R Infosys 37
Equivalence Partitioning
Equivalence Classes – invalid Data
Data outside bounds of the program
Physically impossible data
Proper value supplied in the wrong place
ER/CORP/CRS/SE04/003 E&R Infosys 38
Boundary Value Analysis
Check edge values of all
Output domain
User queries
Numerical data
Mouse picks on menu
Output format requests
Responses to prompts
Command key input
ER/CORP/CRS/SE04/003 E&R Infosys 39
Error Guessing
Ad Hoc method to identify tests likely to expose
errors based on experience and intuition
Some areas to guess are:
Empty or Null lists / strings
Zero instances / Occurrences
Blanks or null characters in strings
Negative numbers
ER/CORP/CRS/SE04/003 E&R Infosys 40
Advantage/Disadvantage
Equivalence Partitioning
Significantly reduces number of input conditions to be
tested
Does not test combinations of input conditions
Boundary Value Analysis
Boundary conditions sometimes subtle and difficult to
identify
Does not test combinations of input conditions
ER/CORP/CRS/SE04/003 E&R Infosys 41
Exercise
Writing a unit test plan…
ER/CORP/CRS/SE04/003 E&R Infosys 42
Recap
What is quality
Dimensions of Quality
Testing Techniques
Unit Test
ER/CORP/CRS/SE04/003 E&R Infosys 43
Integration Testing
E&R Infosys 44
Testing Stages – the V-model
Requirement Analysis User Acceptance Testing
Functional Specifications System Testing
High-Level Design Integration Testing
Reviews
Verification Validation
Detailed Design Unit Testing
Code
ER/CORP/CRS/SE04/003 E&R Infosys 45
Integration Testing
Combining and testing multiple components
together
Integration of modules, programs and functions
Tests Internal Program interfaces
Tests External interfaces for modules
ER/CORP/CRS/SE04/003 E&R Infosys 46
Integration Testing
Banking System External
System
External
interface
CRM HR Financial
Inter
module
Online banking Loans Account Opening
interface
Java Based Based on C, C++ VB based
Intra
module
interface
Unit Module
ER/CORP/CRS/SE04/003 E&R Infosys 47
Approach for testing interfaces
Intra-Module Inter-Module External
Test Techniques White Box Black Box Black Box
Test Development Isolated Simulate live
Environment environment development environment
Setup environment
Test Approach Check for Identify pre-process Identify
parameter / and approach to external /
procedure called. Interface. internal interface
Database Identify the post- and validate
updation process to validate against external
the interface system
ER/CORP/CRS/SE04/003 E&R Infosys 48
Techniques based on order of Integration
Big-bang Integration (non-incremental)
Random Incremental Integration
Top-Down Integration
Bottom-up Integration
ER/CORP/CRS/SE04/003 E&R Infosys 49
Top-down vs. Bottom-up
Top-down Integration
Advantage
Key Interface defects trapped earlier
Disadvantage
Core functionality tested late in the cycle
Bottom-up Integration
Advantage
Core functionality tested early
Disadvantage
Key Interface defects trapped late in the cycle
ER/CORP/CRS/SE04/003 E&R Infosys 50
System Testing
E&R Infosys 51
Testing Stages – the V-model
Requirement Analysis User Acceptance Testing
Functional Specifications System Testing
High-Level Design Integration Testing
Reviews
Verification Validation
Detailed Design Unit Testing
Code
ER/CORP/CRS/SE04/003 E&R Infosys 52
System Testing – What is it?
A Black-box Testing technique
Specifications-based Testing
Typically independent team testing
Simulated environment testing
Live/Simulated user data
Tests the whole system
Functional and non-functional requirements tested
Business transaction-driven testing
Compatibility errors uncovered
Performance limitations uncovered
ER/CORP/CRS/SE04/003 E&R Infosys 53
System Testing – Different Types
Functional testing (Sanity / Regression )
Performance and Scalability testing
Load/Stress testing
Volume Testing
Usability testing
Configuration testing
Installability testing
Disaster and Recovery testing
Reliability / Availability testing
Statistical Testing based on Usage Models
ER/CORP/CRS/SE04/003 E&R Infosys 54
Functional Testing – Sanity Test
Very basic minimal number of tests to verify the
product for the feature / protocol compliance
Could be a subset of regression test used to
validate only high level functionality
Typically an initial testing effort to determine if a
new software version is performing well enough
to accept it for a major testing effort
Sanity testing will be performed in scenarios
when cursory testing is sufficient to prove that
the application is functioning properly
ER/CORP/CRS/SE04/003 E&R Infosys 55
Why Sanity Testing?
Regression testing is more time taking and more
effort is required whereas Sanity testing is
quicker focusing on important functionality of
software
Saves time as only selected high level
functionality areas are only tested. Bugs found
this way would take almost triple the time when
found through regression testing
Efficient in scenarios wherein releases comes in
pretty often with very few changes
ER/CORP/CRS/SE04/003 E&R Infosys 56
Functional Testing – Regression
Re-execution of one or more tests in subsequent
build of the application/product to ensure
Revisiting and testing all prior bug-fixes in response
to a new fix/enhancement
Re-testing all programs that might be affected by the
fix/enhancement
Hidden Bugs are uncovered
The baseline for Regression Testing grows with
every build
ER/CORP/CRS/SE04/003 E&R Infosys 57
Regression Testing
Should Cover
Business Process depending on criticality
User friendliness
Cross functional dependencies
Activities
System appreciation
Preparation of Test case repository
Automate test cases
Execute Regression test suite
ER/CORP/CRS/SE04/003 E&R Infosys 58
Without Regression Testing – Hidden Bugs
Software Builds
A B C
Unknown Unknown
Test PASS PASS? PASS?
Requirement 1
Fix Fix
Test FAIL FAIL PASS
Requirement 2
Test PASS BUG? BUG?
Requirement 3
New Hidden
Problem Bug
ER/CORP/CRS/SE04/003 E&R Infosys 59
With Regression Testing – No Hidden Bugs
Software Builds
A B C
Test PASS PASS PASS
Requirement 1
Fix Fix
Test FAIL FAIL PASS
Requirement 2
Fix
Test PASS BUG PASS
Requirement 3
New
Problem
ER/CORP/CRS/SE04/003 E&R Infosys 60
Acceptance Testing
Demonstrates that the system meets mutually
agreed Acceptance criteria
Critical Requirements
Minimum Performance level
Maximum Defect Detection Rate
Typically, a sub-set of System testing
ER/CORP/CRS/SE04/003 E&R Infosys 61
Alpha / Beta Testing
Forms of Acceptance testing
Testing in the production environment
Alpha testing is performed by end users within a
company but outside development group
Beta testing is performed by a sub-set of actual
customers outside the company
ER/CORP/CRS/SE04/003 E&R Infosys 62
Performance Testing
Number of concurrent users accessing at any point in
given time
System’s performance under high volume of data
Stress testing for systems, which are being scaled up to
larger environments or implemented for the first time
Operational intensive transactions. (Most frequently used
transactions)
Volume intensive transactions (for both volume and
stress testing)
ER/CORP/CRS/SE04/003 E&R Infosys 63
Installation Testing
Basic installation
Installation of various configurations
Installation on various platforms
Regression testing of basic functionality
ER/CORP/CRS/SE04/003 E&R Infosys 64
System Testing
A Banking System
CRM HR Financial
Online banking Loans Account Opening
Java Based Based on C, C++ VB based
ER/CORP/CRS/SE04/003 E&R Infosys 65
When do you stop testing?
When time runs out
Specific test case design methods
Target test coverage attained
Certain number of errors found*
Error detection rate drops below a threshold*
Predicted density of errors left drops below a threshold
Sequence test cases in order of priority
* Infosys Process Capability Baseline
ER/CORP/CRS/SE04/003 E&R Infosys 66
Test Artifacts
Test Plan
Test Environment/Bed
Test Case
Test Data
Test Tools
Test Scripts
Test Log
Bug Reports
ER/CORP/CRS/SE04/003 E&R Infosys 67
Traits of a good Tester
Destructive creativity
Detective skills
Appreciating the users’ perspective
Adapt to and Understand Requirements change
A skeptical, but not hostile attitude
An eagerness to embrace new technologies
ER/CORP/CRS/SE04/003 E&R Infosys 68
Test Engineers
Role - Execute tests, record test scripts,
maintain statistics and metrics, check test data
setup, test environment setup, execute re-tests
Knowledge - Understanding of the importance of
testing, awareness of tools, how to progress
against a plan
Skills - Observation, accuracy, methodical, co-
ordination, problem solver
Experience - Following instructions, problem
reporting and solving, and relevant testing tools
ER/CORP/CRS/SE04/003 E&R Infosys 69
Test Automation
Good candidates:
Complex and time-consuming tests
Tests requiring a great deal of precision
Tests involving many simple, repetitive tests
Tests involving many data combinations
Lower priority candidates:
One-time only tests
Testing peripheral devices
Subjective assessment tests (look and feel based)
ER/CORP/CRS/SE04/003 E&R Infosys 70
Test Automation
Effective
Automated Test
(After many runs)
Manual Test
Economic Evolvable
First Run of
Automated Test
Exemplary
ER/CORP/CRS/SE04/003 E&R Infosys 71
Simple Vs Sophisticated Automation
Maintenance Cost
Implementation Cost
C
O
S
T
Simple Sophisticated
Automation Automation
ER/CORP/CRS/SE04/003 E&R Infosys 72
Testing Tools
E&R Infosys 73
Types of Testing Tools
The software testing tools broadly of the
following type:
Functional / regression
Load /Stress: Performance testing under high load /
stress
Web Testing: testing the Web Sites
Data Generation: Generation of test data
ER/CORP/CRS/SE04/003 E&R Infosys 74
Benefits
Reduces manpower and time
More coverage within the same testing time.
Testing products which are very difficult to test
manually. Example, Load Tests
Can look inside the software for memory leaks,
redundant code etc.
ER/CORP/CRS/SE04/003 E&R Infosys 75
Test Tools (Client Server)
Category Tool Vendor
Source Code Testing BoundsChecker Numega
Pure Coverage Rational
Purify Rational
Jprobe Sitraka Software
ATTOLCoverage ATTOL Software
Functional Testing QA Run Compuware
Win Runner Mercury
Team Test Rational
QA Center Compuware
Performance Testing OptimizeIT VM Gear
Load Runner Mercury
QA Load Compuware
ER/CORP/CRS/SE04/003 E&R Infosys 76
Test Tools (Web Applications)
Category Tool Vendor
Functional Testing Rational Robot Rational
Silk Test Segue Software
Performance Testing Silk Performer Segue Software
Web Load Radview
Link & HTML Testing NetMechanic Monte Sano
Linkbot Tetranet
Site Inspector
Doctor HTML Imageware
ER/CORP/CRS/SE04/003 E&R Infosys 77
Test Tools (Network & Security)
Category Tool Vendor
Security & Communication SNMP Test Suite Interworking Labs Inc.
Sniffer Pro Network Associates Inc.
Lophtcrack
Saint, Satan, Sara SwTech
TCP Wrappers
TCP Dump
ER/CORP/CRS/SE04/003 E&R Infosys 78
Going Forward…
E&R Infosys 79
Testing Lifecycle
Development Lifecycle Iteration n
Project Requirements Analysis & Implementation
Planning Capture Design
Build 0 Build 1 Build 2
Test Lifecycle
Execute
Plan Design Setup
Test
Test Test Test Bed
Evaluate
Test
Defect Tracking/Enhancement Request
ER/CORP/CRS/SE04/003 E&R Infosys 80
Test Planning/Strategizing
Includes –
Testing Objectives and Goals
Test Strategy/Approach based on customer priorities
Test Environment (Hardware, Software, Network,
Communication etc.)
Features to test with priority/criticality
Test Deliverables
Test Procedure – Activities and tools
Test Entry and Exit criteria
Test Organization and Scheduling
Testing Resources and Infrastructure
Test Measurements/Metrics
ER/CORP/CRS/SE04/003 E&R Infosys 81
Test Planning / Strategizing
Benefits
Sets clear and common objectives
Helps prioritize tests
Facilitates Technical tasks
Helps improve coverage
Provides structure to activities
Improves communication
Streamlines tasks, roles and responsibilities
Improves test efficiency
Improves test measurability
ER/CORP/CRS/SE04/003 E&R Infosys 82
Optimizing Testing
Categorize entire gamut of testing into Sanity / Regression
/Performance and Stress
Efficient Review Process
Use Tools to do the testing (both Automation and Simulators )
Look out for Automation if it helps ( leverage 24 hours a day )
Use Traceability Matrix
Tracking and Reporting of bugs and defects – Defining a process
Revisit the Test coverage and categorization on an ongoing basis
Risk strategy
Retest strategy
ER/CORP/CRS/SE04/003 E&R Infosys 83
Recap
Integration Testing
System Testing
Test Automation
Testing Tools
ER/CORP/CRS/SE04/003 E&R Infosys 84
E&R Infosys 85
