Outline Verification, Validation Testing terms Testing: Classification Test Activities

Software Testing: Terminologies

Meenakshi D’Souza

International Institute of Information Technology

Bangalore.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Outline

Verification, validation and testing.

Testing terms.
Testing: Classification.
Test activities.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Software Testing

Software testing is the act of examining the artifacts and the

behavior of the software under test by validation and verification.

Software testing can also provide an objective, independent view of

the software to allow the business to appreciate and understand
the risks of software implementation.

Definition courtesy: Wikipedia.

Outline Verification, Validation Testing terms Testing: Classification Test Activities

Verification, Validation and Testing

As per IEEE-STD-610, the IEEE Standard Glossary of Software

Engineering Terminology,
Validation: The process of evaluating software at the end of
software development to ensure compliance with intended
usage. i.e., checking of the software meets its requirements.
Verification: The process of determining whether the products
of a given phase of the software development process fulfill
the requirements established at the start of that phase.
Testing, as we will see in this course, deals mainly with verification.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Other related areas

Formal methods/verification: Model checking, theorem

proving, program analysis.
Modelling and simulation.
Software quality assurance.
Accreditation.
Test driven development.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Software: Faults, failures and errors

Fault: A static defect in the software. It could be a missing

function or a wrong function in code.
Failure: An external, incorrect behavior with respect to the
requirements or other description of the expected behavior. A
failure is a manifestation of fault when software is executed.
Error: An incorrect internal state that is the manifestation of
some fault.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Historical perspective

The terms bug and fault was used by Edison:

“It has been just so in all of my inventions. The first step is
an intuition, and comes with a burst, then difficulties arise,
this thing gives out and [it is] then that bugs, as such little
faults and difficulties are called, show themselves and months
of intense watching, study and labor are requisite.”
The term error was used by Ada Lovelace:
“An analyzing process must equally have been performed in
order to furnish the Analytical Engine with the necessary
operative data; and that herein may also lie a possible source
of error. Granted that the actual mechanism is unerring in its
processes, the cards may give it wrong orders.”
Outline Verification, Validation Testing terms Testing: Classification Test Activities

In this course...

We will use these terms synonmously in this course:

fault, failure, error, bug, defect ...

will all mean the same.

Outline Verification, Validation Testing terms Testing: Classification Test Activities

Test case

A test case typically involves inputs to the software and

expected outputs.
When test cases are executed and results recorded, if the
actual output matches the expected output, the test case is
said to have passed. Otherwise, the test case is said to have
failed.
A failed test cases indicates an error.
A test case also contains other paramaters like test case id,
traceability details etc.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Types of Testing

There are different types/levels of testing, based on the phase of

software development lifecycle that they are applied to:
Unit Testing: Done by developer during coding.
Integration Testing: Various components are put together and
tested. Components could be only software or software and
hardware components.
System Testing: Done with full system implementation and
the platform on which the system will be running.
Acceptance Testing: Done by end customer to ensure that the
delivered products meets the committed requirements.
Beta testing is done in a (so-called) beta version of the
software, by end users, after release.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Types of testing

System
main class P
testing

class A class B

method m1 method m3 Integration

testing

method m2 method m4

Unit testing
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Types of testing, contd.

Some additional types of testing are (an incomplete list):

Functional Testing: Done to ensure that the software meets
its specified functionality.
Stress Testing: Done to evaluate how the system behaves
under peak/unfavourable conditions/inputs.
Performance Testing: Done to ensure the speed and response
time of the system.
Usability Testing: Done to evaluate the user interface,
aesthetics etc.
Regression Testing: Done after modifying/upgrading a
component, to ensure that the modification is working
correctly and other components are not damaged by the
modification.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Testing Methods

There are two broad methods of testing:

Black-box testing: A method of testing that examines the
functionalities of software/system without looking into its
internal design or code.
White-box testing: A method of testing that tests the internal
structure of the design or code of a software.

Black-box testing White-box testing

Unit, integration, system Unit, integration and
and acceptance testing system testing
Usability, performance, beta
and stress testing
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Types of test activities

Testing can be broken into four activities:

Test design
Test automation
Test execution
Test evaluation
Adjacent activities include test management, test maintenance and
test documentation.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Test design

The most critical job in testing.

Need to design effective test cases— test cases that can find
the errors in the software.
Apart from specifying the inputs, this involves defining the
expected outputs too.
Typically cannot be automated.
Needs knowledge of basic Algorithms, Discrete Mathematics
and the domain of the software.
A part of this course will deal with criteria-based test
case design algorithms.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Test automation

Involves converting the test cases into executable scripts.

Need to specify how to reach deep parts of the code using just
inputs: Observability and Controllability.
Specific to the test execution framework or tool.
We will use a few open source test automation tools in
this course.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Test execution

Involves running the tests on the software and recording the

results.
Can be fully automated. Many good test execution tools are
available, open source and proprietary.
We will use a few open source test automation tools for
test execution.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Test evaluation

Involves evaluating the results of testing, reporting identified

errors.
Difficult problem is to isolate faults, especially in large
software and during integration testing.
Needs domain knowledge and experience.
Outline Verification, Validation Testing terms Testing: Classification Test Activities

Credits

Part of the material used in these slides are derived from the
presentations of the book Introduction to Software Testing, by
Paul Ammann and Jeff Offutt.