Manual Testing
By Prof. Aalia Shaikh
What is Testing?
Software testing assesses software quality and helps reducing the
risk of software failure in operation.
Software testing is a set of activities to discover defects and evaluate
the quality of software artifacts.
Whilst testing involves verification, i.e., checking whether the
system meets specified requirements.
It also involves validation, which means checking whether the
system meets users’ and other stakeholders’ needs in its operational
environment
The typical test objectives are:
Evaluating work products such as requirements, user stories, designs, and code
Triggering failures and finding defects
Ensuring required coverage of a test object
Reducing the level of risk of inadequate software quality
Verifying whether specified requirements have been fulfilled
Verifying that a test object complies with contractual, legal, and regulatory
requirements
Providing information to stakeholders to allow them to make informed decisions
Building confidence in the quality of the test object
Validating whether the test object is complete and works as expected by the
stakeholders
Testing and Debugging
Testing can trigger failures that are caused by defects in the software
(dynamic testing) or can directly find defects in the test object (static
testing).
When dynamic testing triggers a failure, debugging is concerned with
finding causes of this failure (defects), analyzing these causes,
and eliminating them.
The typical debugging process in this case involves:
• Reproduction of a failure
• Diagnosis (finding the root cause)
• Fixing the cause
Why is Testing Necessary?
Testing components, systems, and associated documentation helps to
identify defects in software
Testing provides a cost-effective means of detecting defects
Testing is a form of quality control (QC).
QC is a product-oriented, corrective approach that focuses on those
activities supporting the achievement of appropriate levels of quality.
QA is a process-oriented, preventive approach that focuses on the
implementation and improvement of processes.
It works on the basis that if a good process is followed correctly, then
it will generate a good product
Errors, Defects, Failures, and Root
Causes
Human beings make errors (mistakes),
which produce defects (faults, bugs),
which in turn may result in failures.
A root cause is a fundamental reason for the occurrence of a problem
Root causes are identified through root cause analysis
Testing Principles
1. Testing shows the presence, not the absence of defects
2. Exhaustive testing is impossible
3. Early testing saves time and money.
4. Defects cluster together
5. Tests wear out
6. Testing is context dependent.
7. Absence-of-defects fallacy
Test Activities
Test planning
Test monitoring and control.
Test analysis – what to test
Test design – how to test
Test implementation
Test execution
Test completion
Test Process: the way the testing is carried out
will depend on a number of contextual factors
including:
Stakeholders (needs, expectations, requirements, willingness to cooperate, etc.)
Team members (skills, knowledge, level of experience, availability, training
needs, etc.)
Business domain (criticality of the test object, identified risks, market needs,
specific legal regulations, etc.)
Technical factors (type of software, product architecture, technology used, etc.)
Project constraints (scope, time, budget, resources, etc.)
Organizational factors (organizational structure, existing policies, practices
used, etc.)
Software development lifecycle (engineering practices, development methods,
etc.)
Tools (availability, usability, compliance, etc.)
Testware is created as output work
products from the test activities
Test planning work products include: test plan, test schedule, risk register, and entry and exit
criteria
Risk register is a list of risks together with risk likelihood, risk impact and information about risk
mitigation
Test monitoring and control work products include: test progress reports documentation of control
directives and risk information
Test analysis work products include: (prioritized) test conditions and defect reports regarding
defects in the test basis (if not fixed directly).
Test design work products include: (prioritized) test cases, test charters, coverage items, test
data requirements and test environment requirements.
Test implementation work products include: test procedures, automated test scripts, test suites,
test data, test execution schedule, and test environment elements. Examples of test environment
elements include: stubs, drivers, simulators, and service virtualizations.
Test execution work products include: test logs, and defect reports
Test completion work products include: test completion report action items for improvement of
subsequent projects or iterations, documented lessons learned, and change requests (e.g., as
product backlog items).
Traceability between the Test Basis
and Testware
Accurate traceability supports coverage evaluation
The coverage criteria can function as key performance indicators to drive
the activities that show to what extent the test objectives have been
achieved
Traceability of test cases to requirements can verify that the requirements
are covered by test cases.
Traceability of test results to risks can be used to evaluate the level of
residual risk in a test object.
Good traceability also makes test progress and completion reports more
easily understandable by including the status of test basis elements.
Generic Skills Required for Testing
Testing knowledge (to increase effectiveness of testing, e.g., by using test
techniques)
Thoroughness, carefulness, curiosity, attention to details, being methodical
(to identify defects, especially the ones that are difficult to find)
Good communication skills, active listening, being a team player (to interact
effectively with all stakeholders, to convey information to others, to be
understood, and to report and discuss defects)
Analytical thinking, critical thinking, creativity (to increase effectiveness of
testing)
Technical knowledge (to increase efficiency of testing, e.g., by using
appropriate test tools)
Domain knowledge (to be able to understand and to communicate with end
users/business representatives)
Testing as a Driver for Software
Development
Test-Driven Development (TDD)
Directs the coding through test cases (instead of extensive software design)
(Beck 2003)
Tests are written first, then the code is written to satisfy the tests, and then
the tests and code are refactored
Acceptance Test-Driven Development (ATDD)
Derives tests from acceptance criteria as part of the system design process
(Gärtner 2011)
Tests are written before the part of the application is developed to satisfy the
tests
Behavior-Driven Development (BDD)
Expresses the desired behavior of an application with test cases written in a
simple form of natural language, which is easy to understand by
stakeholders – usually using the Given/When/Then format. (Chelimsky 2010)
Test cases are then automatically translated into executable tests
Retrospectives and Process
Improvement
Retrospectives (also known as “post-project meetings” and project
retrospectives) are often held at the end of a project or an iteration,
at a release milestone, or can be held when needed.
What was successful, and should be retained?
What was not successful and could be improved?
How to incorporate the improvements and retain the successes in the
future?
The results should be recorded and are normally part of the test
completion report
Test Levels
Component testing (also known as unit testing)
focuses on testing components in isolation
Component integration testing (also known as unit integration
testing)
focuses on testing the interfaces and interactions between components
System testing
focuses on the overall behavior and capabilities of an entire system or
product, often including functional testing of end-to-end tasks and the
non-functional testing of quality characteristics
System integration testing
focuses on testing the interfaces of the system under test and other
systems and external services .
Acceptance testing
focuses on validation and on demonstrating readiness for deployment,
which means that the system fulfills the user’s business needs.
Test Types
Functional testing evaluates the functions that a component or
system should perform
Non-functional testing is the testing of “how well the system
behaves”.
Performance efficiency
Compatibility
Usability
Reliability
Security
Maintainability
Portability
Black-box testing The main objective of black-box testing is
checking the system's behavior against its specifications.
White-box testing is structure-based and derives tests from the
system's implementation or internal structure (e.g., code,
architecture, work flows, and data flows).
Confirmation testing (retesting) confirms that an original
defect has been successfully fixed.
Regression testing confirms that no adverse consequences have
been caused by a change, including a fix that has already been
confirmation tested
Differences between Static Testing
and Dynamic Testing
Static Testing Dynamic Testing
Code Actual product
can be applied to non- can only be applied to
executable work products executable work products
Maintainability performance efficiency
Typical defects to find through static
testing
Defects in requirements (e.g., inconsistencies, ambiguities,
contradictions, omissions, inaccuracies, duplications)
Design defects (e.g., inefficient database structures, poor
modularization)
Certain types of coding defects (e.g., variables with undefined values,
undeclared variables, unreachable or duplicated code, excessive code
complexity)
Deviations from standards (e.g., lack of adherence to naming
conventions in coding standards)
Incorrect interface specifications (e.g., mismatched number, type or
order of parameters)
Specific types of security vulnerabilities (e.g., buffer overflows)
Gaps or inaccuracies in test basis coverage (e.g., missing tests for an
acceptance criterion)
Review Process Activities
Planning
Review initiation
Individual review
Communication and analysis
Fixing and reporting
Roles and Responsibilities in
Reviews
Manager – decides what is to be reviewed and provides resources,
such as staff and time for the review
Author – creates and fixes the work product under review
Moderator (also known as the facilitator) – ensures the effective
running of review meetings, including mediation, time management,
and a safe review environment in which everyone can speak freely
Scribe (also known as recorder) – collates anomalies from reviewers
and records review information, such as decisions and new anomalies
found during the review meeting
Reviewer – performs reviews. A reviewer may be someone working on
the project, a subject matter expert, or any other stakeholder
Review leader – takes overall responsibility for the review such as
deciding who will be involved, and organizing when and where the
review will take place
Review Types
Informal review
Walkthrough
Technical Review
Inspection
Black-Box Test Techniques
Equivalence Partitioning
Boundary Value Analysis
Decision Table Testing
State Transition Testing
Equivalence Partitioning
Equivalence Partitioning (EP) divides data into partitions (known as equivalence
partitions)
based on the expectation that all the elements of a given partition are to be
processed
in the same way by the test object
Boundary Value Analysis
Boundary Value Analysis (BVA) is a technique based on exercising the
boundaries of equivalence partitions.
Decision Table Testing
Decision tables are used for testing the implementation of system
requirements that specify how different combinations of conditions
result in different outcomes. Decision tables are an effective way of
recording complex logic, such as business rules.
State Transition Testing
A state transition diagram models
the behavior of a system by
showing its possible states and
valid state transitions.
A transition is initiated by an
event, which may be additionally
qualified by a guard condition.
The transitions are assumed to be
instantaneous and may sometimes
result in the software taking action.
White-Box Test Techniques
White-box techniques can be used in static testing (e.g., during dry
runs of code).
They are well suited to reviewing code that is not yet ready for
execution
Statement testing
Branch testing
Statement Testing
In statement testing, the coverage items are executable statements.
The aim is to design test cases that exercise statements in the code
until an acceptable level of coverage is achieved
When 100% statement coverage is achieved, it ensures that all
executable statements in the code have been exercised at least once.
Branch Testing and Branch
Coverage
A branch is a transfer of control between two nodes in the control
flow graph, which shows the possible sequences in which source code
statements are executed in the test object.
Experience-based Test Techniques
Error Guessing
Exploratory Testing
Checklist-based testing
Acceptance Criteria
Define the scope of the user story
Reach consensus among the stakeholders
Describe both positive and negative scenarios
Serve as a basis for the user story acceptance testing
Allow accurate planning and estimation
Purpose and Content of a Test Plan
Documents the means and schedule for achieving test objectives
Helps to ensure that the performed test activities will meet the
established criteria
Serves as a means of communication with team members and other
stakeholders
Demonstrates that testing will adhere to the existing test policy and
test strategy
The typical content of a test plan
includes:
Context of testing (e.g., scope, test objectives, constraints, test basis)
Assumptions and constraints of the test project
Stakeholders (e.g., roles, responsibilities, relevance to testing, hiring and
training needs)
Communication (e.g., forms and frequency of communication,
documentation templates)
Risk register (e.g., product risks, project risks)
Test approach (e.g., test levels, test types, test techniques, test deliverables,
entry criteria and exit criteria, independence of testing, metrics to be
collected, test data requirements, test environment requirements,
deviations from the organizational test policy and test strategy)
Budget and schedule
Entry Criteria and Exit Criteria
Entry criteria define the preconditions for undertaking a given
activity.
If entry criteria are not met, it is likely that the activity will prove to
be more difficult, time-consuming, costly, and riskier.
Exit criteria define what must be achieved in order to declare an
activity completed.
Entry criteria and exit criteria should be defined for each test level,
and will differ based on the test objectives.
To be continued..
What is Sanity testing?
Sanity testing is testing done at the release level to test the main
functionalities. It’s also considered an aspect of regression testing.
