Introduction Lecture # 1

 This course deals with a very important aspect of software engineering: quality assurance of software
products and services
 We’ll learn different aspects of software quality assurance in this course

What is Quality?
 Can you define quality?
 You must be thinking, what kind of question is that. It is very easy to define quality, but if you think really
hard, it is not that easy to define quality
 Have you come with a definition? Let’s see what I have in store for you

Synonyms of Quality
 Excellence  Superiority  Class
 Eminence  Value  Worth

Antonym of Quality
 Inferiority

Marketability of Quality
 Everyone claims to manufacture / develop / sell / market “good” quality products / services
 You will never come across a person or company selling products or services as low or poor quality
products, even when they are

Software Quality - 1
 Quality as it relates to all aspects of software (requirements / design / code / tests / documents / training)
 Difficult to define
 Software quality is somewhat like the concept of beauty. Each of us has a strong opinion about what
constitutes beauty, and we recognize it when we see it. But when asked to explain exactly why we regard
an object as beautiful, it is hard to put the factors into words

Software Quality - 2
 Good software quality characteristics can be identified
 Bad or undesirable characteristics can also be identified

Software Quality Definitions

 Now we’ll discuss six key factors, which are considered as definitions of software quality, and we’ll use
them throughout this course

Software Quality
 Low levels of defects when deployed, ideally approaching zero
 High reliability, or the capability of running without crashes or strange results
 A majority of clients with high user-satisfaction when surveyed
 A structure that can minimize “bad fixes” or insertion of new defects during repairs
 Effective customer support when problems do occur
 Rapid repairs for defects, especially for high-severity defects
Beyond Absence of Defects
 Sense of beauty  Sense of fitness for purpose
 Sense of elegance that goes beyond the simple absence of overt flaws
 Has well-formed requirements  Robust

Why Software Quality?

 Reduces time to market for new products
 Enhances market share compared to direct competitors
 Minimizes “scrap and rework” expenses
 Attracts and keeps “top-gun” personnel
 Minimizes the risk of serious litigation
 Minimizes the risk of serious operating failures and delays
 Minimizes the risk of bankruptcy or business failures, which may be attributed directly to poor quality or
poor software quality

Software Quality Assurance

 So the term software quality assurance would mean that the software guarantees high quality
 In this course, we’ll learn the different processes, techniques, and activities, which enables us – the
software professionals – to provide that guarantee to ourselves and our clients

Achieving Software Quality

 “For a software application to achieve high quality levels, it is necessary to begin upstream and ensure that
intermediate deliverables and work products are also of high quality levels. This means that the entire
process of software development must itself be focused on quality”
 Capers Jones

Software Defects – I Lecture # 2

 Software defects and related issues
 Have a good discussion on this topic before jumping directly on to the topic

What is a Software Defect?

 A software defect is an error, flaw, mistake, failure, or fault in software that prevents it from behaving as
intended (e.g., producing an incorrect or unexpected result)
 Software defects are also known as software errors or software bugs

Effects of Software Defects - 1

 Bugs can have a wide variety of effects, with varying levels of inconvenience to the user of the software.
Some bugs have only a subtle effect on the program’s functionality, and may thus lie undetected for a long
time. More serious bugs may cause the software to crash or freeze leading to a denial of service
 Others qualify as security bugs and might for example enable a malicious user to bypass access controls in
order to obtain unauthorized privileges
 The results of bugs may be extremely serious
 In 1996, the European Space Agency’s US $1 billion prototype Arian 5 rocket was destroyed less than a
minute after launch, due a bug in the on-board guidance computer program
 In June 1994, a Royal Air Force Chinook crashed into the Mull of Kintyre, killing 29 people. An investigation
uncovered sufficient evidence to convince that it may have been caused by a software bug in the aircraft’s
engine control computer
 In 2002, a study commissioned by the US Department of Commerce’ National Institute of Standards and
Technology concluded that software bugs are so prevalent and detrimental that they cost the US economy
and estimated US $59 billion annually, or about 0.6 percent of the gross domestic product

Categories of Software Defects

 Errors of commission  Errors of omission
 Errors of clarity and ambiguity  Errors of speed or capacity

Errors of Commission
 Something wrong is done
 A classic example at the code level would be going through a loop one time too many or branching on the
wrong address

Errors of Omission
 Something left out by accident
 For example, omitting a parentheses in nested expressions

Errors of Clarity and Ambiguity

 Different interpretations of the same statement
 This kind of error is common with all natural language requirements and specification documents and user
manuals, too.

Errors of Speed and Capacity

 Application works, but not fast enough
 Software defects can be found in any of the documents and work products including very serious ones in
cost estimates and development plans
 However, there are seven major classes of software work products where defects have a strong probability
of triggering some kind of request for warranty repair if they reach the field
Software Defect Origins
 Errors in Requirements  Errors in Design  Errors in Source code
 Errors due to “Bad fixes”  Errors in Data and Tables  Errors in Test Cases
 Errors in User Documentation

Defect Discovery
 Defects are discovered by developers & testers (usually) before release
 Defects are discovered by customers and users (usually) after release
 Defects discovered after release can be embarrassing for the development team

Defect Discovery by Customers

Rule 1: Defect discovery is directly related to the number of users
Rule 2: Defect discovery is inversely related to the number of defects

Software Defect Elimination Strategies

 Effective defect prevention
 High levels of defect removal efficiency
 Accurate defect prediction before the project begins
 Accurate defect tracking during development
 Useful quality measurements
 Ensuring high levels of user-satisfaction

Defect Prevention and Removal

 Both defect prevention and removal techniques are used by the “best-in-the-class” companies
 Defect prevention is very difficult to understand, study, and quantify.
 Both non-test and testing defect removal techniques must be applied

Typical Defect Removal

 Inspections  Direct fault detection and removal
 Testing  Failure observation and fault removal

Inspections - 1
 Inspections are critical examinations of software artifacts by human inspectors aimed at discovering and
fixing faults in the software systems
 Inspections are critical reading and analysis of software code or other software artifacts, such as designs,
product specifications, test plans, etc.
 Inspections are typically conducted by multiple human inspectors, through some coordination process.
Multiple inspection phases or sessions may be used
 Faults are detected directly in inspection by human inspectors, either during their individual inspections or
various types of group sessions
 Identified faults need to be removed as a result of the inspection process, and their removal also needs to
be verified

Non-Test Defect Removal Methods

 Requirement inspections  Design inspections  Code inspections
 Test plan reviews  User documentation editing or reviews
 Test-case inspections

Testing Defect Removal Methods

 Unit test by individual programmers  New function testing
 Regression testing  Performance testing
 Integration testing  System testing
 Field test (external beta test)

Defect Removal
 Not all defects are equal when it comes to removal
 Requirements errors, design problems, and “bad fixes” are particularly difficult
Software Defect Origins & Defect Removal Effectiveness

Defect Removal Efficiency

 Accumulation of defect statistics for errors found prior to delivery, and then for a predetermined period
after deployment (usually one year)
 US averages: 85%
 Best projects in best US companies: 99%

Lecture # 4
 Reasons for poor software quality
 Characteristics of quality laggard companies
 Quality attributes
 Some of the reasons for having high quality software have been discussed in the first lecture of this course,
and so it should be well understood now why software products and services should have high quality
 There are negative consequences for poor or bad quality software
 But still we see that the software industry still suffers from problems related to software quality
 Now we’ll look at six root causes of poor software quality, and discuss them in detail

Root Causes of Poor Software Quality - 1

 Inadequate training of managers and staff
 Inadequate defect and cost measurement
 Excessive schedule pressure
 Insufficient defect removal
 High complexity levels
 Ambiguous and creeping requirements and design (feature race & gimmicks)
 We have just finished the discussion on root causes of poor software quality
 Now let us look at the status of the software industry’s seriousness of the software industry with respect
to the software quality assurance

Quality Assurance Organizations

 No quality assurance 60%  Token quality assurance 20%
 Passive quality assurance 15%  Active quality assurance 5%

 There is another point that must be remembered that software varies from industry to industry
 The focus on software quality naturally is dependent on the industry, as well as the importance of the
software application. More critical applications, naturally, need to have higher software quality than
others

Software Quality in Six Sub- Industries

 Systems software that controls physical devices
 Information systems that companies build for their own use
 Outsource or contract software built for clients
 Commercial software built by vendors for lease or sale
 Military software built following various military standards
 End-user software built for private use by computer literate workers or managers

Characteristics of Quality Laggards

 We’ll now discuss the characteristics of companies, which produce poor quality software

Quality Laggards – (1-5)

1. No software quality measurement program of any kind
2. No usage of formal design and code inspections
3. No knowledge of the concepts of defect potentials and defect removal efficiency
4. Either no quality assurance group or a group that is severely understaffed
5. No trained testing specialists available
6. Few or no automated quality assurance tools
7. Minimal or no software project management tools available
8. No automated risk assessment or avoidance capability
9. From a low of one to a high of perhaps four distinct testing stages
10. No test library or test-case management tools available
11. No complexity analysis tools utilized
12. Defect potentials averaging more than 6 defects per function point
13. Defect removal efficiency averaging less than 80%
14. Executive and managerial indifference (and ignorance) of quality matters

Other Related Issues

 Staff morale and voluntary attrition
 Market shares and competitive positioning
 Litigation and product recalls
 Quality laggards are the very companies with highest probability of cancelled projects, several schedule
overruns, and severe overruns
 It is no coincidence that software groups among the quality laggards also tend to be candidates for
immediate replacement by outsource organizations

Categories of Quality Attributes

 Product-specific quality attributes  Organization-specific quality attributes

Product-Specific Attributes - 1
 Ease of use  Documentation  Defect tolerance
 Defect frequency  Defect impact  Packaging
 Price versus reliability  Performance

Organization-Specific Attributes
 Service and support  Internal processes

Achieving High Levels of Software Quality - 1

 Enterprise-wide quality programs  Quality awareness and training methods
 Quality standards and guidelines  Quality analysis methods
 Quality measurement methods
 Defect prevention methods  Non-test defect removal methods
 Testing methods  User-satisfaction methods
 Post-release quality control

Best in Class Quality Results – (1-2)

 Quality measurements  Defect prevention
 Defect and quality estimation automation  Defect tracking automation
 Complexity analysis tools  Test coverage analysis tools
 Formal inspections  Formal testing by test specialists
 Formal quality assurance group  Executive and managerial understanding of quality

Two Components of
 Software Quality Improvement
 Reductions in total defect potentials using methods of defect prevention
 Improvements in cumulative removal efficiency levels

Lecture # 5 and 6
Quality Measurement Questions
 What should be measured for quality?
 How often should quality measurement be taken and reported?

Quality Measurement Categories

 Measurement of defects or bugs in software
 100% of software projects
 Measurement of user-satisfaction levels
 Only for software projects where clients can be queried and actually use the software consciously

Software Defect Quality Measurements – (1-3)

 Defect volumes (by product, by time period, by geographic region)
 Defect severity levels
 Special categories (invalid defects, duplicates, un-duplicable problems)
 Defect origins (i.e., requirements, design, code, documents, or bad fixes)
 Defect discovery points (i.e., inspections, tests, customer reports, etc.)
 Defect removal efficiency levels
 Normalized data (i.e., defects per function point or per KLOC)
 Causative factors (i.e., complexity, creeping requirements, etc.)
 Defect repair speeds or intervals from the first report to the release of the fix

Software User-Satisfaction Quality Measurements – (1-2)

 User perception of quality and reliability  User perception of features in the software product
 User perception of ease of learning  User perception of ease of use
 User perception of customer support  User perception of speed of defect repairs
 User perception of speed of adding new features  User perception of the value versus the cost of the
package
Who Measures User-Satisfaction?
 Marketing or sales organization of the software company
 User associations  Software magazines  Direct competitors
 User groups on the internet, etc.  Third-party survey groups

Gathering User-Satisfaction Data

 Focus groups of customers  Formal usability laboratories  External beta tests
 Requests from user associations for improvements in usability
 Imitation of usability features of competitive or similar products by other vendors

Barriers to Software Quality Measurement

 Lack of understanding of need to measure quality
 Often technical staff shies away from getting their work measured
 Historically, “lines of code” or LOC and “cost per defect” metrics have been used, which are a poor way of
measuring software quality
Object-Oriented Quality Levels
 OO technology is being adopted world-wide with a claim that it produces better quality software products
 OO technology has a steep learning curve, and as a result it may be difficult to achieve high quality
software
 More data needs to be reported  UML may play a significant role

Orthogonal Defect Reporting – (1-2)

 The traditional way of dealing with software defects is to simply aggregate the total volumes of bug
reports, sometimes augmented by severity levels and assertions as to whether defects originated in
requirements, design, code, user documents, or as “bad fixes”
 In the orthogonal defect classification (ODC) method, defects are identified using the following criteria
 Detection method (design/code inspection, or any of a variety of testing steps)
 Symptom (system completely down, performance downgraded, or questionable data integrity)
 Type (interface problems, algorithm errors, missing function, or documentation error)
 Trigger (start-up / heavy utilization / termination of application, or installation)
 Source (version of the projects using normal configuration control identification)

Outsourcing and Software Quality

 Outsourcing in software industry is done in a variety of ways
 Every situation introduces new challenges for development of high quality software
 Software quality metrics must be mentioned in the outsourcing contract

Quality Estimating Tools – (1-5)

 Estimating defect potentials for bugs in five categories (requirements, design, coding, documentation, and
bad fixes)
 Estimating defect severity levels into four categories, ranging from 1 (total or catastrophic failure) to
severity 4 (minor or cosmetic problem)
 Estimating the defect removal efficiency levels of various kinds of design reviews, inspections, and a dozen
kinds of testing against each kind and severity of defects
 Estimating the number and severity of latent defects present in a software application when it is delivered
to users
 Estimating the number of user-reported defects on an annual basis for up to 20 years
 Estimating the reliability of software at various intervals using mean-time to failure (MTTF) and/or mean-
time between failures (MTBF) metrics
 Estimating the “stabilization period” or number of calendar months of production before users can
execute the application without encountering severe errors.
 Estimating the efforts and costs devoted to various kinds of quality and defect removal efforts such as
inspections, test-case preparation, defect removal, etc.
 Estimating the number of test cases and test runs for all testing stages
 Estimating maintenance costs for up to 20 years for fixing bugs (also for additions)
 Estimating special kinds of defect reports including duplicates and invalid reports which trigger
investigative costs but no repair costs
Quality Process Metrics
 Defect arrival rate  Test effectiveness  Defects by phase
 Defect removal effectiveness  Defect backlog  Backlog management index
 Fix response time  Percent delinquent fixes  Defective fixes

Product Metrics
 Defect density  Defects by severity  Mean time between failures
 Customer-reported problems  Customer satisfaction

Function Point Metric – (1-4)

 It was developed at IBM and reported to public in 1979
 It is a way of determining the size of a software application by enumerating and adjusting five visible
aspects that are of significance to both users and developers
 Inputs that enter the application (i.e., Input screens, forms, commands, etc.)
 Outputs that leave the application (i.e., Output screens, reports, etc.)
 Inquiries that can be made to the application (i.e., Queries for information)
 Logical files maintained by the application (i.e., Tables, text files, etc.)
 Interfaces between the application and others (i.e., shared data, messages, etc.)
 Once the raw total of these five factors has been enumerated, then an additional set of 14 influential
factors are evaluated for impact using a scale that runs from 0 (no impact) to 5 (major impact)

Litigation and Quality

 Relevant factors for software quality
- Correctness, reliability, integrity, usability, maintainability, testability, understandability
 Irrelevant factors for software quality
- Efficiency, flexibility, portability, reusability, interoperability, security
 It is important to narrow down the scope of quality definition similar to hardware warranties

Schedule Pressure and Quality

 Healthy pressure
- Motivates and keeps morale of the personnel high
 Excessive pressure
- Has serious negative impact on the morale of personnel
- Can lead to low quality software
 Project life cycle quality assurance activities are process oriented, in other words, linked to completion of a
project phase, accomplishment of a project milestone, and so forth
 The quality assurance activities will be integrated into the development plan that implements one or more
software development models – waterfall, prototyping, spiral, etc.
 The SQA planners for a project are required to determine
- The list of quality assurance activities needed for a project
- For each quality assurance activity
 Timing
 Who performs the activity and the resources needed
 Resources required for removal of defects and introduction of changes
A Word of Caution
 Some development plans, QA activities are spread throughout the process, but without any time allocated
for their performance or for the subsequent removal of defects. As nothing is achieved without time, the
almost guaranteed result is delay, caused by “unexpectedly” long duration of the QA process
 Hence, the time allocated for QA activities and the defects corrections work that follow should be
examined

 The intensity of the quality assurance activities planned, indicated by the number of required activities, is
affected by project and team factors

Project Factors
 Magnitude of the project  Technical complexity and difficulty
 Extent of reusable software components  Severity of failure outcomes if the project fails

Team Factors
 Professional qualification of the team members
 Team acquaintance with the project and its experience in the area
 Availability of staff members who can professionally support the team
 Familiarity with team members, in other words the percentage of new staff members in the team

Why Error-Prone Modules?

 Excessive schedule pressure on the programmers
 Poor training or lack of experience in structured methods
 Rapidly creeping requirements which trigger late changes
 High complexity levels with cyclomatic ranges greater than 15

“Good Enough” Software Quality – (1-2)

 Rather than striving for zero-defect levels or striving to exceed in 99% in defect removal efficiency, it is
better to ship software with some defects still present in order to speed up or shorten time to market
intervals
 Developed by the fact that major commercial software companies have latent software bugs in their
released products
 Major commercial software companies have cumulative defect removal efficiency of 95% (and 99% on
their best projects)
 This concept is very hazardous for ordinary companies, which usually have their defect removal efficiency
level between 80%-85%
 Quality will be decrease for these companies

Data Quality – (1-2)

 Extremely important to understand issues of data quality
 Data results in (useful | useless) information
 Usually, governments are holders of largest data banks (are they consistent?)
 Companies are increasingly using data to their advantage over competitors
 Data warehouses present a unique challenge to keep data consistent
 Another problem is the interpretation of data
Summary
1. In today’s lecture, we have only discussed what quality is and what software quality is
We have briefly touched upon the need of software quality
In the coming lectures, we will explore software quality assurance in quite a bit of detail, so get ready for a
very exciting course
2. In today’s lecture, we talked about software defects and where are they introduced in the software
product
We discussed the approaches to eliminating these defects
3.

References
1. Software Quality: Analysis and Guidelines for Success by Capers Jones
2. Software Quality: Analysis and Guidelines for Success by Capers Jones
Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement by Jeff Tian
3. Software Quality: Analysis and Guidelines for Success by Capers Jones
Customer-Oriented Software Quality Assurance by Frank Ginac
4. Software Quality: Analysis and Guidelines for Success by Capers Jones
5. Customer-Oriented Software Quality Assurance by Frank Ginac
A Practitioner’s Approach to Software Engineering by Roger Pressman