Cost Benefits Analysis of
Test Automation
STAR West ’99
Douglas Hoffman
Software Quality Methods, LLC.
24646 Heather Heights Place
Saratoga, California 95070-9710
Phone 408-741-4830
Fax 408-867-4550
Copyright © 1999, Software Quality Methods, LLC. No part of these graphic overhead slides may
be reproduced, or used in any form by any electronic or mechanical duplication, or stored in a
computer system, without written permission of the author.
Douglas Hoffman Copyright © 1999, SQM, LLC. 1
Why Are We Doing This?
• Management needs numbers
• Management needs justification
• We want to know where to invest
Douglas Hoffman Copyright © 1999, SQM, LLC. 2
Intangibles
• Not all members of the test team will want to change
• Immediate reduction in perceived productivity of the
test organization
• Quality of tests
• Professionalism of test organization
• Expansion into advanced test issues
• Hands-off testing
• Number of product rolls (test cycles) before release
• Test coverage
Douglas Hoffman Copyright © 1999, SQM, LLC. 3
Falsely Expected Benefits
• All tests will be automated
• Immediate payback from automation
• Automation of existing manual tests
• Zero ramp up time
• Automated comprehensive test planning
• Capture/Playback for regression testing
• One tool that fits perfectly
• Automatic defect reporting (without human
intervention)
Douglas Hoffman Copyright © 1999, SQM, LLC. 4
Background to Costing
• Many changes would equally benefit manual testing
• Most automation benefits come from discipline in
analysis and planning
• Payback from automation is usually in the next
project or thereafter
• Automating usually causes significant negative
schedule and performance impacts at introduction
• Automated tests require more programming and
design skills from testers
• Automated tests frequently require maintenance
• Software metrics aren’t unbiased statistics
Douglas Hoffman Copyright © 1999, SQM, LLC. 5
ROI Factors
• Dependent upon factors besides costs and savings:
• Fixed costs need to be amortized over useful life
• Investments last only as long as they remain in service
• Variable benefits depend on the number of tests and test
runs
• Expected number of runs of tests is most important factor
in relative ROI computations
• Computed for a given time period (t)
• Incremental return on incremental investment vs.
manual testing
Douglas Hoffman Copyright © 1999, SQM, LLC. 6
ROI Computations
En = Aa/Am = (Va + n*Da) / (Vm + n*Dm) †
En = Aa/Am = (Va + n1*Da) / (Vm + n2*Dm)
ROIautomation(in time t) = (Savings from
automation) / (Costs of automation)
ROIautomation(in time t) = ∆(Savings from
automation) / ∆(Costs of automation)
† Linz, T, Daigl, M. “GUI Testing Made Painless. Implementation and results of the ESSI
Project Number 24306”, 1998. Analysis in Case Study: Value of Test Automation
Measurement, p. 52+ of Dustin, et. al., Automated Software Testing, Addison-Wesley, 1999.
Douglas Hoffman Copyright © 1999, SQM, LLC. 7
Relative ROI
• Changes in benefits and costs
• Tangible savings and costs
• Fixed and variable elements
Douglas Hoffman Copyright © 1999, SQM, LLC. 8
Common Cost Examples
• SUT analysis
• Test planning
• Base test design
• Defect reporting
• Management reporting (results)
Douglas Hoffman Copyright © 1999, SQM, LLC. 9
Fixed Cost Examples
• Hardware (additional or upgrades to existing)
• Testware software licenses
• Testware software support
• Per seat licenses
• Automation environment design, implementation
• Scripting tools
• Tool and environment licenses
• Tool training
• Tool introduction and ramp up
Douglas Hoffman Copyright © 1999, SQM, LLC. 10
Variable Cost Examples
• Test case design for automation
• Test case implementation
• Test maintenance
• Oracle creation
• Test running and analysis
Douglas Hoffman Copyright © 1999, SQM, LLC. 11
Usual Benefits
• Test execution savings
• After-hours testing by systems (not
people)
Douglas Hoffman Copyright © 1999, SQM, LLC. 12
Changed Cost Examples
• Automation environment maintenance
• Test case execution
• Test results analysis
• Defect reporting
• Test results reporting
• Test data generation
Douglas Hoffman Copyright © 1999, SQM, LLC. 13
ROI Computation
• Computing costs
• Computing savings
• ROI
Douglas Hoffman Copyright © 1999, SQM, LLC. 14
Cost Computation
• Σ (fixed costs of automated testing) times (t/Useful Life)
• Plus Σ (variable costs of creating automated tests)
• Less Σ (variable costs of creating manual tests)
• Plus Σ (variable costs of maintaining automated tests)
times (n/N)
• Plus increased Changed Costs values
Douglas Hoffman Copyright © 1999, SQM, LLC. 15
Alternate Cost Computation
• Σ (fixed costs of automated testing) times (t/Useful Life)
• Plus Σ (costs of people creating automated tests)
• Less Σ (costs of people creating manual tests)
• Plus Σ (costs of people maintaining automated tests)
• Plus increased Changed Costs values
Douglas Hoffman Copyright © 1999, SQM, LLC. 16
Benefit Computation
• Σ (variable costs of running manual tests n2
times during time t)
• Less Σ (variable costs of running automated
tests n1 times during time t)
• Plus Changed Costs savings values
Douglas Hoffman Copyright © 1999, SQM, LLC. 17
Conclusions
• Figures don’t lie, but …
• Understand why you need ROI
• Planning can increase ROI
• Identify costs and savings
Douglas Hoffman Copyright © 1999, SQM, LLC. 18
ROI Computation Example
• Daily builds and test runs (5 times a week)
• Manual tests take 5 days to design, 2 hours to run
• Only half the manual tests would be run on any given day (1 hour) with
the other half run the following day
• Automated tests take 15 days to design and implement, automatically
run (zero cost)
• Automation is done with batch scripts and integrated into the build
process, requiring $1,000 in added hardware, with a useful life of 3
years
• Automated tests need to be maintained every 25 runs, one day of work
required
• Periods of time (t) selected: 6 months (125 days) and 18 months (375
days)
• People cost $100,000 per year = $400 per day = $50 per hour
Douglas Hoffman Copyright © 1999, SQM, LLC. 19
ROI Computation Example
• A new product with all new tests
• 5 people-years developing manual tests, 15 for automated tests
• 1 person maintenance after 1st year for automated tests
• 10 people full time running manual tests, 1 person for
automated
• Fixed costs for automated tests of $90,000 with useful life of 3
years
• Periods of time (t) selected: 12 months (250 days) and 24
months (500 days)
• People cost $100,000 per year = $400 per day = $50 per hour
Douglas Hoffman Copyright © 1999, SQM, LLC. 20
