Exploring Best Practises in Design of Experiments

Copyright © 2014, SAS Institute Inc. All rights reserved.
Exploring Best Practises in
Design of Experiments
A Holistic Approach to DOE, Increasing Robustness,
Efficiency and Effectiveness

Contents
 Background to DOE
 Why Use DOE?
 Tips for Effective DOE with Classical Designs
 Definitive Screening
 Case Studies 1-3
 Role of Statistical Modelling and DOE in Learning
 Holistic DOE
 Case Study 4

BACKGROUND TO DESIGN OF
EXPERIMENTS (DOE)

FATHER OF DOE RONALD A. FISHER
Rothamstead Experimental Station, England – Early 1920’s

FISHER’S FOUR DESIGN PRINCIPLES
1. Factorial Concept - rather than one-factor-at-a-time
2. Randomization - to avoid bias from lurking variables
3. Blocking - to reduce noise from nuisance variables
4. Replication - to quantify noise within an experiment

AGRICULTURAL IMPACT
US corn yields
Cornell University, http://usda.mannlib.cornell.edu/MannUsda

WHY USE DOE?

Typical Process
The properties of products and processes are often affected
by many factors:
In order to build new or improve products and processes, we
must understand the relationship between the factors (inputs)
and the responses (outputs).
Typical
Process
Machine
Operator
Temperature
Pressure
Humidity
Yield
Cost
…
Inputs
Factors
Outputs
Responses

Traditional One-Factor-at-a-Time
 A common approach is one-factor-at-a-time experimentation.
 Consider experimenting one-factor-at-a-time to determine the
values of temperature and time that optimise yield.

 One-factor-at-a-time
experimentation frequently
leads to sub-optimal
solutions.
 Assumes the effect of one
factor is the same at each
level of the other factors, i.e.
factors do not interact.
 In practice, factors frequently
interact.

Interaction between factors

Experimental Design
 Most efficient way of investigating relationships.
 Runs (factor combinations) chosen to maximize the information
 Ideally balanced for ease of analysis and interpretation

ITERATIVE AND
SEQUENTIAL NATURE
OF CLASSICAL DOE

TIPS FOR EFFECTIVE DOE WITH
CLASSICAL DESIGNS

Stages of Experimental Design
 Designing an experiment involves much more
than just selecting the sequence of experimental
runs:
 Historically, improper planning is the most
common cause of failed experiments.
Plan Design Conduct Analyse Confirm

Some Planning Steps
 Review what we know
• Have peer discussions
 Determine new questions to answer
 Identify factors and ranges to investigate
 Define responses
• Easy and precise to measure

Common Experimental Objectives
Identify
Important
Factors
Screening
Design
Classical
Fractional
Factorial
Optimise
Process
RSM Design
Classical
Central
Composite
Optimise
Ingredients
Mixtures
Classical
Simplex &
Extreme
Vertices

Sequential Experimentation Reduces Total Cost
Identify
Important
Factors
Screening
Design
Classical
Fractional
Factorial
Optimise
Process
RSM Design
Classical
Central
Composite
Optimise
Ingredients
Mixtures
Classical
Simplex &
Extreme
Vertices

Sequential Experimentation
Identify
Important
Factors
Screening
Design
Classical
Fractional
Factorial
Optimise
Process
RSM Design
Classical
Central
Composite
Optimise
Ingredients
Mixtures
Classical
Simplex &
Extreme
Vertices
Definitive Screening Design Simplifies Experimental Workflow

Sequential Experimentation
Identify
Important
Factors
Screening
Design
Classical
Fractional
Factorial
Optimise
Process
RSM Design
Classical
Central
Composite
Optimise
Ingredients
Mixtures
Classical
Simplex &
Extreme
Vertices
Definitive Screening Design
Optimal Design Manages Experimental Constraints

Determining the Appropriate Factors
 Determining the factors to be included in your experiment is a
critical part of planning.
• Exploring too many factors may be costly and time
consuming.
• Exploring too few may limit the success of your experiment.
 Prior knowledge and analysis of existing data are useful aids to
identifying and prioritising factors for study. Other methods may
include:
• Brainstorming
• Ishikawa

Selection of Factor Range is Critical With
Two Level Designs …

By experimenting at the two settings in
yellow, X would be declared unimportant

By using half and often times much less than than
half the factor range X is declared important

By using half and often times much less than than
half the factor range X is declared important
Often leads to narrow factor ranges
to force linear relationships but
consequence is high risk of
determining sub-optimal solution

Determining the Appropriate Responses
 Selection of your responses will also be critical to the success of
your experiment. Whenever possible:
• Choose variables that correlate to internal or external
customer requirements
• Find responses that are easy to measure
• Make sure your measurement systems are precise, accurate,
and stable
 Analysis of current data, prior knowledge, measurement systems
analysis are useful aids.

DEFINITIVE SCREENING

Fractional Factorials: Complex workflow
from many factors to optimum settings
Tempting to miss out
middle step which can
result in selection of
wrong factors and decisions

Definitive Screening Design
 Identifies active main effects, uncorrelated with other
effects.
 May identify significant quadratic effects, uncorrelated with
main effects and at worst weakly correlated with other
quadratic effects.
 If few factors turn out to be important, can identify
significant two-way interactions uncorrelated with main
effects and weakly correlated with other higher order
effects.
 One stage experiment if three or fewer factors important:
• progress straight to full quadratic model
• optimise process with no further experimentation
• otherwise augment DSD for optimization goals

New Class of Screening Design
 Three-level screening
design
• 2m + 1 runs based on
m fold-over pairs and
an overall center point,
where m is number of
factors
• the values of the ±1
entries in the odd-
numbered runs are
determined using
optimal design.

Use of Three Level Designs
Advantageous
 Scientists and engineers are uncomfortable using two-level designs
• Restricting factor ranges may result in sub-optimal solutions
• Scientific/engineering judgment suggests relationships nonlinear over
wide ranges
 Investigators frequently have an opinion regarding the “best” levels
of each factor for optimizing a response
• Experimental region centered at these levels.
• Two-level design might screen out an important factor when
experimental region centred at “best”
• Adding centre points allows test for curvature
• However ambiguity over factors causing curvature
• DSD avoids ambiguity by making it possible to uniquely identify the
source(s) of curvature.

CASE STUDIES

Case Study 1: Optimising a Chemical
Process
Why Consider Definitive Screening Designs?

Background
 Five factors
 One response yield
 Goal optimise yield
 Keep total cost of experimentation to minimum
 Contrast traditional approach of main effect screening
design plus augmentation to RSM with DSD

 Traditional screening approach correlates main
effects with two factor interaction effects
 Cost constraint and inexperience with such
designs can lead to missed DOE steps
 Investigator missed step of augmenting main
effect design to separate correlated interaction
effects from assumed important main effects
 Resulted in wrong set of factors selected for
RSM design which results in wrong solution
Background

Traditional Approach with Missed Step

Resolution III Design Perfectly Correlates
Main Effects With Interaction Effects

Model Interpretation
 Fitted Model
Y = b0 + b1*X1 + b2*X2 + b3*X3 + Error
 Correct Interpretation of Fitted Model
Y = b0 + b1*(X1+X2X3) + b2*(X2+X1X3) + b3*(X3+X1X2) + Error

Missed Step Augments Initial Design to
Separate Main Effects From Interactions

Model Interpretation of Augmented Design
 Correct Interpretation of Model Fitted to Augmented design
Y = b0 + b1*X1 + b2*X2 + b3*X3 + b12*X1X2 + b13*X1X3 + b23*X2X3 + Error
 Allows clear separation of main and interaction effects
 This step was missed in case study prior to modelling curvature

 DSD results in correct identification of important
factors due to non correlated main and two factor
interaction effects
 Because just three factors are important DSD
results in one step design:
• In addition to correctly identifying correct factors
• DSD requires no augmentation to identify optimal
settings of important factors
Background

CASE STUDY 1

Conclusions
 Fractional factorial designs can lead to selection of
wrong factor set
 Complex workflow for avoiding this risk which may be
misunderstood or not applied by users new to DOE
 May lead to conclusion that DOE does not work for us!
 DSD simplifies DOE process and removes risk of
selecting wrong factor set
 Provides one step DOE when three or fewer important
factors
• Sufficient to identify correct factor set and determine best
settings of selected factors

Case Study 2: Optimising Reaction
Conditions for Chemical Methods
Augmenting Definitive Screening Designs

Background
 How effective are DSD when more than three
factors are important?
 Use example from literature
• Response-Surface Co-optimization of Reaction
Conditions in Clinical Chemical Methods, Gopal S.
Rautela, Ronald 0. Snee,’ and Warren K. Miller,
CLINICAL CHEMISTRY, Vol. 25, No. 11, 1979
• CCF RSM in six factors
• Five factors are important
• Use model from this experiment to contrast CCF with
augmented DSD

 Aspartate Aminotransferase Assay:
http://www.chem.qmul.ac.uk/iubmb/enzyme/EC2/6/1/1.html
 Six factors (reagent conditions):
tris(hydroxymethyl)aminomethane, pH, L-aspartic acid,
pyridoxal-5’-phosphate, 2-oxoglutarate, and malate
dehydrogenase
 Response: aspartate aminotransferase activity measured
for human serum with above normal activity at 30C
 Goal: select reagent conditions that maximise aspartate
aminotransferase activity
 Example selected to stress DSD when >3 factors are
important, in this case 5 factors are important.
Background

CASE STUDY 2

Conclusions
 When >3 factors are important, augmenting DSD
works
 When >3 factors are important, an augmented
DSD approach is more efficient than classical
Response Surface Designs

CASE STUDY 3

Case Study 3: Optimising Yield
What About Constrained Factor Spaces?

Background
 From chapter 5 of Goos &
Jones
 Chemical reaction
 Goal: maximise yield
 2 factors: Temperature and
Time

Background
 Expert knowledge tells
us
• Certain conditions will
give poor results (hence,
constraints)
• Behaviour very non-linear
 We will show
• Design where prior
knowledge is ignored.
• Fitting the design to the
problem

Example of Process Constraint

Shrink Experimental Range to Factorial

Optimal Design: Use Actual Factor Range

… optimal designs allow investigation of complete factor
space properly adjusted for constraints
Typical
Process
Machine
Operator
Temperature
Pressure
Humidity
Yield
Cost
…
Inputs
Factors
Outputs
Responses
Optimal Design: Fit to Model
Model
Y = f(X)
The process is not seen as a black box anymore…

Conclusions
 Custom Design permits studying any:
• combination of factors with or without constraints,
• number of factor levels,
• blocking structure.
 Build your design to suit the problem instead of
fitting the problem into a design

Case Study 4: Designing Products People
Want to Buy
Holistic DOE

ROLE OF STATISTICAL MODELLING
AND DOE IN LEARNING

Data Sources
 DOE and/or observational (historical)
 Potential problems with observational data:
• X’s are correlated – identification of “best” model
difficult
• Outliers (potential or real) - bias model estimation
• Missing data cells – result in loss of whole data rows
with traditional least squares based analysis
• Range over which X’s varied may be limited –
restricting model usefulness
• May not have measured all relevant X’s
 In some situations these can also be issues with
DOE datasets

WHAT IS HOLISTIC DOE?

Holistic DOE Approach: Integrating
Statistical Modelling and DOE
 Learning is incremental and effective statistical modelling of
observational data aids design of next experiment.
 Analysis approach needs to manage real (messy) data simply
• Correlated X’s, outliers, missing cells
• Quickly deliver “best” current model to revise with new DOE data
• Aid better analysis of new experimental data when unexpected
occurs
• Build models based on individual datasets and aggregated data
 Good statistical modelling integrated with DOE helps reduce
total learning time, effort and cost
 It would be a shame to not use pre-existing data that comes
for free

Holistic DOE Example

Background
 PC retailer is observing appreciably retail price
variation in its laptop computer line.
 Goals:
• Investigate factors associated with retail price variation.
• Perform further experimentation in key factors to
optimise and standardise pricing across stores.

CASE STUDY 4

Conclusions
 Analysis of prior data helps identify factors and
ranges to use in next DOE.
 Analysis of prior data helps reduce risk and
increase efficiency and effectiveness of future
experiments.
 DOE is not just for science and engineering.

Holistic DOE: Integrated Statistical
Modelling and DOE
 Supports wide range of user skills
 Exploratory analysis and statistical modelling of historical
messy data simplifies and shortens whole DOE process.
 Next generation DOE enables more staff to apply DOE with
reduced learning and implementation effort
 Interact with model predictions to build consensus
 Integrated simulation capabilities enables rapid progression
from models to decisions
 Drag and drop charts help monitor processes and identify
potential causes of issues
 Manage risk better by correctly identifying signal from noise

Exploring Best Practises in Design of Experiments

More Related Content

Viewers also liked

Similar to Exploring Best Practises in Design of Experiments

More from JMP software from SAS

Recently uploaded

Exploring Best Practises in Design of Experiments