Is bigger better?
An introduction to sample size calculations
Presented by:
Dr Adrian Esterman
Flinders Centre for Epidemiology &
Scenario 1 All studies Scenario 2
Precision Power
Descriptive Hypothesis testing
Sample surveys Simple - 2 groups
Quality control
Complex studies
Flinders Centre for Epidemiology &
Scenario 1
Suppose we want to estimate the proportion
of people in our target population with a
given characteristic:
• The proportion with depression
• The proportion with an artficial leg
• The proportion receiving incorrect medication
Flinders Centre for Epidemiology &
Scenario 1
Example
• My target population is all South
Australians aged 17 and over
• I want to find out what proportion have
an undergraduate degree
• Please raise your hand if you have an
undergraduate degree
Flinders Centre for Epidemiology &
Scenario 1
Target Random
Sample
Population
Infer
Measure
Characteristic
Flinders Centre for Epidemiology &
Scenario 1
True proportion in target population = P
Estimated proportion from sample = p
How likely is it that p is exactly equal to P?
Flinders Centre for Epidemiology &
Scenario 1
We would like 95 times out of 100,
P to fall in this range
0 p 1
Sample
Flinders Centre for Epidemiology &
Scenario 1
The range of plausible values of our sample
proportion p in which the true population
proportion P is likely to fall 95 times out of
100 is called the 95% Confidence Interval
for P
Flinders Centre for Epidemiology &
Scenario 1
95% CI
for P
0 p 1
Sample
Flinders Centre for Epidemiology &
Scenario 1
The 95% CI for p is a measure of how
accurate your sample estimate is of the true
population proportion
95% Confidence
Interval
Sample size
Flinders Centre for Epidemiology &
Scenario 1
Example
We want to estimate the proportion of the
South Australian population with COPD.
We think it will be about 12%.
We would like a 95% CI of p ± 2%.
Flinders Centre for Epidemiology &
Scenario 1
Flinders Centre for Epidemiology &
Flinders Centre for Epidemiology &
Flinders Centre for Epidemiology &
Flinders Centre for Epidemiology &
Flinders Centre for Epidemiology &
Flinders Centre for Epidemiology &
Flinders Centre for Epidemiology &
p=50% with 95% CI 50% +/- 5%
400
Required sample size
300
200
100
0
Size of target population
Flinders Centre for Epidemiology &
Statcalc
Statcalc is included as part of the Epiinfo
suite of programs. This is available free of
charge from:
http://www.cdc.gov/epiinfo/
Flinders Centre for Epidemiology &
Scenario 2
We wish to formally test the difference
between two means or two proportions
Flinders Centre for Epidemiology &
Scenario 2
Three bits of information required to determine
the sample size
Type I & II Variation
errors Clinical
effect
Flinders Centre for Epidemiology &
Process of hypothesis testing Type I &
II errors
1. State a Null hypothesis (H0)
2. State an Alternative hypothesis (HA)
3. Decide on a suitable statistical test based on
the Null hypothesis
4. Calculate the test statistic
5. Check the associated probability (p-value)
6. If p 0.05 reject the Null hypothesis
Flinders Centre for Epidemiology &
Process of hypothesis testing Type I &
II errors
Note
If the Alternative hypothesis is:
parameter 1 parameter 2
we calculate the p-value for a two-sided test
If the Alternative hypothesis is:
parameter 1 > parameter 2
we calculate the p-value for a one-sided test
Flinders Centre for Epidemiology &
Type I &
II errors
What is a p-value?
1. It is a probability, and hence lies between 0 and 1.
2. It is a measure of surprise. In fact how surprised we
are to get a test statistics that large, if the Null
hypothesis were true.
Flinders Centre for Epidemiology &
Type I &
II errors
Type I and II errors
Statistical True state of null hypothesis
decision
Hypothesis true Hypothesis false
Reject Null Type I error Correct (Power)
hypothesis
Accept Null Correct Type II error
hypothesis
Flinders Centre for Epidemiology &
Type I &
What causes a Type I error II errors
• Bias
• Confounding
• Effect modification
• Misclassification
Flinders Centre for Epidemiology &
Type I &
What causes a Type II error II errors
• Sample size too small
• Confounding
• Effect modification
• Misclassification
Flinders Centre for Epidemiology &
Example of setting error levels Type I &
II errors
New drug for lowering cholesterol
• Slightly better efficacy than existing drugs
• Much more expensive than existing drugs
What are the consequences of making a Type I error?
What are the consequences of making a Type II error?
Flinders Centre for Epidemiology &
Example 1 Type I &
II errors
New drug for lowering cholesterol
Slightly better efficacy than existing drugs
• Much more expensive than existing drugs
Conclusion
• Requires stringent Type I error (say 0.01)
• Can managed with relaxed Type II error (say 0.20)
Flinders Centre for Epidemiology &
Example 2 Type I &
II errors
Trial of new brochure to help people quit smoking
• Successful in 20% of smokers
• Negligible cost
What are the consequences of making a Type I error?
What are the consequences of making a Type II error?
Flinders Centre for Epidemiology &
Example 2 Type I &
II errors
Trial of new brochure to help people quit smoking
• Successful in 20% of smokers
• Negligible cost
Conclusion
• Can relax Type I error (say 0.10)
• Requires stringent Type II error (say 0.05)
Flinders Centre for Epidemiology &
Scenario 2
Three bits of information required to determine
the sample size
Type I & II Variation
errors Clinical
effect
Flinders Centre for Epidemiology &
Clinical
Your Alternative hypothesis states effect
that you expect one group to have a
different mean or proportion to the
other group, but how much by?
• From the literature • 15% change
• From a pilot study • Change of 1 SD
• Clinically judgement • Interim analysis
Flinders Centre for Epidemiology &
Scenario 2
Three bits of information required to determine
the sample size
Type I & II Variation
errors Clinical
effect
Flinders Centre for Epidemiology &
Variation
Is there a difference between the two means?
Mean 1 Mean 2
Systolic Blood Pressure
Flinders Centre for Epidemiology &
Variation
It depends upon the range of the distributions
Systolic Blood Pressure
Flinders Centre for Epidemiology &
Variation
To judge whether the difference between
two means is large or small, we compare it
with some measure of the variability of the
distributions
Flinders Centre for Epidemiology &
Variation
Variability
All statistical tests are based on the following ratio:
Difference between parameters
Test Statistic =
v / n
As n v/n Test statistic
Flinders Centre for Epidemiology &
Variation
2
v x Test statistic
n =
Difference
Flinders Centre for Epidemiology &
Variation
The test-statistic is usually:
• Chi-squared for comparing two proportions
• Student’s t for comparing two means
• F-statistic for comparing two variances
• Z-statistic for comparing two correlation coefficients
but may be more complicated
Flinders Centre for Epidemiology &
Scenario 2
Example for two means
We wish to undertake an RCT of an intervention to
improve quality of life. At the end of the study, the
mean PCS of the SF-36 for the control group is
expected to be 35. We expect that in the
intervention group, the mean PCS will be 45. The
standard deviation of the PCS is 10.
Flinders Centre for Epidemiology &
Flinders Centre for Epidemiology &
1 – Type I
Error
1 – Type II
Error
Flinders Centre for Epidemiology &
Scenario 2
Example for two proportions
In a prospective study of hip protectors, we expect
that in the untreated group 10% of elderly people
will suffer a hip fracture. In the treated group we
expect this to reduce to 5%.
Flinders Centre for Epidemiology &
Flinders Centre for Epidemiology &
Winepiscope
Winepiscope is available free of charge
from:
http://www.clive.ed.ac.uk/winepiscope/
Flinders Centre for Epidemiology &
Allowing for dropouts
dropouts
Nearly all studies have at least some subjects who
withdraw, are lost to follow up, or who die
If n is the sample size computed by the program,
and we expect lose d% of subjects, then the
requires sample size is N is given by:
N = (100 x n) / (100 – d)
Flinders Centre for Epidemiology &
Allowing for dropouts
dropouts
Example
The sample size program tells us that we need 120
in each group and we are expecting a 15%
drop out.
N = (100 x 120) / (100 – 15)
= 141
Flinders Centre for Epidemiology &
Is bigger better?
For both descriptive and hypothesis testing
studies, the answer is yes.
1. Increasing the sample size will have no effect
on Type I errors which are largely due to bias
and/or confounding.
2. There is no point in having a larger sample size
than that required for precision or power.
Flinders Centre for Epidemiology &
Is bigger better?
For both descriptive and hypothesis testing
situations, the answer is yes. However:
1. Increasing the sample size will have no effect
on Type I errors which are largely due to bias
and/or confounding.
2. There is no point in having a larger sample size
than that required for precision or power.
Flinders Centre for Epidemiology &
For copies of this presentation
Please email Kylie Thomas at:
kylie.thomas@flinders.edu.au
Flinders Centre for Epidemiology &
