Advertising Effectiveness

By Jerry W. Thomas, Decision Analyst

T he advertising industry, as a whole, has the poorest qual-

ity-assurance systems and turns out the most inconsistent
product (their ads and commercials) of any industry in the
in the way. The first great barrier to better advertising is
self-delusion. Most of us believe, in our heart-of-hearts, that
we know what good advertising is and that there is no need
world. This might seem like an overly harsh assessment, but for any kind of independent, objective evaluation. Agencies
it is based on testing thousands of ads over several decades. and clients alike often think that they know how to create and
In our experience, only about half of all commercials actually judge good advertising. Besides, once agencies and clients
work; that is, have any positive effects on consumers’ pur- start to fall in love with the new creative, they quickly lose
chasing behavior or brand choice. Moreover, a small share interest in any objective evaluation. No need for advertising
of ads actually appear to have negative effects on sales. How testing. Case closed.
could these assertions possibly be true? Don’t advertising
agencies want to produce great ads? Don’t clients want great Strangely, after 40 years of testing advertising, we cannot tell
advertising? Yes, yes, they do, but they face formidable bar- you if a commercial is any good or not, just by viewing it.
riers. Sure, we have opinions, but they are almost always wrong.
In our experience, advertising agencies and their clients
Unlike most of the business world, are just as inept at judging advertis-
which is governed by numerous feed- ing as we are. It seems that none of
back loops, the advertising industry us is smart enough to see advertising
receives little objective, reliable feed- through the eyes of the target audience,
back on its advertising. First, few ads based purely on our own judgment.
and commercials are ever tested among
consumers (less than one percent, ac- A second barrier to better advertising
cording to some estimates). So, no is the belief that sales performance will
one—not agency or client—knows if tell if the advertising is working. Un-
the advertising is any good. If no one less the sales response to the advertis-
knows when a commercial is good or ing is immediate and overwhelming, it
bad, or why, how can the next com- is almost impossible to use sales data to
mercial be any better? Second, once the advertising goes on judge the effectiveness of the advertising. So many variables
air, sales response (a potential feedback loop) is a notoriously are beyond our control, as noted, that it’s impossible to iso-
poor indicator of advertising effectiveness because there is late the effects of media advertising alone. Moreover, some
always so much “noise” in sales data (competitive activity, advertising works in a few weeks, while other advertising
out-of-stocks, weather, economic trends, promotional influ- might take many months to show positive effects, and this
ences, pricing variation, etc.). Third, some of the feedback delayed response can confound our efforts to read the sales
is confusing and misleading: agency and client preferences data. Also, advertising often has short-term effects that sales
and biases, the opinions of the client’s wife, feedback from data might reflect, and long-term (years later) effects that
dealers and franchisees, complaints from the lunatic fringe, most of us might easily overlook in subsequent sales data.
and so on. Because of these limitations, sales data tend to be confusing
and unreliable as an indicator of advertising effectiveness.
Barriers to Great Advertising

Advertising testing could provide a reliable feedback loop

and lead to much better advertising, but many obstacles stand
S ophisticated marketing mix modeling is one way to
measure these advertising effects on sales, but it often
takes millions of dollars and years of effort, and requires

604 Avenue H East

Arlington, TX 76011-3100, USA http://www.decisionanalyst.com
The global leader in analytical research systems 1.817.640.6166 or 1.800. ANALYSIS © 2007 Decision Analyst, Inc.
the building of pristine databases of sales information along feedback from target consumers. Big creative egos tend to
with all of the marketing input variables. Few companies resist such evolutionary improvements. We have seen great
have the budget, the patience, the accurate databases, and the campaigns abandoned because agencies would not accept
technical knowledge necessary to succeed at marketing mix minor tweaks to the advertising. To be fair, big egos are not
modeling. Even so, marketing mix modeling does not help limited to advertising agencies. Big client egos can also be
us evaluate the contribution of a single commercial but rather a barrier to good advertising. Research firm egos are yet
the cumulative effects of many different commercials over a another problem. Big egos create barriers because emotion is
long period of time. Also, marketing mix modeling does not driving advertising decision making, instead of logic, reason,
tell us why the advertising worked, or failed to work. Was and consumer feedback. Big egos lead to bad advertising.
it message, or media weight, or media mix that made the
advertising effective? Generally, marketing mix modeling
cannot answer these types of questions. So, again, sales data
is of limited value when you make critical decisions about
A fifth barrier to better advertising is the widespread belief
that one’s major competitors know what they are doing.
So, just copy the advertising approaches of the competition,
your advertising. and success will surely follow. We recently had a client
who was about to copy the advertising strategy of a major
A third barrier to better advertising is a pervasive tendency of competitor, but we were able to persuade the client to test
many (but not all) advertising agencies to delay, undermine, all major competitive commercials as a precaution before
and thwart efforts to objectively test their creative “babies.” blindly copying the competitor’s advertising approach. This
Who wants a report card on the quality of their work? It’s competitor was the industry leader in market share and profit-
very threatening. The results can upset the creative folks. ability. Our testing quickly revealed that this industry leader
The results can upset clients. The agency can lose control. was the industry leader in spite of its bad advertising. The
Agencies can be quite creative in coming up with reasons to testing also revealed that another competitor, in contrast, had
avoid copy testing. Some of our favorites: great advertising. Needless to say, the client’s desire to copy
the industry leader quickly vanished.
• There’s no time. We have to be on air in five days, so
we’ll just have to skip the testing. The sixth barrier to better advertising is lack of strategy,
• These ads are built on emotion and feelings, and you or having a poor strategy. The client is most often at fault
can’t measure such delicate, artful subtleties. here. The client has not done his homework, has not thought
• We’ve already tested the ads with a focus group during deeply about his brand and its future, and has not developed
the development process. and tested strategy alternatives. The client tells the agency to
• These are image ads, and you can’t test imagery with go forth and create great advertising, without providing any
standard advertising testing techniques. strategy guidelines. The agency is left to guess and specu-
• We have so much equity in this campaign that it late about strategy. Great advertising is rarely created in a
doesn’t matter what the testing results are. We can’t strategy vacuum. If the client cannot define a sound strategy,
afford to change. the agency cannot create great advertising. Again, the re-
• We’re in favor of testing, but let’s remove those ques- sponsibility for strategy falls squarely on the client. A seventh
tions about purchase intent and persuasion from the barrier to better advertising is client ineptness. Some clients’
questionnaire. processes, policies, and people tend to discourage the cre-
• We are in a new age, with new media and new mes- ation of great advertising. Arrogance, ambiguity, impatience,
sages, and none of the old copy-testing measures ap- ignorance, risk aversion, and inconsistency tend to be the
ply any more. hallmarks of these “agency killer” clients. Bad clients rarely
stimulate or tolerate great advertising.

T he fourth barrier to more effective advertising is the

big creative ego. The belief that only the “creatives” in
the agency can create advertising—and the conviction that
The eighth and last barrier to better advertising is poor copy
testing by research companies. Many advertising-testing
creativity is their exclusive domain—constitute a major bar- systems are limited to a few markets (and cannot provide
rier. Great advertising tends to evolve over time, with lots representative samples). Some systems are so expensive that
of hard work, fine-tuning, and tinkering—based on objective the cost of testing exceeds the value of the results. Research


companies have been guilty of relying on one or two simplis- tem over and over, so that everyone (client, agency,
tic measures of advertising effectiveness, while completly and researchers) learns how to interpret the pretest-
ignoring many other very important variables. For instance, ing results for the category and the specific brand.
for several years research companies argued publicly over
which was more important, persuasion measures or recall 4. If budget permits, test the advertising at an early
measures? The truth is that both are important, but of greater stage in the creative process (i.e., the “storyboard”
import is the fact that neither of these measures alone, or in or “animatic” stage) and also test at the finished com-
combination, measures advertising effectiveness. To judge mercial stage. Early-stage testing allows rough com-
the effectiveness of an ad, many different variables must be mercials to be tweaked and fine-tuned before you
measured and considered simultaneously. spend the big dollars on final production. Early-
stage testing tends to be highly predictive of finished
Creating Better Advertising commercial scores, but not always. Testing the fin-
ished commercials gives you extra assurance that
Given all of these barriers to better advertising, how can cli- your advertising is “on strategy” and working.
ent, agency, and research company work together to create
more effective advertising? 5. Build your own “action standards” over time. As
you test every execution, you will begin to learn
1. The client must craft a sound strategy for its brand, what works and what doesn’t work. Think of the
based on facts, not wishful thinking and self-delu- pretesting company’s norms as very crude, rough
sion. The client must carefully define the role of indicators to help you get started with a testing
advertising in the marketing plan and set precise program. But, as quickly as possible, develop your
communication objectives for the advertising. What own norms for your category and your brand (yes,
exactly does the client want the advertising to con- all of the advertising effectiveness measures vary by
vey, to accomplish? Agencies are too often asked to product category and brand). What you are search-
create advertising in an informational vacuum. ing for, long-term, are not norms, but action stan-
Agencies are not miracle workers. Once strategy dards (that is, the knowledge that certain advertising
and positioning alternatives are identified and testing scores will translate into actual sales increases).
tested, the strategy should be locked down…and
rarely changed thereafter. 6. Use a mathematical model to derive an overall score
for each execution. It doesn’t matter that an ad has
2. As creative executions are developed against the great persuasion if it does not register the brand
strategy, each execution should be pretested among name. It doesn’t matter that an ad registers the brand
members of the target audience (pretesting refers to name if no one will notice the commercial itself. It
testing advertising before it is aired, and/or before doesn’t matter that an ad increases short-term pur-
final production. When the term “testing” is used in chase interest if it will damage the brand’s quality
this article, it is a shorthand term for “pretesting.”) reputation over time. So, all of the key variables
The greater the number of executions pretested, must be put together intelligently to come up with a
the more likely it is that great advertising will composite or overall measure of advertising effec-
emerge. Testing the creative provides a reliable tiveness.
feedback loop that helps agency and client alike
become smarter over time. Once a conceptual family 7. Use the pretesting results as a guide, as an indicator,
of commercials is identified as the optimal campaign but do not become a slave to the mathematical
of the future, then the campaign should be locked model. Read all of the open-ended questions care-
down. Long-term continuity of advertising message fully. Make sure you really understand the underly-
is essential to maximizing effectiveness. ing reasons. Base your decisions on this comprehen-
sive assessment of the results, and leave yourself
3. Use the same pretesting system consistently. There some wiggle room. No model or system can antici-
is no perfect advertising pretesting system. Some pate every marketing situation, or give a 100%
are better than others, but any system will help im- perfect solution every time. Informed human judgment
prove your advertising. The secret is to use one sys- remains important.


8. Client and agency need to accept that “continuous The Power of Advertising
improvement” of the advertising is an important
goal. This means that every execution is tested and We believe in the power of advertising, based on thousands
tweaked based on scientific evidence from the target of studies in our archives. Advertising has the power to
audience. We are not talking about changing the persuade, the power to influence the mind and shape des-
strategy or changing the campaign, but making sure tiny. It has the power to change markets and improve profit
that every execution is “on strategy” and working as margins. Advertising has short-term power (conveying new
hard as possible. information, building awareness, enhancing credibility, etc.)
and long-term power (conveying brand
9. The ultimate goal of testing is an image, attaching emotional values to the
advertising success formula that brand, building positive reputation, etc.).
works. That is, the goal of advertising The great power of advertising is seldom
creative development, and the goal of achieved in practice, but we can’t give
advertising testing, is to identify the up. The potential and the promise are
elements/ideas essential to advertising too great. The companies that master the
effectiveness, and then to make creative guidance and the testing systems
sure that those elements/ideas are to consistently develop and deploy great
consistently communicated by all advertising will own the future and the
advertising executions. fortunes that go with it. Great advertising
is a cloak of invincibility.

About the Author

Jerry W. Thomas (jthomas@decisionanalyst.com) is the President/CEO of Decision Analyst.
Decision Analyst is a leading international marketing research and marketing consulting firm.
The company specializes in advertising testing, strategy research, new product development,
and advanced modeling for marketing decision optimization. The author may be reached at
800.262.5974 or 1.817.640.6166.

604 Avenue H East

Arlington, TX 76011-3100, USA http://www.decisionanalyst.com
The global leader in analytical research systems 1.817.640.6166 or 1.800. ANALYSIS © 2007 Decision Analyst, Inc.