Introduction of A/B testing?

A/B testing , also known as split testing, refers to a randomized experimentation

process wherein two or more versions of a variable (web page, page element, etc.) are
shown to different segments of website visitors at the same time to determine which
version leaves the maximum impact and drives business metrics.

Essentially, A/B testing eliminates all the guesswork out of website optimization and
enables experience optimizers to make data-backed decisions. In A/B testing, A refers
to ‘control’ or the original testing variable. Whereas B refers to ‘variation’ or a new
version of the original testing variable.

The version that moves your business metric(s) in the positive direction is known as
the ‘winner.’ Implementing the changes of this winning variation on your tested
page(s) / element(s) can help optimize your website and increase business ROI.

The metrics for conversion are unique to each website. For instance, in the case of
eCommerce, it may be the sale of the products. Meanwhile, for B2B, it may be the
generation of qualified leads.

A/B testing is one of the components of the overarching process of Conversion Rate
Optimization (CRO) , using which you can gather both qualitative and quantitative
user insights. You can further use this collected data to understand user behavior,
engagement rate, pain points, and even satisfaction with website features, including
new features, revamped page sections, etc. If you’re not A/B testing your website ,
you’re surely losing out on a lot of potential business revenue.

Why should you consider A/B testing?

If B2B businesses today are unhappy with all the unqualified leads they get per
month, eCommerce stores, on the other hand, are struggling with a high cart
abandonment rate . Meanwhile, media and publishing houses are also dealing with low
viewer engagement. These core conversion metrics are affected by some common
problems like leaks in the conversion funnel, drop-offs on the payment page, etc.

Let’s see why you should do A/B testing:

1. Solve visitor pain points

Visitors come to your website to achieve a specific goal that they have in mind. It
may be to understand more about your product or service, buy a particular product,
read/learn more about a specific topic, or simply browse. Whatever the visitor’s goal
may be, they may face some common pain points while achieving their goal. It can be
a confusing copy or hard to find the CTA button like buy now, request a demo, etc.

Not being able to achieve their goals leads to a bad user experience. This increases
friction and eventually impacts your conversion rates. Use data gathered
through visitor behavior analysis tools such as heatmaps, Google Analytics, and
website surveys to solve your visitors’ pain points. This stands true for all businesses:
eCommerce, travel, SaaS, education, media, and publishing.

2. Get better ROI from existing traffic

As most experience optimizers have come to realize, the cost of acquiring quality
traffic on your website is huge. A/B testing lets you make the most out of your
existing traffic and helps you increase conversions without having to spend additional
dollars on acquiring new traffic. A/B testing can give you high ROI as sometimes,
even the minutest of changes on your website can result in a significant increase in
overall business conversions .

3. Reduce bounce rate

One of the most important metrics to track to judge your website’s performance is its
bounce rate. There may be many reasons behind your website’s high bounce rate ,
such as too many options to choose from, expectations mismatch, confusing
navigation, use of too much technical jargon, and so on.

Since different websites serve different goals and cater to different segments of
audiences, there is no one-size-fits-all solution to reducing bounce rate. However,
running an A/B test can prove beneficial. With A/B testing, you can test multiple
variations of an element of your website till you find the best possible version. This
not only helps you find friction and visitor pain points but helps improve your
website visitors’ overall experience, making them spend more time on your site and
even converting into a paying customer.

4. Make low-risk modifications

Make minor, incremental changes to your web page with A/B testing instead of
getting the entire page redesigned. This can reduce the risk of jeopardizing your
current conversion rate.

A/B testing lets you target your resources for maximum output with minimal
modifications, resulting in an increased ROI. An example of that could be product
description changes. You can perform an A/B test when you plan to remove or update
your product descriptions. You do not know how your visitors are going to react to
the change. By running an A/B test, you can analyze their reaction and ascertain
which side the weighing scale may tilt.

Another example of low-risk modification can be the introduction of a new feature

change. Before introducing a new feature, launching it as an A/B test can help you
understand whether or not the new change that you’re suggesting will please your
website audience.

Implementing a change on your website without testing it may or may not pay off in
both the short and long run. Testing and then making changes can make the outcome
more certain.

5. Achieve statistically significant improvements

Since A/B testing is entirely data-driven with no room for guesswork, gut feelings, or
instincts, you can quickly determine a “winner” and a “loser” based on statistically
significant improvements on metrics like time spent on the page, number of demo
requests, cart abandonment rate, click-through rate, and so on.

6. Redesign website to increase future business gains

Redesigning can range from a minor CTA text or color tweak to particular web pages
to completely revamping the website . The decision to implement one version or the
other should always be data-driven when A/B testing. Do not quit testing with the
design being finalized. As the new version goes live, test other web page elements to
ensure that the most engaging version is served to the visitors.

Which statistical approach to use to run an A/B test?

Post learning about four different types of A/B testing experimentation methods, it’s equally
important to understand which statistical approach to adopt to successfully run an A/B test
and draw the right business conclusion.

Ideally, there are two types of statistical approaches used by A/B/n experimenters across the
globe: Frequentist and Bayesian. Each of these approaches has its own pros and cons.
However, we, at VWO, use, support, and promote the Bayesian approach.
The comparison between the two approaches given below will help you understand why.

Frequentist approach:
The frequentist approach of probability defines the probability of an event with relation to
how frequently (hence the name) a particular event occurs in a large number of trials/data
points. When applied to the world of A/B testing, one can see that anyone going with the
frequentist approach would need more data (a function of more number of visitors tested and
over longer durations) to come to the right conclusions. This is something that limits you in
scaling up any A/B testing effort. According to the Frequentist approach, it is essential to
define your A/B test’s duration based on sample size to reach the right test conclusions. The
tests are based on the fact that every experiment can be repeated infinite times.
Following this approach calls for a lot of attention to detail for every test that you run because
for the same set of visitors, you’ll be forced to run longer duration tests than the Bayesian
approach. Hence, each test needs to be treated with extreme care because there are only a few
tests that you can run in a given timeframe. Unlike Bayesian statistics, the Frequentist
approach is less intuitive and often proves difficult to understand.
Bayesian approach:
As compared to the Frequentist approach, Bayesian statistics is a theory-based approach that
deals with the Bayesian interpretation of probability, where probability is expressed as a
degree of belief in an event. In other words, the more you know about an event, the better and
faster you can predict the end outcomes. Rather than being a fixed value, probability under
Bayesian statistics can change as new information is gathered. This belief may be based on
past information such as the results of previous tests or other information about the event.
Unlike the frequentist approach, the Bayesian approach provides actionable results almost
50% faster while focusing on statistical significance. At any given point, provided you have
enough data at hand, the Bayesian approach tells you the probability of variation A having a
lower conversion rate than variation B or the control. It does not have a defined time limit
attached to it, nor does it require you to have an in-depth knowledge of statistics.
In the simplest of terms, the Bayesian approach is akin to how we approach things in
everyday life. For instance, you misplaced your mobile phone in your house. As a frequentist,
you would only use a GPS tracker to track it and only check the area the tracker is pointing
to. While as a Bayesian, you will not only use a GPS tracker but also check all the places in
the house you earlier found your misplaced phone. In the former, the event is considered a
fixed value, while in the latter, all past and future knowledge are utilized to locate the phone.
How to perform an A/B test?
A/B testing offers a very systematic way of finding out what works and what doesn’t work in
any given marketing campaign. Most marketing efforts are geared toward driving more
traffic. As traffic acquisition becomes more difficult and expensive, it becomes paramount to
offer your users the best experience who comes to your website. This will help them achieve
their goals and allow them to convert in the fastest and most efficient manner possible. A/B
testing in marketing allows you to make the most out of your existing traffic and increase
revenue inflow.

A structured A/B testing program can make marketing efforts more profitable by pinpointing
the most crucial problem areas that need optimization. A/B testing is now moving away from
being a standalone activity that is conducted once in a blue moon to a more structured and
continuous activity, which should always be done through a well-defined CRO process.
Broadly, it includes the following steps:
Step 1: Research
Before building an A/B testing plan, one needs to conduct thorough research on how the
website is currently performing. You will have to collect data on everything related to how
many users are coming onto the site, which pages drive the most traffic, the various
conversion goals of different pages, etc. The A/B testing tools used here can include
quantitative website analytics tools such as Google Analytics, Omniture, Mixpanel, etc.,
which can help you figure out your most visited pages, pages with most time spent, or pages
with the highest bounce rate. For example, you may want to start by shortlisting pages that
have the highest revenue potential or the highest daily traffic. Following this, you may want
to dive deeper into the qualitative aspects of this traffic.
Heatmap tools are the leading technology used to determine where users are spending the
most time on, their scrolling behavior, etc. This can help you identify problem areas on your
website. Another popular tool used to do more insightful research is website user surveys.
Surveys can act as a direct conduit between your website team and the end-user and often
highlight issues that may be missed in aggregate data.
Further, qualitative insights can be derived from session recording tools that collect data on
visitor behavior, which helps in identifying gaps in the user journey. In fact, session recording
tools combined with form analysis surveys can uncover insights on why users may not be
filling your form. It may be due to some fields that ask for personal information or users,
maybe abandoning your forms for too long.
As we can see, both quantitative and qualitative research can help us prepare for the next step
in the process, making actionable observations for the next steps.

Step 2: Observe and formulate hypothesis

Get closer to your business goals by logging research observations and creating data-backed
hypotheses aimed at increasing conversions. Without these, your test campaign is like a
directionless compass. The qualitative and quantitative research tools can only help you with
gathering visitor behavior data. It is now your responsibility to analyze and make sense of
that data. The best way to utilize every bit of data collated is to analyze it, make keen
observations on them, and then draw websites and user insights to formulate data-backed
hypotheses. Once you have a hypothesis ready, test it against various parameters such as how
much confidence you have of it winning, its impact on macro goals, and how easy it is to set
up, and so on.
Step 3: Create variations
The next step in your testing program should be to create a variation based on your
hypothesis, and A/B test it against the existing version (control). A variation is another
version of your current version with changes that you want to test. You can test multiple
variations against the control to see which one works best. Create a variation based on your
hypothesis of what might work from a UX perspective. For example, enough people not
filling forms? Does your form have too many fields? Does it ask for personal information?
Maybe you can try a variation with a shorter form or another variation by omitting fields that
ask for personal information.
Step 4: Run test
Before we get to this step, it’s important to zero upon the type of testing method and approach
you want to use. Once you’ve locked down on either one of these types and approaches based
(refer to the above-written chapters) on your website’s needs and business goals, kick off the
test and wait for the stipulated time for achieving statistically significant results. Keep one
thing in mind – no matter which method you choose, your testing method and statistical
accuracy will determine the end results.
For example, one such condition is the timing of the test campaign. The timing and duration
of the test have to be on point. Calculate the test duration keeping in mind your average daily
and monthly visitors, estimated existing conversion rate, minimum improvement in
conversion rate you expect, number of variations (including control), percentage of visitors
included in the test, and so on.
Use our Bayesian Calculator to calculate the duration for which you should run your A/B
tests for achieving statistically significant results.
Step 5: Analyse results and deploy changes
Even though this is the last step in finding your campaign winner, analysis of the results is
extremely important. Because A/B testing calls for continuous data gathering and analysis, it
is in this step that your entire journey unravels. Once your test concludes, analyze the test
results by considering metrics like percentage increase, confidence level, direct and indirect
impact on other metrics, etc. After you have considered these numbers, if the test succeeds,
deploy the winning variation. If the test remains inconclusive, draw insights from it, and
implement these in your subsequent tests.
 Conclusion
After reading this comprehensive piece on A/B testing, you should now be fully equipped to
plan your own optimization roadmap. Follow each step involved diligently and be wary of all
major and minor mistakes that you can commit if you do not give data the importance it
deserves. A/B testing is invaluable when it comes to improving your website’s conversion
rates.

If done with complete dedication, and with the knowledge you now have, A/B testing can
reduce a lot of risks involved when undertaking an optimization program. It will also help
you significantly improve your website’s UX by eliminating all weak links and finding the
most optimized version of your website.
 RESEARCH PROJECT SYNOPSIS
on
“A\B TESTING: Conduct A/B tests to evaluate the
effectiveness of different marketing campaigns”

JAGRAN LAKECITY UNIVERSITY

BHOPAL

Master of Business Administration

2022-23

Submitted to: ANKUSH SHARMA Submitted by: ALANKRIT SINGH