ab test hypothesis format

AB testing template: how to plan and document experiments

Quarter beginnings usually resonate with growth marketers.

Maybe because that’s when they are preparing their tactical plan, establishing their goals, or just because it's the time for them to think more deeply about their strategies. Regardless, it hasn’t been different to our customers; and that’s exactly the time of the year they usually reach us asking for help to plan their AB tests.

These are the most common questions we get:

How to organize your testing hypotheses

How to prioritize test ideas, which metric to use for analyzing tests performance, how to estimate the test duration beforehand, which method to use to analyze the results, how to document the process for future reviews.

Formulating a roadmap is not always enough for efficient split tests. That’s why we came up with this complete guide to help you test your hypotheses and plan your next marketing strategies.

The step-by-step for a good AB testing plan

We've already covered some of those topics in other blog posts. Let's revisit them now.

Listing testing hypothesis requires both data knowledge and creativity, and there's nothing better than brainstorming . Invite all your team members to a 1-hour meeting and let them brood about what should make the conversion rates improve. The more diverse this team is, the better: ask help from designers, copywriters, engineers, data scientists, marketing, and product people.

You should list from 10 to 20 hypotheses to plan your quarter tests.

Example: Donna, a holistic therapist, is trying to get more clients in Singapore, where she lives. These are some ideas she decided to test on her website:

• Discounts for specific services and fidelity campaigns
• First purchase discounts
• A new CTA for limited time to purchase with discounts
• New videos and blog posts for engagement
• New landing pages based on the weather
• Clients testimonials
• Online courses with simple practices users can try at home
• Personalizations based on users’ last sessions
• New heroes with engaging images and copies
• New headlines

If you’re not sure what to begin with, here are some important questions to consider:

• What is the goal of each test?
• What KPIs are you trying to improve?
• What data you have available for the area of improvement?]
• What is the impact of confirming these hypotheses?
• How long will it take to implement each test?
• Who needs to be involved in each task?

If you don’t know how to answer some of these questions, invite your team to collaborate and score the impact, confidence, and ease of the ideas you’ve come up with. You can use the ICE Score method to do that, which we’ll cover in the next section.

Deciding where to start can be one of the most challenging steps. Luckily, a smart method can help you with that: the ICE scoring.

The ICE score model is widely used by growth and product teams to prioritize features and experiments. It helps you evaluate each option by pointing out its impact, your confidence in its potential result, and the ease of implementation. Then, you can rank all options by multiplying these three values to calculate the score.

If you wish to know more about how it works, check out this blog post .

If, just like in the table above, you have different audiences you’d like to test, remember to take a step back to “how to organize your testing hypotheses”. Consider the relevance of your targeted audiences and which pain points you can address by testing new hypotheses.

You should also consider:

• What approaches your competitors already validated
• How your ads are performing
• What keywords bring you most traffic
• What trends there are in your industry right now
• Which personas are interacting the most with your product

Collecting existing data for your experiments and implementing it will have a huge impact on your marketing strategies throughout the next months (or years). Remember that prioritizing the right ideas saves you both time and money.

This should be the easiest step. Usually, your primary metric is very straightforward and highly related to your business goal. However, we strongly suggest you define secondary metrics to help you in the analysis: it is not unusual to run experiments that don't impact the primary conversion rate but change the mid-funnel metrics significantly.

The metrics you choose are generally defined by the goals you expect to achieve with your experiment. However, these are common points to pay attention to:

CTR: which specific elements in your test got the most interactions (a button, an image, a new CTA)? Is this change applicable to other slots throughout your website?

CAC and NPS: has the cost of acquiring new customers decreased? Are customers happy with their current experience?

ROI: did you get an equivalent return on investment of both time invested and costs?

AB tests have specific metrics you should analyze to validate hypotheses. But don’t forget to be creative in your analysis and formulate more hypotheses on why an experiment had more interaction, or how your audience would answer to a minor change. This will allow you to continue creating engaging content that resonates with all variations of your winning test.

From a purely statistical perspective, estimating the test duration is easy after determining the sample size. However, you have to take some things into account:

• What is your current conversion rate?
• What is the minimum improvement you expect to detect in your experiment?
• How many variations will the test have?

All these factors can affect the duration. But it is also important to highlight that you will only know it after your test runs. If the impact of the variant over the baseline is too small, you would probably want to run the test for at least a little while to observe statistical confidence.

You can use the calculator we provide in our free template .

The most used methods are the frequentist and the Bayesian .

The frequentist inference was developed in the 20th century and became the dominant statistical paradigm, widely used in experimental science. It is a statistically sound approach with valid results, but it presents limitations that aren't attractive in AB testing. On the other hand, the Bayesian approach has become the industry standard based on our benchmark, providing richer decision-making information, although the frequentist is still widely used.

Documenting AB tests should be a very straightforward exercise, but many folks dread this aspect of running experiments. It doesn't need to be demanding, so we made a template to help you organize the most critical information. It should guide you on documenting the hypothesis, the target metrics, the results, etc.

A free template guide for you

To help you plan your AB tests, we've designed a free template in a spreadsheet format .

This guide should provide you with:

• A list of ideas to test on your website
• A tool to help you prioritize your experiments using the ICE score
• A calculator to estimate how long you should run your tests
• A template for documenting your experiments

Feel free to download it and share it with your friends if you find it useful!

And if you want to rely on an easy to use platform for creating your tests autonomously and without needing daily help from devs, create your free account and explore Croct.

Learn practical tactics our customers use to grow by 20% or more.

Learn More About:

• Customer Acquisition
• Optimization
• Customer Experience
• Data & Analytics

How to Create an Effective A/B Test Hypothesis

Here's your comprehensive guide to A/B testing – how to develop a hypothesis, when to employ your test, and what to expect from your results.

The A/B test hypothesis you develop can make or break the effectiveness of your A/B test results. Moreover, your A/B test results can make or break the effectiveness of your ecommerce conversion optimization work. And we all know that your conversion rate can make or break the profitability of your business.

Here’s the quintessential rhetorical question for a digital marketing manager: Would it be worthwhile to learn more about creating an effective A/B test hypothesis?

Whether you were the type of student who worshipped the scientific method and couldn’t wait for chemistry class, or the type who still isn’t sure about the difference between a hypothesis and a hypotenuse, learning how to build a kick-butt A/B test hypothesis can open the door to increased profits and a steadily climbing return on investment for your ecommerce website.

You may already understand the basics of conversion rate optimization (CRO) testing, or you may just be getting started. You may already be familiar with how the A/B test hypothesis is formed, or this may be the first time you’ve heard of it. Either way, this article will help you leverage the power of CRO to get more conversions.

By the time we’re finished here, you’ll know all about the A/B test hypothesis: what it is, how to create it, and how to use it.

What is an A/B test hypothesis?

In ancient Greece, science and philosophy were considered partners. The aim of science was to increase wisdom, and the “hypothesis” (literally “placed under”) was the basis of a well-constructed argument. Socrates, Plato, and Aristotle knew that a sound hypothesis could lead one to a sound conclusion.

That unyielding search for truth led to the development of what we now know as the “scientific method,” which is the basis of the A/B testing we do today.

Here are the components of that process:

• Observe the situation, define a problem, and ask a question
• Formulate a hypothesis (a proposed solution or explanation)
• Test the hypothesis with experiments
• Collect and analyze the results
• Interpret the results and keep testing until you’re satisfied with the solution

Without the hypothesis, there can be no test – since proving or disproving the hypothesis is exactly what the test is about!

TIP:  While it’s true there is no “bad hypothesis,” (since you always learn something from each test), a tightly constructed A/B test hypothesis will get you closer to the solution quicker. Spend a little more time on developing the hypothesis, and you’ll need to spend less money on testing.

EXAMPLE:   TreeRing  helps schools and students create less expensive, creative yearbooks. They came to The Good for help getting more leads. We used the scientific process (listed above) to conduct A/B testing aimed at conversion rate optimization.

One part of the process went like this:

• Our team conducted a  conversion audit of the TreeRing website, which highlighted areas of concern based on data like the clicks and movements of their site visitors. We wanted to know how leads were generated and why lead generation conversion wasn’t hitting TreeRing’s goals.
• We formulated a hypothesis about the dropdown menu in the header . We theorized that moving the link to request a free sample to the top of the menu list would draw more clicks and boost the conversion rate.
• We tested the hypothesis by using the current menu as the “control”  and the new arrangement as the variation for the A/B test.
• We collected the results and crunched the data.  The test configuration sent 42 percent more visitors to the free sample landing page and drew 12 percent more requests for the sample.
• Since testing proved our first assumption true,  we then proceeded to develop and test another A/B test hypothesis to see if we could push the conversion rate even higher.

See the TreeRing case study for more insight.

Notes on the creation of an A/B testing hypothesis

Your ability to identify potential trouble spots and develop high-probability theories about how to repair them will grow over time. At The Good, our team members are skilled CRO researchers with years of entrenched experience. The more you work with conversion rate optimization and A/B testing in particular, the quicker and easier the process becomes.

To learn how to build an A/B test hypothesis, you need to get in there and practice. There is no substitute for hands-on experience. If you’re not sure how to get started, no problem.  Contact The Good . We’ll help you launch a robust CRO testing program.

TIP:  Finding something to test isn’t difficult. You can test every part of your ecommerce website – every word of copy, every layout, every configuration. The tough part is deciding which parts you  should  test. You’ll want to identify the areas that present the most potential for increasing conversions.

Begin with research

Closely observe the path your visitors take on their way to becoming customers and repeat customers. The journey to sales includes many micro conversions that lead to the macro conversion, the desired result – typically placing an order.

Your aim is to find areas where traffic is moving smoothly and areas where traffic is slowing down or dropping off. These are good tools to help you dig deep and uncover root problems. At The Good, we use a comprehensive conversion audit process. Our Stuck Score™ self-evaluation can also provide an excellent starting point.

You must ask yourself, “What business or customer experience problems do we think we can solve for this element of the site, and why do we think those changes will impact a certain KPI?”

The success of your optimization program is directly related to your ability to identify an A/B test hypothesis that will move the needle for your business.

Here’s a short list of tactics you can employ to take your research deeper:

• Look for case studies and blog articles related to the problems your  initial assessment  identified
• Collect and study customer feedback
• Collect and study feedback from customer service, your sales team, and your support team
• Use customer surveys to solicit specific feedback
• Pour over analytics data
• Employ tools like heatmapping to observe visitor behavior
• Employ user testing to observe the entire visitor experience

With experience, you’ll get a feel for how many of these you need to employ. For cost-effective CRO testing, you’ll want to make sure your investment is balanced favorably against the potential returns.

Create a list of possibilities and select your hypothesis

Your research will illuminate one or more stuck points – barriers to sales. Choose one of them to investigate further. What could you do to free up the flow of traffic at that point on the sales journey? Where would the metrics you’re observing need to be before you consider the effort successful? How can you achieve those metrics?

Before you settle on a hypothesis, brainstorm for possibilities. There are times when your first hunch is correct, and there are times when the lightbulb comes on after a dozen potential solutions are proposed.

Don’t evaluate the proposals until you’re out of ideas. Take a break, come back to the table and try one more time: What could we change that would move the results we’re seeing towards increased conversions? How could we make that change? What exactly would the change look like? What do we think would happen as a result of the change?

Rank your ideas and choose the theory most likely to create the desired effect. At The Good we use an RDI ranking scale: Risk, Difficulty, and Impact. Each is assigned a one to ten rating. Before investing time and money in a test, we want the average of those scores to be six or better. The winner is your hypothesis.

NOTE:  Your hypothesis isn’t a question. It’s a clear and measurable statement that answers your questions. For example, our hypothesis in the TreeRing example given above could be stated like this: “Moving the free sample link from the bottom of the menu to the top of the menu should increase the final conversion rate by at least 10 percent.”

Should you conduct A/B or multivariate tests?

The simplest, most direct testing method is to focus on changing just one thing at a time. A/B testing does just what it says. It compares version A against version B. In our TreeRing example, we tested leaving the free sample link at the bottom of the menu versus moving it to the top of the menu.

NOTE:  In A/B testing, it’s critical that you change only one thing at a time. Otherwise, you won’t be able to clearly determine the cause of the results.

Multivariate testing allows you to test more than one change at a time. You test combinations of variables simultaneously. The benefit to multivariate testing is that you can get to a total solution quicker than with A/B testing. Drawbacks are that it’s more complicated, more expensive, and will require more traffic to get dependable results .

Create your A/B test hypothesis and start testing!

You know what an A/B test hypothesis is, and you know how to create it. You know why testing is critical to conversion rate optimization, and you have access to the data you need to get started.

Use the steps and tips covered here to find a problem, develop a hypothesis, test the hypothesis, and evaluate the results. Use the information gleaned from that process to keep optimizing your conversions at every point on the sale journey.

If you need help (or even if you just want access to the tool), get your free customized Stuck Score™ from The Good. It benchmarks key conversion areas of your ecommerce website to find growth opportunities that may now be hidden.

Many of your competitors think creating an A/B test hypothesis is too complicated. They won’t take the next steps. But now you’re armed with the tools to move forward. That gives you a huge advantage. Move on it!

• How to Optimize a Website the Right Way
• How to Develop an Effective Conversion Strategy for your Business
• What Everybody Ought to Know About A/B and Multivariate Testing

Find out what stands between your company and digital excellence with a custom 5-Factors Scorecard™ .

About the author, jon macdonald.

Jon MacDonald is founder and President of The Good, a digital experience optimization firm that has achieved results for some of the largest companies including Adobe, Nike, Xerox, Verizon, Intel and more. Jon regularly contributes to publications like Entrepreneur and Inc.

A/B Testing Statistics: An Intuitive Guide For Non-Mathematicians

A/B testing statistics made simple. A guide that will clear up some of the more confusing concepts while providing you with a solid framework to AB test effectively.

Here’s the deal. You simply cannot A/B test effectively without a sound understanding of A/B testing statistics. It’s true. Data integrity is the foundation of everything we do as a Conversion Rate Optimization Agency .

And while there has been a lot of exceptional content written on AB testing statistics, I’ve found that most of these articles are either overly simplistic or they get very complex without anchoring each concept to a bigger picture.

Today, I’m going to explain the statistics of AB testing within a linear, easy-to-follow narrative. It will cover everything you need to use AB testing software effectively and I will make A/B Testing statistics simple.

Maybe you are currently using AB testing software. And you might have been told that plugging a few numbers into a statistical significance calculator is enough to validate a test. Or perhaps you see the green “test is significant” checkmark popup on your testing dashboard and immediately begin preparing the success reports for your boss.

In other words, you might know just enough about split testing statistics to dupe yourself into making major errors , and that’s exactly what I’m hoping to save you from today. Whether you are executing a testing roadmap in house or utilizing in 3rd party conversion optimization services , you need to understand the statistics so you can trust the results.

Here’s my best attempt at making statistics intuitive.

Why Statistics Are So Important To A/B Testing

The first question that has to be asked is “Why are statistics important to AB testing?”

The answer to that questions is that AB testing is inherently a statistics-based process. The two are inseparable from each other.

An AB test is an example of statistical hypothesis testing , a process whereby a hypothesis is made about the relationship between two data sets and those data sets are then compared against each other to determine if there is a statistically significant relationship or not.

To put this in more practical terms, a prediction is made that Page Variation #B will perform better than Page Variation #A. Then, data sets from both pages are observed and compared to determine if Page Variation #B is a statistically significant improvement over Page Variation #A.

This process is an example of statistical hypothesis testing.

But that’s not the whole story. The point of AB testing has absolutely nothing to do with how variations #A or #B perform. We don’t care about that.

What we care about is how our page will ultimately perform with our entire audience.

And from this birdseye view, the answer to our original question is that statistical analysis is our best tool for predicting outcomes we don’t  know using information we do  know .

For example, we have no way of knowing with 100% accuracy how the next 100,000 people who visit our website will behave. That is information we cannot  know today, and if we were to wait o until those 100,000 people visited our site, it would be too late to optimize their experience.

What we can  do is observe the next 1,000 people who visit our site and then use statistical analysis to predict how the following 99,000 will behave.

If we set things up properly, we can make that prediction with incredible accuracy, which allows us to optimize how we interact with those 99,000 visitors. This is why AB testing can be so valuable to businesses .

In short, statistical analysis allows us to use information we know to predict outcomes we don’t know with a reasonable level of accuracy.

A/B Testing Statistics: The Complexities Of Sampling, Simplified

That seems fairly straightforward. So, where does it get complicated?

The complexities arrive in all the ways a given “sample” can inaccurately represent the overall “population”, and all the things we have to do to ensure that our sample can accurately represent the population.

Let’s define some terminology real quick.

A little sampling terminology.

The “ population ” is the group we want information about. It’s the next 100,000 visitors in my previous example. When we’re testing a webpage, the true population is every future individual who will visit that page.

The “ sample ” is a small portion of the larger population. It’s the first 1,000 visitors we observe in my previous example.

In a perfect world, the sample would be 100% representative of the overall population.

For example:

Let’s say 10,000 out of those 100,000 visitors are going to ultimately convert into sales. Our true conversion rate would then be 10%.

In a tester’s perfect world, the mean  (average) conversion rate of any sample(s) we select from the population would always be identical to the population’s true conversion rate. In other words, if you selected a sample of 10 visitors, 1 of them (10%) would buy, and if you selected a sample of 100 visitors, then 10 would buy.

But that’s not how things work in real life.

In real life, you might have only 2 out of the first 100 buy or you might have 20… or even zero. You could have a single purchase from Monday through Friday and then 30 on Saturday.

The Concept of Variance

This variability across samples is expressed as a unit called the “ variance ”, which measures how far a random sample can differ from the true mean (average).

The Freakonomics podcast makes an excellent point about what “random” really is. If you have one person flip a coin 100 times, you would have a random list of heads or tails with a high variance.

If we write these results down, we would expect to see several examples of long streaks, five or seven or even ten heads in a row. When we think of randomness, we imagine that these streaks would be rare. Statistically, they are quite possible in such a dataset with high variance.

The higher the variance, the more variable the mean will be across samples. Variance is, in some ways, the reason statistical analysis isn’t a simple process. It’s the reason I need to write an article like this in the first place.

So it would not be impossible to take a sample of ten results that contain one of these streaks. This would certainly not be representative of the entire 100 flips of the coin, however.

Regression toward the mean

Fortunately, we have a phenomenon that helps us account for variance called “regression toward the mean”.

Regression toward the mean  is “the phenomenon that if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement.”

Ultimately, this ensures that as we continue increasing the sample size and the length of observation, the mean of our observations will get closer and closer to the true mean of the population.

Image Source

In other words, if we test a big enough sample for a sufficient length of time, we will get accurate “enough” results.

So what do I mean by accurate “enough”?

Understanding Confidence Intervals & Margin of Error

In order to compare two pages against each other in an Ab test, we have to first collect data on each page individually.

Typically, whatever AB testing tool you are using will automatically handle this for you, but there are some important details that can affect how you interpret results, and this is the foundation of statistical hypothesis testing, so I want to go ahead and cover this part of the process.

Let’s say you test your original page with 3,662 visitors and get 378 conversions. What is the conversion rate?

You are probably tempted to say 10.3%, but that’s inaccurate. 10.3% is simply the mean of our sample. There’s a lot more to the story.

To understand the full story, we need to understand two key terms:

• Confidence Interval
• Margin of Error

You may have seen something like this before in your split testing dashboard.

Understanding confidence intervals and margin of error.

The original page above has a conversion rate of 10.3% plus or minus 1.0%. The 10.3% conversion rate value is the mean . The ± 1.0 % is the margin for error , and this gives us a confidence interval  spanning from 9.3% to 11.3%.

10.3% ± 1.0 % at 95% confidence is our actual conversion rate for this page.

What we are saying here is that we are 95% confident that the true mean of this page is between 9.3% and 11.3%. From another angle, we are saying that if we were to take 20 total samples, we can know with complete certainty that the 19 of those samples would contain the true conversion rate within their confidence intervals.

The confidence interval  is an observed range in which a given percentage of test outcomes fall. We manually select our desired confidence level at the beginning of our test, and the size of the sample we need is based on our desired confidence level.

The range of our confidence level is then calculated using the mean  and the margin of error .

The easiest way to demonstrate this with a visual.

Confidence interval example.

The confidence level is decided upon ahead of time and based on direct observation. There is no prediction involved. In the above example, we are saying that 19 out of every 20 samples tested WILL, with 100% certainty, have an observed mean between 9.3% and 11.3%.

The upper bound  of the confidence interval is found by adding the margin of error to the mean. The lower bound  is found by subtracting the margin of error from the mean.

The margin for error  is a function of the standard deviation , which is a function of the variance . Really all you need to know is that all of these terms are measures of variability across samples.

Confidence levels are often confused with significance levels (which we’ll discuss in the next section) due to the fact that the significance level is set based on the confidence level, usually at 95%.

You can set the confidence level to be whatever you like. If you want 99% certainty, you can achieve it, BUT it will require a significantly larger sample size. As the chart below demonstrates, diminishing returns make 99% impractical for most marketers, and 95% or even 90% is often used instead for a cost-efficient level of accuracy.

In high-stakes scenarios (live-saving medicine, for example), testers will often use 99% confidence intervals, but for the purposes of the typical CRO specialist, 95% is almost always sufficient.

Advanced testing tools will use this process to measure the sample conversion rate for both the original page AND Variation B, so it’s not something you are really going to ever have to calculate on your own, but this is how our process starts, and as we’ll see in a bit, it can impact how we compare the performance of our pages.

Once we have our conversion rates for both the pages we are testing against each other, we use statistical hypothesis testing to compare these pages and determine whether the difference is statistically significant.

Important Note About Confidence Intervals

It’s important to understand the confidence levels your AB testing tools are using and to keep an eye on the confidence intervals of your pages’ conversion rates.

If the confidence intervals of your original page and Variation B overlap, you need to keep testing even if your testing tool is saying that one is a statistically significant winner.

Significance, Errors, & How To Achieve The Former While Avoiding The Latter

Remember, our goal here isn’t to identify the true conversion rate of our population. That’s impossible.

When running an AB test, we are making a hypothesis that Variation B will convert at a higher rate for our overall population than Variation A will. Instead of displaying both pages to all 100,000 visitors, we display them to a sample instead and observe what happens.

• If Variation A (the original) had a better conversion rate with our sample of visitors, then no further actions need to be taken as Variation A is already our permanent page.
• If Variation B had a better conversion rate, then we need determine whether the improvement was statistically large “enough” for us to conclude that the change would be reflected in the larger population and thus warrant us changing our page to Variation B.

So why can’t we take the results at face value?

The answer is variability across samples. Thanks to the variance, there are a number of things that can happen when we run our AB test.

• Test says Variation B is better & Variation B is actually better
• Test says Variation B is better & Variation B is not actually better ( type I error )
• Test says Variation B is not better & Variation B is actually better ( type II error)
• Test says Variation B is not better & Variation B is not actually better

As you can see, there are two different types of errors that can occur. In examining how we avoid these errors, we will simultaneously be examining how we run a successful AB test.

Before we continue, I need to quickly explain a concept called the null hypothesis.

The null hypothesis  is a baseline assumption that there is no relationship between two data sets. When a statistical hypothesis test is run, the results either disprove the null hypothesis or they fail to disprove the null hypothesis.

This concept is similar to “innocent until proven guilty”: A defendant’s innocence is legally supposed to be the underlying assumption unless proven otherwise.

For the purposes of our AB test, it means that we automatically assume Variation B is NOT a meaningful improvement over Variation A. That is our null hypothesis. Either we disprove it by showing that Variation B’s conversion rate is a statistically significant  improvement over Variation A, or we fail to disprove it.

And speaking of statistical significance…

Type I Errors & Statistical Significance

A type I error occurs when we incorrectly reject the null hypothesis.

To put this in AB testing terms, a type I error would occur if we concluded that Variation B was “better” than Variation A when it actually was not.

Remember that by “better”, we aren’t talking about the sample. The point of testing our samples is to predict how a new page variation will perform with the overall population. Variation B may have a higher conversion rate than Variation A within our sample, but we don’t truly care about the sample results. We care about whether or not those results allow us to predict overall population behavior with a reasonable level of accuracy.

So let’s say that Variation B performs better in our sample. How do we know whether or not that improvement will translate to the overall population? How do we avoid making a type I error?

Statistical significance.

Statistical significance  is attained when the p-value  is less than the significance level . And that is way too many new words in one sentence, so let’s break down these terms real quick and then we’ll summarize the entire concept in plain English.

The p-value  is the probability of obtaining at least as extreme results given that the null hypothesis is true.

In other words, the p-value is the expected fluctuation in a given sample, similar to the variance. Imagine running an A/A test, where you displayed your page to 1,000 people and then displayed the exact same page to another 1,000 people.

You wouldn’t expect the sample conversion rates to be identical. We know there will be variability across samples. But you also wouldn’t expect it be drastically higher or lower. There is a range of variability that you would expect to see across samples, and that, in essence, is our p-value.

The significance level  is the probability of rejecting the null hypothesis given that it is true.

Essentially, the significance level is a value we set based on the level of accuracy we deem acceptable. The industry standard significance level is 5%, which means we are seeking results with 95% accuracy.

So, to answer our original question:

We achieve statistical significance in our test when we can say with 95% certainty that the increase in Variation B’s conversion rate falls outside the expected range of sample variability.

Or from another way of looking at it, we are using statistical inference to determine that if we were to display Variation A to 20 different samples, at least 19 of them would convert at lower rates than Variation B.

Type II Errors & Statistical Power

A type II error occurs when the null hypothesis is false, but we incorrectly fail to reject it.

To put this in AB testing terms, a type II error would occur if we concluded that Variation B was not “better” than Variation A when it actually was better.

Just as type I errors are related to statistical significance, type II errors are related to statistical power , which is the probability that a test correctly rejects the null hypothesis.

For our purposes as split testers, the main takeaway is that larger sample sizes over longer testing periods equal more accurate tests. Or as Ton Wesseling of Testing.Agency  says here :

“You want to test as long as possible – at least 1 purchase cycle – the more data, the higher the Statistical Power of your test! More traffic means you have a higher chance of recognizing your winner on the significance level your testing on!

Because…small changes can make a big impact, but big impacts don’t happen too often – most of the times, your variation is slightly better – so you need much data to be able to notice a significant winner.”

Statistical significance is typically the primary concern for AB testers, but it’s important to understand that tests will oscillate between being significant and not significant over the course of a test. This is why it’s important to have a sufficiently large sample size and to test over a set time period that accounts for the full spectrum of population variability.

For example, if you are testing a business that has noticeable changes in visitor behavior on the 1st and 15th of the month, you need to run your test for at least a full calendar month.  This is your best defense against one of the most common mistakes in AB testing… getting seduced by the novelty effect.

Peter Borden explains the novelty effect in this post:

“Sometimes there’s a “novelty effect” at work. Any change you make to your website will cause your existing user base to pay more attention. Changing that big call-to-action button on your site from green to orange will make returning visitors more likely to see it, if only because they had tuned it out previously. Any change helps to disrupt the banner blindness they’ve developed and should move the needle, if only temporarily.

More likely is that your results were false positives in the first place. This usually happens because someone runs a one-tailed test that ends up being overpowered. The testing tool eventually flags the results as passing their minimum significance level. A big green button appears: “Ding ding! We have a winner!” And the marketer turns the test off, never realizing that the promised uplift was a mirage.”

By testing a large sample size that runs long enough to account for time-based variability, you can avoid falling victim to the novelty effect.

Important Note About Statistical Significance

It’s important to note that whether we are talking about the sample size or the length of time a test is run, the parameters for the test MUST be decided on in advance.

Statistical significance cannot be used as a stopping point or, as Evan Miller details , your results will be meaningless.

As Peter alludes to above, many AB testing tools will notify you when a test’s results become statistical significance. Ignore this. Your results will often oscillate between being statistically significant and not being statistically significant.

Statistical significance. Image source: Optimizely.

The only point at which you should evaluate significance is the endpoint that you predetermined for your test.

Terminology Cheat Sheet

We’ve covered quite a bit today.

For those of you who have just been smiling and nodding whenever statistics are brought up, I hope this guide has cleared up some of the more confusing concepts while providing you with a solid framework from which to pursue deeper understanding.

If you’re anything like me, reading through it once won’t be enough, so I’ve gone ahead and put together a terminology cheat sheet that you can grab. It lists concise definitions for all the statistics terms and concepts we covered in this article.

Download The Cheat Sheet

• First Name *
• Recent Posts

• AB Testing Research: Do Your Conversion Homework - December 8, 2018
• 8 Elements of a High Converting Squeeze Page - June 21, 2018
• 10 Conversion Lessons For Online Retail from Amazon - November 29, 2017

You might also like

The “endpoint you predetermined” – – are you referring to effect size or…

Hey Andrew, no that is in reference to time. Statistical significance should be evaluated at a predetermined end date that allows for a full cycle of customer behavior. One month is a good baseline.

Hey Jacob, A rarely good article, indeed! Statistical power is so often overlooked in A/B tests, that it’s nice to see it mentioned and in detail, too! The assumption of having a fixed sample size (as also discussed in other comments) is what makes robust application of the statistical tools especially hard to apply in A/B testing practice. I mean, everyone wants to end a test as quickly as possible, either to reap the benefits or cut the loses, but the statistical method does not allow for sequential evaluation of the data. Right? Not exactly – the method has been modified to fit the use case of medical testing, which in fact is no different than the one in online marketing and UX. If a robust solution for statistical design and sequential evaluation of A/B tests that allows you to run tests 20-80% faster would be interesting for you, I’d suggest you look into the AGILE A/B Testing statistical approach. A free white paper detailing it is available here: https://www.analytics-toolkit.com/whitepapers.php?paper=efficient-ab-testing-in-cro-agile-statistical-method . Best, Georgi

There is no such thing as bad data!

You’re going to have to say more about that…

Theres a such thing as incomplete data which makes it useless for scientific inquiry

The truth is, David, that ALL data is incomplete. But there is no good argument that it is all useless for scientific inquiry. The more you have the better your decisions.

You got the confidence intervals wrong! “From another angle, we are saying that if we were to take 20 total samples, we can know with complete certainty that the sample conversion rate would fall between 9.3% and 11.3% in at least 19 of those samples.” You can only say that in 19 out of the 20 samples, their corresponding confidence intervals will contain the true conversion rate. Your interval endpoints, 9.3% and 11.3%, are simply random draws here. You can’t put them in direct relation with the population conversion rate as you did here.

Anton, I believe you are correct. I’ve updated the article. I hope this doesn’t obscure the point, but it is statistics. Thanks for pointing this out.

Thanks for explaining it in a way that non-mathematicians can actually understand. I’ve started AB testing in mailings recently and couldn’t quite interpret the data correctly. I also ended my tests too soon, eliminating the possibility of having decent statistics… This made some things clearer and I look forward to learning more!

Trackbacks & Pingbacks

[…] Reference: https://conversionsciences.com/ab-testing-statistics/ […]

[…] 7. AB testing […]

[…] how to analyze statistical data from experiments, including A/B tests. The chapter discusses two ways to statistically analyze results from A/B tests, those include: T-Tests and Bayesian statistical applications. Then for the purpose of analyzing […]

[…] A/B Testing Statistics: A Guide for Non-Mathematicians — 全面降低A / B测试和初学者的统计数据的难度。  […]

[…] A/B Testing Statistics: A Guide for Non-Mathematicians — 全面降低A / B测试和初学者的统计数据的难度。 […]

[…] For a closer look at the statistics behind A/B testing, check out this in-depth post: AB Testing Statistics: An Intuitive Guide For Non-Mathematicians […]

[…] a worst case scenario, a competitor can see what hypotheses you are testing. They can then test those same ideas and perhaps win more […]

[…] One piece of this that most people are familiar with is statistical significance. Unfortunately, very few people actually understand statistical significance at the level needed to set up split tests. If you suspect that might be you, check out AB Testing Statistics: An Intuitive Guide For Non-Mathematicians. […]

[…] https://support.abtasty.com/hc/en-us/articles/205811297 https://www.abtasty.com/blog/clever-stats-finally-statistics-suited-to-your-needs/ https://betterexplained.com/articles/an-intuitive-and-short-explanation-of-bayes-theorem/ https://blog.kissmetrics.com/how-ab-testing-works/ https://conversionsciences.com/ab-testing-statistics/ […]

[…] It’s not about guessing or being creative. It’s about analyzing how visitors are engaging with a site and then turning that data into testable hypotheses that can be measured against the existing site via a statistically valid testing process. […]

[…] Depending on the size of your email list you might need to perform this test more than once to get statistically significant results. […]

[…] If you have a smaller list, you might need to run this experiment several times in order to get statistically significant results, but what you should start to see is that certain days and times will consistently perform better […]

[…] Why? Because your site doesn’t even have enough conversions to make A/B testing worthwhile. At such low conversion rates, you’d have to let the test run for months or years to get a statistically valid result. […]

[…] emotions and motivations are representative of the broader market. Quantitative data gives us more statistical confidence that what we are seeing represents the larger market. However, this data isn’t seasoned with […]

[…] our post on AB testing statistics, we discussed type I and type II errors. We work to avoid these errors at all […]

[…] our post on AB Testing Statistics, we make this offer at the […]

[…] an AB test with two variations, we may be able to reach statistical significance in two weeks, and bank a 10% increase in conversions. However, in a test with six variations, we […]

[…] If you promote via paid channels, simply create two ads, using a different headline for each, and set up a normal AB test using proper statistical analysis. […]

[…] Marketers are picking up the slide rules of modern digital marketing: analytics, statistics and experimenting. They are asking the nagging question of conversion optimization. And they are building their chops on statistical analysis. […]

[…] Marketers are picking up the slide rules of modern digital marketing: analytics, statistics and experimenting. They are asking the nagging question of conversion optimization. And they are building their chops on statistical analysis. […]

[…] testing is a statistical approach to gathering data and making decisions. There is a minimum number of transactions you will want […]

[…] every test, we seek to “beat” the existing control, the page or experience that is currently performing the best of all treatments we’ve tried. […]

[…] faster than bad data. In order to do testing right, there are some things you need to know about AB testing statistics. Otherwise, you’ll spend a lot of time trying to get answers, but instead of getting answers, […]

[…] Tools for collecting data in order to make good hypotheses […]

Leave a Reply

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

I agree to the terms and conditions laid out in the Privacy Policy

• Optimization Services
• Guaranteed Redesign
• Fully-managed CRO
• AI Optimization
• CRO Training & Coaching
• Conversion Solutions
• eCommerce Optimization
• Lead Generation Solutions
• CRO for Website Redesign
• CRO for Mobile Lead Gen
• CRO for Advertising
• About Conversion Sciences
• Success Stories
• CRO Resources
• Your Customer Creation Equation Book
• Free CRO Course
• CRO Calculator
• Press & Speaking Dates

This site uses cookies. By continuing to browse the site, you agree to our use of cookies. Or find out how to manage cookies .

Cookie and Privacy Settings

We may request cookies to be set on your device. We use cookies to let us know when you visit our websites, how you interact with us, to enrich your user experience, and to customize your relationship with our website.

Click on the different category headings to find out more. You can also change some of your preferences. Note that blocking some types of cookies may impact your experience on our websites and the services we are able to offer.

These cookies are strictly necessary to provide you with services available through our website and to use some of its features.

Because these cookies are strictly necessary to deliver the website, refusing them will have impact how our site functions. You always can block or delete cookies by changing your browser settings and force blocking all cookies on this website. But this will always prompt you to accept/refuse cookies when revisiting our site.

We fully respect if you want to refuse cookies but to avoid asking you again and again kindly allow us to store a cookie for that. You are free to opt out any time or opt in for other cookies to get a better experience. If you refuse cookies we will remove all set cookies in our domain.

We provide you with a list of stored cookies on your computer in our domain so you can check what we stored. Due to security reasons we are not able to show or modify cookies from other domains. You can check these in your browser security settings.

We also use different external services like Google Webfonts, Google Maps, and external Video providers. Since these providers may collect personal data like your IP address we allow you to block them here. Please be aware that this might heavily reduce the functionality and appearance of our site. Changes will take effect once you reload the page.

Google Webfont Settings:

Google Map Settings:

Google reCaptcha Settings:

Vimeo and Youtube video embeds:

You can read about our cookies and privacy settings in detail on our Privacy Policy Page.

• A/B (or Split) Testing
• Affiliate Marketing
• Cart Abandonment
• Click Through Rate
• Compliance & Regulation
• Conversion Rate Optimization
• Customer Data Platform
• Customer Engagement
• Customer Experience
• Data Security & Privacy
• Grow Traffic and Subscribers
• Landing Page Optimization
• Mobile App Insights
• Mobile App Testing
• Multivariate Testing
• Partner Ecosystem
• Personalization
• Segmentation & Targeting
• Server-Side Testing
• Usability Testing
• Visitor Behavior Analytics
• Web Insights
• Web Testing
• Website Analysis
• Website Optimization
• Website Redesign
• Product Updates

A/B Testing Solutions to Maximize Your Website Conversions

Create and A/B test different versions of your website and its elements to continuously discover the best-performing versions that improve your conversions.

Follow us and stay on top of everything CRO

How to create a strong a/b testing hypothesis.

Experimenting without a hypothesis is akin to getting lost in a labyrinth—it may appear that it’s leading you somewhere , but such pointless wandering can seldom lead you to your desired destination.

Whether it’s reaching your destination or hitting a conversion goal on your website, the process involves numerous baby steps that you take in the form of iterative testing and hypothecation. Along with the process, they keep evolving to find an optimal route to desired conversions. 

Download Free: A/B Testing Guide

In this post, you will learn about constructing strong A/B testing hypotheses , which will ensure you are moving in the right direction to reach your destination in time.

What is a hypothesis?

A hypothesis is a proposed explanation or solution to a problem. Think of it as a glue that ties the problem to a solution. For instance, you could hypothesize that adding trust badges to your payment page could cater to the problem of low conversion rates on that page. As you’d notice, the hypothesis is made up of two variables, namely the cause (the action we want to test) and effect (the outcome we expect).

One day you wake up and want to run a test for the color of the CTA button on your website. You rush and achieve conversion lifts with this change, but you do not know exactly what you wanted from this test? 

You did not pay any heed to your conversion funnel . That is, you did not analyze the test fully before executing it! 

There is a difference between running a test scientifically to prove something and generating random results.Now, this something is your hypothesis.

Components of a hypothesis

As you would notice here, in the world of user experience optimization , a strong hypothesis is made up of three components: defining a problem, describing a proposed solution, and measuring results.

For instance, you could hypothesize that adding trust badges to your payment page could fix low conversion rates on that page. You can find out why that happens by identifying the right metrics for success.

But how does one begin to start formulating a hypothesis?

How to formulate a winning hypothesis?

It is not wise to blindly follow best practices crafted by someone else for your business. Every business is unique, so are its strategies. It is always recommended to construct your own best practices. 

Below are some essential elements that make a solid hypothesis:

1. They aim to alter customer behavior, either positively or negatively.

A psychological principle often forms the basis of a hypothesis that triggers a reaction from prospects. 

It changes the way they perceive your offer/brand, which alters their behavior during the test. Like in the headline example below, the urgency of the new message is why the variation headline is expected to perform better than the original headline.

Changing the headline from ‘Grab your tickets now!’ to 

‘Tickets filling out soon – only last 50 left!’ could increase ticket sales online. 

Only because you follow the above syntax to formulate a hypothesis doesn’t mean that you’ve got the winning hypothesis.

2. They focus on deriving customer learning from tests.

When pros develop a hypothesis , their focus is on the big picture. 

So your test might give you any result. What’s important is for you to analyze ‘why’ your visitors behaved in a certain way? 

As data-driven marketers, it might seem difficult sometimes to make your peace with negative lifts. 

But if the test reveals significant customer learning, it can pave the way for colossal conversion lifts in the future. 

For example, when Michael Aagaard, the conversion copywriter of Content Verve, conducted a CTA copy test on his client’s website. He learned that changing the CTA copy from ‘Start your 30 day free trial’ to ‘Start my 30-day free trial’ resulted in a 90% increase in sign-ups.

However, he cautioned that applying this technique would not work across the linguistic globe though.

3. They are derived from at least some evidence.

You can explore many avenues to construct good hypotheses. 

The image below by the popular Conversion Expert, Craig Sullivan, shows several ways to collect meaningful insights.

a. Usability testing

In simple words, you can sit and observe how your test participants use your website. 

Make notes where they get stuck or confused. Form your hypotheses to improve these situations and let A/B tests decide if they work for you.

This method gives you exceptional insights into your customers’ workarounds and struggles in using your website. 

Write down the exact questions you would like to ask during your research. Asking questions related to the homepage, checkout, pricing page, and navigation reveal great insights. 

Some sample questions you can ask are: 

• Was it easy to find what you were looking for? 
• Were the words/vocab used to define categories/sub-categories clear to you? 
• Do you have any suggestions to improve our website navigation? 
• Does our website look credible to you? 
• Is our pricing transparent? 
• Is there anything else you’d like to know before signing up with us? 
• Will you shop with us again? Why/why not? 
• Do you think the form has any confusing/unnecessary input fields?

b. Customer surveys

Surveys are an effective tool for understanding your customer hesitations and knowing their intent/pain points or concerns. They help you identify the optimization opportunities on your websites .

Here are some sample questions to use for on-site surveys:

Visiting the pricing page shows the intent of buying. See how Qualaroo leverages targeted traffic on their pricing page to understand customer pain points:

Ask these survey questions during cancellation of a subscription:

To existing customers, you can ask: 

Have you ever criticized/praised us to someone in the past few months? 

What did you say? 

If you are made the owner of [insert your company name], what would you change? 

For those who have just signed up with you, you can send an auto-generated mail asking: 

Did you have any doubts or concerns about our product/service before you signed up? 

What made you overcome those doubts? How did you hear about us? 

Too many questions can be annoying, especially in on-site surveys. Make sure you respect your prospects’ choice if they choose not to answer your questions. 

Similarly, off-site surveys also serve the same purpose of gathering feedback via email or third-party survey websites. 

Apart from SurveyMonkey, you could also use Fluid Surveys and Confirmit for designing off-site feedback surveys.

c. Heatmaps

The data in your testing tool can tell you what your visitors are doing but not why they are doing it. 

Heatmaps can help you identify interest areas of your prospects as well as what areas they choose to ignore . Sometimes this can help you identify great insights when a vital page element goes unnoticed by visitors because some other element on the page is stealing its thunder. 

The heatmap below shows how a non-clickable element in the image takes away all the attention from the CTA on the page:

Later, when this element was removed, the heatmap shows a clear emphasis on the call-to-action of the page (as it should be):

Without referring to their heatmap, TechWyse had never understood the reason for their dropping conversion numbers.

If not this, heatmaps can sometimes also help you figure out a page element/navigation item that may be taking too much of important online real estate while being completely ignored by users. 

You can consider replacing it with something more relevant to your conversion goal and conduct an A/B test to see how it performs. 

d. Website analytics

You might know this already, but exploring data in your website analytics can give you some great insights to get started. 

For instance, the hypothesis of “adding trust badges on payment page” we formed earlier, could have originated from a “high exit rate of the page.” The exit rate of the page — along with other metrics — can be found within your website analytics. 

Website analytics tools like Google Analytics can show you quantitative data on how visitors navigate your website on a site architecture level. 

Some of the important metrics that you could track to validate an idea and build a hypothesis are:

• Traffic report: Metrics like total traffic, the total number of visitors (overall and on individual pages) could help you track how many people the test will impact and how long it would take to finish it. 
• Acquisition report: This could help you determine where your visitors are coming from (your best traffic sources) and how the performance differs between different channels. 
• Landing page report: Your top landing and exit pages show how visitors enter and leave the site. 
• Funnel report: This would give you insights into questions like where your visitors enter into or exit from your marketing funnel and how they navigate between the different pages.
• Device type: This will help you decide whether you should focus on optimizing the user experience on a particular device on priority. 

For any observation that you come across from analyzing these, ask yourself enough ‘why’s to form a solid hypothesis.

Watch this video to understand the hypothesis workflow in VWO.

Test with confidence

Quality of your insights and knowledge of how you can use them is what sets solid hypotheses apart from random testing. Now since the grunt work has been done, nothing should stop you. Forming a well-structured hypothesis is a critical piece in the conversion optimization puzzle. It helps you identify and remove the friction along your conversion funnel.

Document every outcome of your test. Remember that each test you conduct is an opportunity to learn. You might find that you have multiple hypotheses to try. But instead of running in all directions, prioritize your tests and monitor them for conversions and insights, which lay the foundation to create solid hypotheses for your subsequent experiments.

Categories:

Related content

More from vwo on a/b testing.

A/B Testing Will Get You a Promotion

I was talking to a colleague yesterday and one of his offhand remarks struck a…

Paras Chopra

Should you hire an agency for your A/B testing or should you do it yourself?

To be the most successful with your website testing efforts and results, you need more…

[Infographic] 14 Times In Business You Should A/B Test

A/B testing is the scientific way of arriving at the truth, or at least the…

Mohita Nagpal

Scale your A/B testing and experimentation with VWO.

Opt for higher roi on your optimization efforts.

Take a 30 -day, all-inclusive free trial with VWO.

By signing up, you agree to our Terms & Privacy Policy

DOWNLOAD A/B TESTING FREE E-BOOK

Thanks for sharing the details. Please check your inbox for the free guide.

Talk to a sales representative

Get in touch

By submitting, you agree to our Terms & Privacy Policy

Thank you for writing to us!

One of our representatives will get in touch with you shortly.

Signup for a full-featured trial

Free for 30 days. No credit card required

Set up your password to get started

Awesome! Your meeting is confirmed for at

Thank you, for sharing your details.

Hi 👋 Let's schedule your demo

To begin, tell us a bit about yourself

By continuing, you agree to our Terms & Privacy Policy

While we will deliver a demo that covers the entire VWO platform, please share a few details for us to personalize the demo for you.

Select the capabilities that you would like us to emphasise on during the demo., which of these sounds like you, please share the use cases, goals or needs that you are trying to solve., please share the url of your website..

We will come prepared with a demo environment for this specific website.

I can't wait to meet you on at

, thank you for sharing the details. Your dedicated VWO representative, will be in touch shortly to set up a time for this demo.

We're satisfied and glad we picked VWO. We're getting the ROI from our experiments. Christoffer Kjellberg CRO Manager

VWO has been so helpful in our optimization efforts. Testing opportunities are endless and it has allowed us to easily identify, set up, and run multiple tests at a time. Elizabeth Levitan Digital Optimization Specialist

As the project manager for our experimentation process, I love how the functionality of VWO allows us to get up and going quickly but also gives us the flexibility to be more complex with our testing. Tara Rowe Marketing Technology Manager

You don't need a website development background to make VWO work for you. The VWO support team is amazing Elizabeth Romanski Consumer Marketing & Analytics Manager

Trusted by thousands of leading brands

Share on Mastodon

✂️ The Future of Marketing Is Personal: Personalize Experiences at Scale with Ninetailed AI Platform Ninetailed AI →

• Growth, 
• Experimentation

A/B Testing Best Practices: How to Create Experiments That Convert

Like any tool, the efficacy of A/B testing lies in its correct usage.

It's not as simple as changing the color of a button on your landing page or tweaking the subject line of an email. The process involves careful planning, execution, and analysis.

In this blog post, we will delve into the best practices for A/B testing. We'll explore how to:

formulate a strong hypothesis,

select the right variables to test,

ensure your sample size is representative,

and accurately interpret the results.

We'll also discuss the common pitfalls to avoid and how to ensure your tests contribute to a better understanding of your audience and their preferences.

By the end of this post, you'll be equipped with the knowledge to create A/B testing experiments that not only convert but also provide valuable insights to fuel your future marketing strategies.

1. Start with a Hypothesis

A hypothesis, in the realm of A/B testing, is an educated guess or assumption about what you believe could improve the performance of your webpage, email, or other marketing assets. It's a prediction about the relationship between two variables: the element you are changing (independent variable) and the outcome you want to influence (dependent variable).For example, let's say you have noticed that the conversion rate on your product page is lower than industry standards. You might hypothesize that changing the color of the "Add to Cart" button from grey (which might blend with the background) to a bright and bold color like red (which stands out) will make it more noticeable and therefore increase click-throughs and conversions.

In this case, your hypothesis might be stated as: "If we change the 'Add to Cart' button color to red, then the conversion rate will increase because the button will be more noticeable."

Starting with a hypothesis is crucial for a few reasons:

Direction: It gives your test a clear direction and purpose. Knowing what you're testing and why helps you focus on achieving specific goals.

Measurement: It enables you to measure the impact of your changes. By defining what you expect to happen, you can better assess whether the change had the desired effect.

Insight: It provides valuable insights into user behavior. Even if your hypothesis turns out to be incorrect, you still gain useful information about what doesn't work, helping you refine future tests.

Efficiency: It saves time and resources. By focusing on testing elements based on a well-thought-out hypothesis, you avoid random testing, which may not yield meaningful results.

Remember, a good hypothesis is specific, testable, and based on research and data. It's not just a random guess but a well-informed assumption that guides your A/B testing towards meaningful improvements.

2. Test One Element at a Time

The importance of testing one element at a time during A/B testing cannot be stressed enough.

This approach, also known as "isolated testing," is crucial to identify what is driving changes in your performance metrics accurately.

Let's consider an example. Suppose you decide to test a new headline and a different call-to-action (CTA) button color simultaneously on your landing page. If you notice an improvement in conversion rates, it would be impossible to discern whether the change was due to the new headline, the altered CTA color, or a combination of both.

By testing multiple elements at once, you muddy the waters and make it difficult to draw clear conclusions from your data. The results become ambiguous, and you lose the opportunity to gain precise insights about the impact of each individual change.

On the other hand, if you test one element at a time - first the headline, then the CTA color - you can clearly attribute any change in performance to the specific element you modified. This provides more actionable insights that you can use to optimize further.

To implement this approach effectively:

Prioritize Your Tests: Not all elements have the same impact on conversions. Prioritize testing those elements that are likely to have a significant effect on user behavior, such as headlines, CTAs, or images.

Plan Your Tests: Create a testing roadmap where you outline what elements you will test and in what order. This helps you stay organized and ensures you don’t skip important elements.

Analyze and Iterate: After each test, analyze the results, implement the winning version, and then move on to the next element. Remember, CRO is a continuous process of testing, learning, and improving.

3. Use a Representative Sample Size

Having a representative sample size is another critical component of successful A/B testing. It's the key to obtaining reliable and statistically significant results.

In A/B testing, your sample size refers to the number of users who are exposed to each version of your test. If your sample size is too small, your results may be influenced by random chance rather than reflecting genuine user behavior or preferences. On the other hand, if you have a large enough sample size, you're more likely to capture a true representation of your audience's responses.

Let's illustrate this with an example: Imagine you're testing two headlines on your website, and you only expose each version to 10 visitors. Even if one headline outperforms the other, with such a small sample size, it's hard to confidently say that the result wasn't due to chance. However, if you tested each headline with 1,000 visitors, your results would be much more reliable.

Here are some tips to ensure a representative sample size in your A/B tests:

Calculate the required sample size before starting the test. There are many online tools and calculators available that can help you determine the optimal sample size based on your website's traffic, expected conversion rates, and desired confidence level.

Run the test until you reach your desired sample size. Cutting a test short could lead to inaccurate results. Be patient and allow the test to run until you've reached your pre-determined sample size.

Ensure your sample is diverse. To get a true representation of your audience, make sure your sample includes a mix of different types of users (new visitors, returning visitors, users from different locations, etc.).

Remember, the goal of A/B testing is not just to find out which version is better, but to gain insights that you can confidently apply to optimize your marketing strategy.

4. Allow Sufficient Run Time

The statistical significance and reliability of test results greatly depend on not just the sample size, but also on the duration of the test.

If you stop a test too early, you risk making decisions based on incomplete or misleading data. For instance, if you launch a test and see a dramatic increase in conversions within the first few hours or days, it might be tempting to declare a winner and implement changes immediately. However, such a hasty decision can be problematic due to several reasons:

Initial Fluctuations: It's common to see large swings in performance when a test first starts. These often settle down over time, and early results may not reflect the true effect of the change.

Variability in User Behavior: User behavior can vary significantly depending on the day of the week, time of the day, or even season of the year. Running a test for a short period may only capture a subset of your audience's behavior.

Statistical Significance: The longer a test runs (assuming it's receiving enough traffic), the more confident you can be in the results. Short tests are more susceptible to random variations that can lead to false positives or negatives.

As a rule of thumb, it's recommended to run a test for at least one full business cycle (usually a week) to account for daily and weekly variations in user behavior. However, the exact duration can depend on factors like your website's traffic, baseline conversion rate, and the minimum detectable effect.

5. Analyze and Interpret the Results Correctly

Analyzing the test results is not just about identifying the winning variant, but also understanding why one version performed better than the other and how these insights can be applied to future optimization efforts.

Surface-level data such as conversion rates and click-through rates can provide a quick overview of which variant performed better. However, deeper analysis is required to fully understand the implications of your test results. Here's how you can go about it:

Segment Your Data: Break down your results by different user segments such as new vs. returning visitors, different traffic sources, device types, geographic locations, etc. This can reveal valuable insights and help you understand if certain changes work better for specific segments of your audience.

Analyze Secondary Metrics: Don't just focus on your primary conversion goal. Look at how the test affected secondary metrics like time on page, bounce rate, pages per visit, etc. This can provide a more holistic view of user behavior and the overall impact of the test.

Look for Statistical Significance: Ensure that your results are statistically significant. This means that the difference in performance between the two versions is not due to random chance. Tools like a p-value calculator can help with this.

Draw Conclusions and Hypotheses: Based on your analysis, draw conclusions about why one version outperformed the other. Use these insights to form new hypotheses for future tests.

Document Everything: Keep a record of all your tests, results, and learnings. This will help you build a knowledge base and avoid repeating unsuccessful tests in the future.

Remember, the goal of A/B testing is not just to get a lift in conversions and engagement, but also to gain a deeper understanding of your users and their behavior. By analyzing and interpreting your results correctly, you can ensure that your testing efforts contribute to long-term, sustainable growth.

6. Iterate and Improve

The goal of CRO is not just to find a "winning" version and stop there, but to continuously learn about your users, iterate on your designs, and improve your website's performance over time.

A/B testing is essentially a scientific method applied to your website or app. You formulate a hypothesis, design an experiment (the A/B test), collect data, and then analyze the results. But the process doesn't end there. Based on what you've learned, you then create a new hypothesis and start the process over again.

Let's say, for example, you run an A/B test on your product page, changing the color of the "Add to Cart" button from blue to green. The green button results in a 10% increase in clicks. Great! But don't stop there. Now you might ask: "Would a different shade of green result in even more clicks?" or "What if we make the button larger?" or "What if we change the text on the button?" Each of these questions can form the basis of a new A/B test.

Here are some tips for iterating and improving through A/B testing:

Be Methodical: Don't change too many things at once. If you do, you won't know which change caused the difference in performance. Stick to one variable at a time whenever possible.

Keep Learning: Even "failed" tests—those where there was no significant difference between versions or where the original version outperformed the new one—are valuable. They give you insights into what doesn't work for your audience.

Prioritize Your Tests: Not all changes are created equal. Prioritize tests based on potential impact and ease of implementation.

Patience and Persistence: Optimization is a long-term process. Don't be discouraged by tests that don't result in a big lift. Even small, incremental improvements can add up over time.

To sum up, A/B testing is about much more than finding a "winning" version. It's a tool for continuous learning and improvement. Always keep testing, tweaking, and learning from your findings.

7. Document Everything

Documentation is a crucial part of the optimization process. It might seem like an administrative task, but it serves several important purposes in your CRO strategy.

By documenting everything, you create a historical record of your tests, which can be extremely valuable for several reasons:

Learning from Past Tests: By documenting the results of each test, you can see what worked and what didn't. This can help you avoid repeating the same mistakes and also build upon successful strategies.

Understanding Your Audience: Over time, your testing documents will provide a composite picture of your audience's preferences and behavior. For instance, you may notice that certain types of headlines consistently perform better, or that your audience responds well to specific calls to action. These insights can guide future tests and broader marketing strategies.

Informing Future Tests: When planning new tests, it's helpful to look back at previous ones for ideas and insights. You may find patterns that suggest new hypotheses to test.

Maintaining Consistency: Documenting your tests also helps ensure consistency in how you conduct and evaluate them. For example, you can note down the statistical significance level you're using, how you segment your data, etc. This makes it easier to compare results across different tests.

Communicating Results: If you're part of a larger team, documentation can help you communicate your findings to other stakeholders. It provides a clear, objective record of what was tested, the results, and any changes that were implemented as a result.

In terms of what to document, you should include the hypothesis of the test, the elements that were changed, the duration of the test, the results (including statistical significance), and any observations or conclusions. Tools like Google Sheets or project management software can be used to keep track of all this information

The Bottom Line

The true power of A/B testing lies not just in executing tests but in adopting a systematic, data-driven approach to understanding your users and their behavior.

From formulating a strong hypothesis, designing effective experiments, correctly analyzing and interpreting results, to continuously iterating based on findings, each step plays a crucial role in the success of your A/B tests. Remember, it's not just about finding a winning variant, but about gaining insights that can lead to ongoing improvements in your conversion rate.

Documenting your tests and results is equally important. It helps build a knowledge base, informs future tests, and provides a clearer understanding of your audience over time.

A/B testing isn't a one-time effort but a journey of continuous learning and improvement. With these best practices in mind, you're well-equipped to create experiments that convert, ultimately boosting your business's bottom line.

Download Now 27 A/B Testing Ideas to Boost Conversions Learn Now

Keep Reading on This Topic

This guide dives into the details of A/B testing inside Contentful, illustrating how to increase conversions through a successful experimentation strategy.

In this post, we’ll explore four benefits of using Ninetailed and Contentful together for composable A/B testing and experimentation.

11 A/B Testing Examples From Real Businesses

Published: April 21, 2023

Whether you're looking to increase revenue, sign-ups, social shares, or engagement, A/B testing and optimization can help you get there.

But for many marketers out there, the tough part about A/B testing is often finding the right test to drive the biggest impact — especially when you're just getting started. So, what's the recipe for high-impact success?

Truthfully, there is no one-size-fits-all recipe. What works for one business won't work for another — and finding the right metrics and timing to test can be a tough problem to solve. That’s why you need inspiration from A/B testing examples.

In this post, let's review how a hypothesis will get you started with your testing, and check out excellent examples from real businesses using A/B testing. While the same tests may not get you the same results, they can help you run creative tests of your own. And before you check out these examples. be sure to review key concepts of A/B testing.

A/B Testing Hypothesis Examples

A hypothesis can make or break your experiment, especially when it comes to A/B testing. When creating your hypothesis, you want to make sure that it’s:

• Focused on one specific problem you want to solve or understand
• Able to be proven or disproven
• Focused on making an impact (bringing higher conversion rates, lower bounce rate, etc.)

When creating a hypothesis, following the "If, then" structure can be helpful, where if you changed a specific variable, then a particular result would happen.

Here are some examples of what that would look like in an A/B testing hypothesis:

• Shortening contact submission forms to only contain required fields would increase the number of sign-ups.
• Changing the call-to-action text from "Download now" to "Download this free guide" would increase the number of downloads.
• Reducing the frequency of mobile app notifications from five times per day to two times per day will increase mobile app retention rates.
• Using featured images that are more contextually related to our blog posts will contribute to a lower bounce rate.
• Greeting customers by name in emails will increase the total number of clicks.

Let’s go over some real-life examples of A/B testing to prepare you for your own.

A/B Testing Examples

Website a/b testing examples, 1. hubspot academy's homepage hero image.

Most websites have a homepage hero image that inspires users to engage and spend more time on the site. This A/B testing example shows how hero image changes can impact user behavior and conversions.

Based on previous data, HubSpot Academy found that out of more than 55,000 page views, only .9% of those users were watching the video on the homepage. Of those viewers, almost 50% watched the full video.

Chat transcripts also highlighted the need for clearer messaging for this useful and free resource.

That's why the HubSpot team decided to test how clear value propositions could improve user engagement and delight.

A/B Test Method

HubSpot used three variants for this test, using HubSpot Academy conversion rate (CVR) as the primary metric. Secondary metrics included CTA clicks and engagement.

Variant A was the control.

For variant B, the team added more vibrant images and colorful text and shapes. It also included an animated "typing" headline.

Variant C also added color and movement, as well as animated images on the right-hand side of the page.

As a result, HubSpot found that variant B outperformed the control by 6%. In contrast, variant C underperformed the control by 1%. From those numbers, HubSpot was able to project that using variant B would lead to about 375 more sign ups each month.

2. FSAstore.com’s Site Navigation

Every marketer will have to focus on conversion at some point. But building a website that converts is tough.

FSAstore.com is an ecommerce company supplying home goods for Americans with a flexible spending account.

This useful site could help the 35 million+ customers that have an FSA. But the website funnel was overwhelming. It had too many options, especially on category pages. The team felt that customers weren't making purchases because of that issue.

To figure out how to appeal to its customers, this company tested a simplified version of its website. The current site included an information-packed subheader in the site navigation.

To test the hypothesis, this A/B testing example compared the current site to an update without the subheader.

This update showed a clear boost in conversions and FSAstore.com saw a 53.8% increase in revenue per visitor.

3. Expoze’s Web Page Background

The visuals on your web page are important because they help users decide whether they want to spend more time on your site.

In this A/B testing example, Expoze.io decided to test the background on its homepage.

The website home page was difficult for some users to read because of low contrast. The team also needed to figure out how to improve page navigation while still representing the brand.

First, the team did some research and created several different designs. The goals of the redesign were to improve the visuals and increase attention to specific sections of the home page, like the video thumbnail.

They used AI-generated eye tracking as they designed to find the best designs before A/B testing. Then they ran an A/B heatmap test to see whether the new or current design got the most attention from visitors.

The new design showed a big increase in attention, with version B bringing over 40% more attention to the desired sections of the home page.

This design change also brought a 25% increase in CTA clicks. The team believes this is due to the added contrast on the page bringing more attention to the CTA button, which was not changed.

4. Thrive Themes’ Sales Page Optimization

Many landing pages showcase testimonials. That's valuable content and it can boost conversion.

That's why Thrive Themes decided to test a new feature on its landing pages — customer testimonials .

In the control, Thrive Themes had been using a banner that highlighted product features, but not how customers felt about the product.

The team decided to test whether adding testimonials to a sales landing page could improve conversion rates.

In this A/B test example, the team ran a 6-week test with the control against an updated landing page with testimonials.

This change netted a 13% increase in sales. The control page had a 2.2% conversion rate, but the new variant showed a 2.75% conversion rate.

Email A/B Testing Examples

5. hubspot's email subscriber experience.

Getting users to engage with email isn't an easy task. That's why HubSpot decided to A/B test how alignment impacts CTA clicks.

HubSpot decided to change text alignment in the weekly emails for subscribers to improve the user experience. Ideally, this improved experience would result in a higher click rate.

For the control, HubSpot sent centered email text to users.

For variant B, HubSpot sent emails with left-justified text.

HubSpot found that emails with left-aligned text got fewer clicks than the control. And of the total left-justified emails sent, less than 25% got more clicks than the control.

6. Neurogan’s Deal Promotion

Making the most of email promotion is important for any company, especially those in competitive industries.

This example uses the power of current customers for increasing email engagement.

Neurogan wasn't always offering the right content to its audience and it was having a hard time competing with a flood of other new brands.

An email agency audited this brand's email marketing, then focused efforts on segmentation. This A/B testing example starts with creating product-specific offers. Then, this team used testing to figure out which deals were best for each audience.

These changes brought higher revenue for promotions and higher click rates. It also led to a new workflow with a 37% average open rate and a click rate of 3.85%.

For more on how to run A/B testing for your campaigns, check out this free A/B testing kit .

Social Media A/B Testing Examples

7. vestiaire’s tiktok awareness campaign.

A/B testing examples like the one below can help you think creatively about what to test and when. This is extra helpful if your business is working with influencers and doesn't want to impact their process while working toward business goals.

Fashion brand Vestaire wanted help growing the brand on TikTok. It was also hoping to increase awareness with Gen Z audiences for its new direct shopping feature.

Vestaire's influencer marketing agency asked eight influencers to create content with specific CTAs to meet the brand's goals. Each influencer had extensive creative freedom and created a range of different social media posts.

Then, the agency used A/B testing to choose the best-performing content and promoted this content with paid advertising .

This testing example generated over 4,000 installs. It also decreased the cost per install by 50% compared to the brand's existing presence on Instagram and YouTube.

8. Underoutfit’s Promotion of User-Generated Content on Facebook

Paid advertising is getting more expensive, and clickthrough rates decreased through the end of 2022 .

To make the most of social ad spend, marketers are using A/B testing to improve ad performance. This approach helps them test creative content before launching paid ad campaigns, like in the examples below.

Underoutfit wanted to increase brand awareness on Facebook.

To meet this goal, it decided to try adding branded user-generated content. This brand worked with an agency and several creators to create branded content to drive conversion.

Then, Underoutfit ran split testing between product ads and the same ads combined with the new branded content ads. Both groups in the split test contained key marketing messages and clear CTA copy.

The brand and agency also worked with Meta Creative Shop to make sure the videos met best practice standards.

The test showed impressive results for the branded content variant, including a 47% higher clickthrough rate and 28% higher return on ad spend.

9. Databricks’ Ad Performance on LinkedIn

Pivoting to a new strategy quickly can be difficult for organizations. This A/B testing example shows how you can use split testing to figure out the best new approach to a problem.

Databricks , a cloud software tool, needed to raise awareness for an event that was shifting from in-person to online .

To connect with a large group of new people in a personalized way, the team decided to create a LinkedIn Message Ads campaign. To make sure the messages were effective, it used A/B testing to tweak the subject line and message copy.

The third variant of the copy featured a hyperlink in the first sentence of the invitation. Compared to the other two variants, this version got nearly twice as many clicks and conversions.

Mobile A/B Testing Example

7. hubspot's mobile calls-to-action.

On this blog, you'll notice anchor text in the introduction, a graphic CTA at the bottom, and a slide-in CTA when you scroll through the post. Once you click on one of these offers, you'll land on a content offer page.

While many users access these offers from a desktop or laptop computer, many others plan to download these offers to mobile devices.

But on mobile, users weren't finding the CTA buttons as quickly as they could on a computer. That's why HubSpot tested mobile design changes to improve the user experience.

Previous A/B tests revealed that HubSpot's mobile audience was 27% less likely to click through to download an offer. Also, less than 75% of mobile users were scrolling down far enough to see the CTA button.

So, HubSpot decided to test different versions of the offer page CTA, using conversion rate (CVR) as the primary metric. For secondary metrics, the team measured CTA clicks for each CTA, as well as engagement.

HubSpot used four variants for this test.

For variant A, the control, the traditional placement of CTAs remained unchanged.

For variant B, the team redesigned the hero image and added a sticky CTA bar.

For variant C, the redesigned hero was the only change.

For variant D, the team redesigned the hero image and repositioned the slider.

All variants outperformed the control for the primary metric, CVR. Variant C saw a 10% increase, variant B saw a 9% increase, and variant D saw an 8% increase.

From those numbers, HubSpot was able to project that using variant C on mobile would lead to about 1,400 more content leads and almost 5,700 more form submissions each month.

11. Hospitality.net’s Mobile Booking

Businesses need to keep up with quick shifts in mobile devices to create a consistently strong customer experience.

A/B testing examples like the one below can help your business streamline this process.

Hospitality.net offered both simplified and dynamic mobile booking experiences. The simplified experience showed a limited number of available dates and the design is for smaller screens. The dynamic experience is for the larger mobile device screens. It shows a wider range of dates and prices.

But the brand wasn’t sure which mobile optimization strategy would be better for conversion.

This brand believed that customers would prefer the dynamic experience and that it would get more conversions. But it chose to test these ideas with a simple A/B test. Over 34 days, it sent half of the mobile visitors to the simplified mobile experience, and half to the dynamic experience, with over 100,000 visitors total.

This A/B testing example showed a 33% improvement in conversion. It also helped confirm the brand's educated guesses about mobile booking preferences.

A/B Testing Takeaways for Marketers

A lot of different factors can go into A/B testing, depending on your business needs. But there are a few key things to keep in mind:

• Every A/B test should start with a hypothesis focused on one specific problem that you can test.
• Make sure you’re testing a control variable (your original version) and a treatment variable (a new version that you think will perform better).
• You can test various things, like landing pages, CTAs, emails, or mobile app designs.
• The best way to understand if your results mean something is to figure out the statistical significance of your test.
• There are a variety of goals to focus on for A/B testing (increased site traffic, lower bounce rates, etc.), but you should be able to test, support, prove, and disprove your hypothesis.
• When testing, make sure you’re splitting your sample groups equally and randomly, so your data is viable and not due to chance.
• Take action based on the results you observe.

Start Your Next A/B Test Today

You can see amazing results from the A/B testing examples above. These businesses were able to take action on goals because they started testing. If you want to get great results, you've got to get started, too.

Editor's note: This post was originally published in October 2014 and has been updated for comprehensiveness.

Don't forget to share this post!

Related articles.

How to Do A/B Testing: 15 Steps for the Perfect Split Test

What Most Brands Miss With User Testing (That Costs Them Conversions)

Multivariate Testing: How It Differs From A/B Testing

How to A/B Test Your Pricing (And Why It Might Be a Bad Idea)

15 of the Best A/B Testing Tools for 2024

How to Determine Your A/B Testing Sample Size & Time Frame

These 20 A/B Testing Variables Measure Successful Marketing Campaigns

How to Understand & Calculate Statistical Significance [Example]

What is an A/A Test & Do You Really Need to Use It?

The Ultimate Guide to Social Testing

Learn more about A/B and how to run better tests.

Marketing software that helps you drive revenue, save time and resources, and measure and optimize your investments — all on one easy-to-use platform

• Business Essentials
• Leadership & Management
• Credential of Leadership, Impact, and Management in Business (CLIMB)
• Entrepreneurship & Innovation
• Digital Transformation
• Finance & Accounting
• Business in Society
• For Organizations
• Support Portal
• Media Coverage
• Founding Donors
• Leadership Team

• Harvard Business School →
• HBS Online →
• Business Insights →

Business Insights

Harvard Business School Online's Business Insights Blog provides the career insights you need to achieve your goals and gain confidence in your business skills.

• Career Development
• Communication
• Decision-Making
• Earning Your MBA
• Negotiation
• News & Events
• Productivity
• Staff Spotlight
• Student Profiles
• Work-Life Balance
• AI Essentials for Business
• Alternative Investments
• Business Analytics
• Business Strategy
• Business and Climate Change
• Design Thinking and Innovation
• Digital Marketing Strategy
• Disruptive Strategy
• Economics for Managers
• Entrepreneurship Essentials
• Financial Accounting
• Global Business
• Launching Tech Ventures
• Leadership Principles
• Leadership, Ethics, and Corporate Accountability
• Leading Change and Organizational Renewal
• Leading with Finance
• Management Essentials
• Negotiation Mastery
• Organizational Leadership
• Power and Influence for Positive Impact
• Strategy Execution
• Sustainable Business Strategy
• Sustainable Investing
• Winning with Digital Platforms

What Is A/B Testing & What Is It Used For?

• 15 Dec 2016

You may have experienced meetings where a lot of ideas are circulated about how to improve an existing product or service. In these meetings, differing opinions can quickly turn into a battle of long-winded defenses. Fortunately, the emergence of A/B testing—once thought to be exclusive to tech firms—has become a viable and cost-effective way for all types of businesses to identify and test value-creating ideas.

Related: The Advantages of Data-Driven Decision Making

What Is an A/B test?

In statistical terms, A/B testing is a method of two-sample hypothesis testing. This means comparing the outcomes of two different choices (A and B) by running a controlled mini-experiment. This method is also sometimes referred to as split testing .

What Is A/B Testing Used For?

A/B testing is often discussed in the context of user experience (UX), conversion rate optimization (CRO), and other marketing and technology-focused applications; however, it can be valuable in other situations as well.

Although the concept of A/B testing was galvanized by Silicon Valley giants, the rationale behind A/B testing isn’t new. The practice borrows from traditional randomized control trials to create smaller, more scalable experiments.

For this reason, professionals also perform A/B testing to gather valuable insights and guide important business decisions, such as determining which product features are most important to consumers.

A/B testing is a popular method of experimentation in the fields of digital marketing and web design. For example, a marketer looking to increase e-commerce sales may run an experiment to determine whether the location of the “buy now” button on the product page impacts a particular product’s number of sales.

In this scenario, version A of the product page might feature the button in the top-right corner of the page while version B places the button in the bottom-right corner. With all other variables held constant, randomly selected users interact with the page. Afterward, the marketer can analyze the results to determine which button location resulted in the greatest percentage of sales.

An Example of A/B Testing

As a basic example, let’s say you’re an abstract artist. You’re confident in your technique but aren’t sure how the outside world—and, more importantly, art critics—will respond to your new paintings. Assessing the quality of art is a famously challenging process.

To employ A/B testing for this scenario, start by creating two different paintings that are alike. As you paint both pieces, change one small thing—for instance, add a red square to one painting and not the other. Again, this means that everything about the paintings is alike except for this one modification. Once the change is made, display the two paintings in randomly selected art galleries across the country and wait for your art agent, or another unbiased third party, to gather reactions and report back.

After each painting has been placed in a reasonable amount of art galleries, the feedback may reflect that the painting with the red square received significantly more praise, or maybe it didn’t. The hypothetical outcome doesn’t matter. Rather, what matters is that you can be reasonably confident that your change will or will not make the painting better, and you can go on to create better art as a result.

The randomization aspect of this design is explicitly emphasized because randomization is the gold-standard for eliminating biases. Art is a subjective field and evolves over time. So do the preferences and opinions of customers, clients, or coworkers. A/B testing isn’t a static process, and tests can be repeated or complemented if companies believe that findings may not be valid or applicable anymore.

As a final note, it’s imperative that A/B testing design be rigorous to ensure the validity of results. Furthermore, there may be some decisions where internal opinions are more cost-effective or timely.

Making Data-Driven Business Decisions

Companies like Google, Amazon, and Facebook have all used A/B testing to help create more intuitive web layouts or ad campaigns, and firms in all industries can find value in experimentation. Using data-driven decision-making techniques empowers business leaders to confidently pursue opportunities and address challenges facing their firms. Customers benefit and companies can reap measurable monetary returns by catering to market preferences.

Do you want to learn how to apply fundamental quantitative methods to real business problems? Explore Business Analytics and our other analytics courses to find out how you can use data to inform business decisions.

This post was updated on January 12, 2021. It was originally published on December 15, 2016.

About the Author

Home Articles A/B Test Hypothesis Definition, Tips and Best Practices

A/B Test Hypothesis Definition, Tips and Best Practices

Emily Healy

Incomplete, irrelevant or poorly formulated A/B test hypotheses are at the root of many neutral or negative tests.

Often we imagine that doing A/B tests to improve your e-commerce site’s performance means quickly changing the color of the “add to cart” button will lead to a drastic increase in your conversion rate, for example. However, A/B testing is not always so simple.

Unfortunately, implementing random changes to your pages won’t always significantly improve your results – there should be a reason behind your web experiments .

This brings us to the following question: how do you know which elements to experiment with and how can you create an effective AB test hypothesis?

Determine the problem and the hypothesis

Far too few people question the true origins of the success (or failure) of the changes they put in place to improve their conversion rate.

However, it’s important to know how to determine both the problem and the hypothesis that will allow you to obtain the best results.

Instead of searching for a quick “DIY” solution, it’s often more valuable in the long term to take a step back and do two things:

• Identify the real problem – What is the source of your poor performance? Is it a high bounce rate on your order confirmation page, too many single-page sessions,  a low-performing checkout CTA or something more complex?
• Establish a hypothesis – This could show the root of the problem. For example, a great hypothesis for A/B testing could be: “Our customers do not immediately understand the characteristics of our products when they read the pages on our e-commerce site. Making the information more visible will increase the clicks on the “add-to-cart” button.”

The second step may seem very difficult because it requires a capacity for introspection and a critical look at the existing site. Nevertheless, it’s crucial for anyone who wants to see their KPIs improve drastically .

If you’re feeling a bit uncomfortable with this type of uncertainty around creating an effective hypothesis, know that you’ve come to the right place.

What is an A/B test hypothesis?

Technically speaking, the word hypothesis has a very simple definition:

“A proposal that seeks to provide a plausible explanation of a set of facts and which must be controlled against experience or verified in its consequences.”

The first interesting point to notice in this definition is “the set of facts to be explained.” In A/B testing, a hypothesis must always start with a clearly identified problem .

A/B tests should not be done randomly, or you risk wasting time.

Let’s talk about how to identify the problem:

• Web analytics data – While this data does not explain digital consumers’ behavior exactly, it can highlight conversion problems (identifying abandoned carts , for example) and help prioritize the pages in need of testing.
• Heuristic evaluation and ergonomic audit – These analyses allow you to assess the site’s user experience at a lower cost using an analysis grid.
• User tests – This qualitative data is limited by the sample size but can be very rich in information that would not have been detected with quantitative methods. They often reveal problems understanding the site’s ergonomics. Even if the experience can be painful given the potential for negative remarks, it will allow you to gather qualified data with precise insights.
• Eye tracking or heatmaps – These methods provide visibility into how people interact with items within a page – not between pages.
• Customer feedback – As well as analyzing feedback, you can implement tools such as customer surveys or live chats to collect more information.

The tactics above will help you highlight the real problems that impact your site’s performance and save you time and money in the long run.

A/B test hypothesis formula

Initially, making an A/B test hypothesis may seem too simple. At the start, you mainly focus on one change and the effect it produces. You should always respect the following format: If I change this, it will cause that effect . For example:

Changing (the element being tested) from ___________ to ___________ will increase/decrease (the defined measurement).

At this stage, this formula is only a theoretical assumption that will need to be proven or disproven, but it will guide you in solving the problem.

An important point, however, is that the impact of the change you want to bring must always be measurable in quantifiable terms ( conversion rate , bounce rate , abandonment rate , etc.).

Here are two examples of hypotheses phrased according to the formula explained above and that can apply to e-commerce:

• Changing our CTA from “BUY YOUR TICKETS NOW” to “TICKETS ARE SELLING FAST – ONLY 50 LEFT!” will improve our sales on our e-commerce site.
• Shortening the sign-up form by deleting optional fields such as phone and mailing address will increase the number of contacts collected.

In addition, when you think about the solution you want to implement, include the psychology of the prospect by asking yourself the following:

What psychological impact could the problem cause in the digital consumer’s mind?

For example, if your problem is a lack of clarity in the registration process which impacts the purchases, then the psychological impact could be that your prospect is confused when reading information.

With this in mind, you can begin to think concretely about the solution to correct this feeling on the client side. In this case, we can imagine that one fix could be including a progress bar that shows the different stages of registration.

Be aware : the psychological aspect should not be included when formulating your test hypothesis.

Once you have gotten the results, you should then be able to say whether it is true or false. Therefore, we can only rely on concrete and tangible assumptions.

Best practice for e-commerce optimization based on A/B hypotheses

There are many testable elements on your website. Looking into these elements and their metrics can help you create an effective test hypothesis.

We are going to give you some concrete examples of common areas to test to inspire you on your optimization journey:

• The header/main banner explaining the products/services that your site offers can increase customers’ curiosity and extend their time on the site.
• A visible call-to-action appearing upon arrival will increase the chance visitors will click.
• A very visible “about” section will build prospects’ trust in the brand when they arrive on the site.

PRODUCT SECTIONS

• Filters save customers a lot of time by quickly showing them what they are looking for.
• Highlighting a selection of the most popular products at the top of the sections is an excellent starting point for generating sales.
• A “find out more” button or link under each product will encourage users to investigate.

PRODUCT PAGES

• Product recommendations create a more personal experience for the user and help increase their average shopping cart
• A visible “add to cart” button will catch the prospect’s attention and increase the click rate.
• An “add to cart and pay” button saves the customer time, as many customers have an average of one transaction at a time.
• Adding social sharing buttons is an effective way of turning the product listing into viral content.

Want to start A/B testing elements on your website? AB Tasty is the best-in-class experience optimization platform to help you convert more customers by leveraging intelligent search and recommendations to create a richer digital experience – fast. From experimentation to personalization, this solution can help you achieve the perfect digital experience with ease.

• The presence of logos such as “Visa certified” enhances customer confidence in the site.
• A very visible button/link to “proceed to payment” greatly encourages users to click.
• A single page for payment reduces the exit rate.
• Paying for an order without registration is very much appreciated by new prospects, who are not necessarily inclined to share their personal information when first visiting the site.
• Having visibility over the entire payment process reassures consumers and will nudge them to finalize their purchase.

These best practices allow you to build your A/B test hypotheses by comparing your current site with the suggestions above and see what directly impacts conversion performance.

The goal of creating an A/B test hypothesis

The end goal of creating an A/B test hypothesis is to identify quickly what will help guarantee you the best results. Whether you have a “winning” hypothesis or not, it will still serve as a learning experience.

While defining your hypotheses can seem complex and methodical, it’s one of the most important ways for you to understand your pages’ performance and analyze the potential benefits of change.

You might also like...

What is one-to-one personalization in marketing (with 8 examples).

Jun 24, 2024

How To Build A Customer Journey Map

Chloe Annas

Jun 18, 2024

Customization vs Personalization: What’s the Difference?

Robin Nichols

Jun 10, 2024

Subscribe to our Newsletter

We will process and store your personal data to respond to send you communications as described in our  Privacy Policy .

• Data Science

Caltech Bootcamp / Blog / /

What is A/B Testing in Data Science?

• Written by John Terra
• Updated on May 22, 2024

Statistics help digital marketers understand how successful their ad campaigns, marketing events, and websites are. Testing yields those necessary statistics, and many different forms of testing are available. Today, we’re answering the question, “What is A/B testing?”

This article focuses on A/B testing in data science, including defining the term, explaining its importance, showing how it works and how to conduct it, when to use it, and other valuable tidbits. We’ll round things out by discussing the common mistakes associated with A/B testing, real-world applications, what tools data scientists use to conduct A/B testing, and a data science bootcamp professionals can take to boost their careers.

Let’s get the ball rolling with a definition. What is A/B testing in data science?

A/B testing, also known as “split testing,” is a method employed extensively in data science. It allows data scientists to generate accurate, evidence-based decisions using the insights gained from testing two different variables. A/B testing is an experiment on two variants using a given metric to see which performs better.

A/B testing divides traffic into two groups and serves one group, the A/B version, and the other, the control. It helps data scientists determine what works and doesn’t work for the organization and enables them to evaluate the impact of different versions on conversion and response rates.

Also Read: What is Exploratory Data Analysis? Types, Tools, Importance, etc.

Why A/B Testing is Important

A/B testing in data science is critical to data-driven decision-making because it helps data scientists eliminate guesswork by comparing two versions of a marketing campaign, web page, or product feature to see which performs better.

Additionally, A/B testing in data science helps marketers better understand user behavior, which is vital in user experience (UX) design, conversion rate optimization, and similar fields.

A data scientist or marketing professional who runs an A/B test can isolate the variables directly affecting the outcome. This process lets data scientists identify whether the changes made had a positive, negative, or null impact on user behavior. The insights gleaned from A/B testing can then be used to make better, more informed decisions and optimize the various aspects of a service or product.

A/B testing is a valuable part of data science and marketing efforts because our world increasingly relies on the ever-growing volumes of data generated daily. Numbers must back up business decisions, and A/B testing helps fill that gap.

When to Use A/B Testing

Since every form of testing has strong points and places where it does the most good, when should we use A/B testing? A/B testing excels in situations like testing incremental changes. Incremental changes include UX adjustments, new features, page load times, and ranking. Here, researchers can compare outcomes before and after the modifications to ascertain whether the changes have the desired effect.

On the other hand, A/B testing only functions effectively when used to test significant changes, such as new branding, new products, or a whole new user experience.

Now, let’s look at how to perform an A/B test.

How to Conduct an A/B Test

There are three stages in conducting A/B tests.

Generate Your Hypothesis

Before running your tests, you must generate your hypothesis. A hypothesis is an unproven assumption about how the natural world functions. Alternatively, it’s a reasonable prediction about something in the immediate environment that can be verifiable via observation or experimentation. You must generate a null hypothesis and an alternative hypothesis.

• Null hypothesis. A null hypothesis declares that sample observations result completely from chance. In the context of an A/B test, the null hypothesis states that there is no difference between the control and variant groups.
• Alternative hypothesis. The alternative hypothesis states that a non-random cause influences sample observations. In the context of an A/B test, the alternative hypothesis says there’s a difference between the control and variant groups.

Regardless of the hypothesis, you should follow the PICOT rules when formulating it.

• Population. This is the group of people participating in the experiment.
• Intervention. This is the new variant in the study.
• Comparison. This refers to what reference group you are using to compare against your intervention.
• Outcome. The outcome signifies what result you plan on measuring.
• Time. Time refers to the duration of the experience, including when and for how long the data will be collected.

Create the Control and Test Groups

Once you have developed your hypotheses, you need to create your control and test (variant) groups. In this step, remember these two vital concepts: random sampling and sample size.

• Random Sampling. In random sampling, each sample in the population has an equal chance of getting selected. Random sampling is crucial in hypothesis testing because it removes sampling bias; it’s essential to eliminate bias because the A/B test results must represent the whole population rather than the sample itself.
• Sample Size. Before conducting the test, determining the minimum sample size for the A/B test is essential. This way, you eliminate under-coverage bias or bias from sampling too few observations.

Run the A/B Tests and Gather the Results, Either Rejecting or Keeping the Null Hypothesis

After you conduct the experiment and collect the data, determine if the difference between the control and variant groups is statistically significant. How do you do this? By following these three simple steps:

• Set your alpha, which is the probability of making a type 1 error. In most cases, the alpha is set at 5% or 0.05.
• Determine the probability value (p-value). Start by calculating the t-statistic using the formula below, provided by isixsigma.com.
• Finally, compare the p-value to the alpha. Don’t reject the null if the p-value is greater than the alpha.

Alternately, some sources posit that there are five stages associated with A/B tests:

• Run the experiment
• Measure the results
• Determine the conversion to improve
• Hypothesize changes
• Identify the variables and create variations

Also Read: What is Data Wrangling? Importance, Tools, and More

The Common Mistakes to Avoid in A/B Testing

There are a few significant mistakes that data science experts risk committing. They are:

• Invalid Hypothesis. The entire experiment is predicated solely on the hypothesis. What needs to be changed, what justifies these changes, and what are the desired results? The chance of the test succeeding diminishes if you start with an incorrect hypothesis.
• Testing too many components simultaneously. Try to run as few tests as possible at once. Running too many tests simultaneously might be challenging to discern which aspect contributed to success or failure. Therefore, it’s vital to prioritize tests for effective A/B testing.
• Ignoring Statistical Significance. Your opinion of the test doesn’t matter. Let the test run its full course, whether it succeeds or fails, so that it acquires statistical significance.
• Not taking external factors into account. Tests should be run during comparable times so that you may obtain significant findings. For example, comparing website hits on high-traffic days to days with the lowest traffic because of external factors such as sales or holidays is unfair and will yield a flawed conclusion.

Real-World Applications of A/B Testing

So, how does A/B testing work in the real world? Check out this pair of examples and see how A/B testing in data science contributes to the digital economy.

User Experience Design

A/B testing is used in user experience (UX) design to identify obstacles that prevent customers from optimally interacting with a website, service, or product. It helps UX designers determine what adjustments are required on the website or application to give consumers a seamless and delightful user experience. For example, UX designers could run an A/B test for two different shopping cart/checkout process versions on an e-commerce site and see which one results in a more complete, effortless purchase. So, A/B testing lets designers make better data-driven design decisions.

Marketing Analytics

A/B testing is widely used in marketing analytics to optimize marketing efforts. It lets marketers test different versions of their campaigns and messages and discern which resonates the most with their prospective customers.

Only after conducting extensive A/B testing can marketers accurately decide which changes are worth the effort.

From landing page designs to e-mail marketing campaigns, A/B testing plays a significant part in today’s digital marketing strategies. A/B testing minimizes risk and increases the chances of a successful marketing campaign.

Tools Used for A/B Testing in Data Science

A/B testing in data science has many tools to make the job easier. Here’s a list of 13 popular A/B testing tools. Choosing the ideal A/B testing tool largely depends on your unique needs. When you’re ready to shop for A/B testing tools, consider crucial factors such as pricing, ease of use, and analysis level.

Ensure the tool you choose supports your marketing goals, including conversion rate optimization, boosting user engagement, or even reducing churn rate. Picking the right testing tool will play a significant role in conducting a successful A/B test and leveraging data for organizational success.

• Adobe Target
• Dynamic Yield
• Google Optimize
• LaunchDarkly
• Omniconvert
• Oracle Maxymiser

Also Read: What is Spatial Data Science? Definition, Applications, Careers & More

Do You Want Data Science Training?

Data science is gaining greater prominence in digital marketing thanks to the profusion of data available online. Consequently, data science is a great career option, offering generous compensation, job security, and growth opportunities. If you’re interested in gaining job-ready skills in data science, either to round out your skill set or as part of a career change, consider this online Post Graduate Program in Data Science .

This 44-week online bootcamp imparts data science and generative AI skills through a high-engagement learning experience. You will learn about descriptive statistics, ensemble learning, conversational AI, data visualization, and more. In addition, you will gain exposure to generative AI tools like ChatGPT, DALL-E, Midjourney, etc.

Indeed.com reports that data scientists can earn an annual average salary of $124,393. Sign up today and gain that valuable set of skills and training that could take you far in a new career in data science.

You might also like to read:

Data Science and Marketing: Transforming Strategies and Enhancing Engagement

An Introduction to Natural Language Processing in Data Science

Why Use Python for Data Science?

A Beginner’s Guide to the Data Science Process

What Is Data Mining? A Beginner’s Guide

Data Science Bootcamp

• Learning Format:

Online Bootcamp

Leave a comment cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Recommended Articles

Big Data and Analytics: Unlocking the Future

Unlock the potential and benefits of big data and analytics in your career. Explore essential roles and discover the advantages of data-driven decision-making.

Five Outstanding Data Visualization Examples for Marketing

This article gives excellent data visualization examples in marketing, including defining data visualization and its advantages.

Data Science Bootcamps vs. Traditional Degrees: Which Learning Path to Choose?

Need help deciding whether to choose a data science bootcamp or a traditional degree? Our blog breaks down the pros and cons of each to help you make an informed decision.

Career Roundup: Data Scientist vs. Machine Learning Engineer

This article compares data scientists and machine learning engineers, contrasting their roles, responsibilities, functions, needed skills, and salaries.

What is Natural Language Generation in Data Science, and Why Does It Matter?

This article explores natural language generation in data science, including its definition, importance, stages, and best practices.

What is Exploratory Data Analysis? Types, Tools, Importance, etc.

This article highlights exploratory data analysis, including its definition, role in data science, types, and overall importance.

Learning Format

Program Benefits

• 12+ tools covered, 25+ hands-on projects
• Masterclasses by distinguished Caltech CTME instructors
• Caltech CTME Circle Membership
• Industry-specific training from global experts
• Call us on : 1800-212-7688

• Subscribers
• How To Use a New AI App and AI Agents To Build Your Best Landing Page
• The MECLABS AI Guild in Action: Teamwork in Crafting Their Optimal Landing Page
• How MECLABS AI Is Being Used To Build the AI Guild
• MECLABS AI’s Problem Solver in Action
• MECLABS AI: Harness AI With the Power of Your Voice
• Harnessing MECLABS AI: Transform Your Copywriting and Landing Pages
• MECLABS AI: Overcome the ‘Almost Trap’ and Get Real Answers
• MECLABS AI: A brief glimpse into what is coming!
• Transforming Marketing with MECLABS AI: A New Paradigm
• Creative AI Marketing: Escaping the ‘Vending Machine Mentality’

A/B Testing: Example of a good hypothesis

Want to know the secret to always running successful tests?

The answer is to formulate a hypothesis .

Now when I say it’s always successful, I’m not talking about always increasing your Key Performance Indicator (KPI). You can “lose” a test, but still be successful.

That sounds like an oxymoron, but it’s not. If you set up your test strategically, even if the test decreases your KPI, you gain a learning , which is a success! And, if you win, you simultaneously achieve a lift and a learning. Double win!

The way you ensure you have a strategic test that will produce a learning is by centering it around a strong hypothesis.

So, what is a hypothesis?

By definition, a hypothesis is a proposed statement made on the basis of limited evidence that can be proved or disproved and is used as a starting point for further investigation.

Let’s break that down:

It is a proposed statement.

• A hypothesis is not fact, and should not be argued as right or wrong until it is tested and proven one way or the other.

It is made on the basis of limited (but hopefully some ) evidence.

• Your hypothesis should be informed by as much knowledge as you have. This should include data that you have gathered, any research you have done, and the analysis of the current problems you have performed.

It can be proved or disproved.

• A hypothesis pretty much says, “I think by making this change , it will cause this effect .” So, based on your results, you should be able to say “this is true” or “this is false.”

It is used as a starting point for further investigation.

• The key word here is starting point . Your hypothesis should be formed and agreed upon before you make any wireframes or designs as it is what guides the design of your test. It helps you focus on what elements to change, how to change them, and which to leave alone.

How do I write a hypothesis?

The structure of your basic hypothesis follows a CHANGE: EFFECT framework.

While this is a truly scientific and testable template, it is very open-ended. Even though this hypothesis, “Changing an English headline into a Spanish headline will increase clickthrough rate,” is perfectly valid and testable, if your visitors are English-speaking, it probably doesn’t make much sense.

So now the question is …

How do I write a GOOD hypothesis?

To quote my boss Tony Doty , “This isn’t Mad Libs.”

We can’t just start plugging in nouns and verbs and conclude that we have a good hypothesis. Your hypothesis needs to be backed by a strategy. And, your strategy needs to be rooted in a solution to a problem .

So, a more complete version of the above template would be something like this:

In order to have a good hypothesis, you don’t necessarily have to follow this exact sentence structure, as long as it is centered around three main things:

Presumed problem

Proposed solution

Anticipated result

After you’ve completed your analysis and research, identify the problem that you will address. While we need to be very clear about what we think the problem is, you should leave it out of the hypothesis since it is harder to prove or disprove. You may want to come up with both a problem statement and a hypothesis .

For example:

Problem Statement: “The lead generation form is too long, causing unnecessary friction .”

Hypothesis: “By changing the amount of form fields from 20 to 10, we will increase number of leads.”

When you are thinking about the solution you want to implement, you need to think about the psychology of the customer. What psychological impact is your proposed problem causing in the mind of the customer?

For example, if your proposed problem is “There is a lack of clarity in the sign-up process,” the psychological impact may be that the user is confused.

Now think about what solution is going to address the problem in the customer’s mind. If they are confused, we need to explain something better, or provide them with more information. For this example, we will say our proposed solution is to “Add a progress bar to the sign-up process.”  This leads straight into the anticipated result.

If we reduce the confusion in the visitor’s mind (psychological impact) by adding the progress bar, what do we foresee to be the result? We are anticipating that it would be more people completing the sign-up process. Your proposed solution and your KPI need to be directly correlated.

Note: Some people will include the psychological impact in their hypothesis. This isn’t necessarily wrong, but we do have to be careful with assumptions. If we say that the effect will be “Reduced confusion and therefore increase in conversion rate,” we are assuming the reduced confusion is what made the impact. While this may be correct, it is not measureable and it is hard to prove or disprove.

To summarize, your hypothesis should follow a structure of: “If I change this, it will have this effect,” but should always be informed by an analysis of the problems and rooted in the solution you deemed appropriate.

Related Resources:

A/B Testing 101: How to get real results from optimization

The True Value of Data

15 Years of Marketing Research in 11 Minutes

Marketing Analytics: 6 simple steps for interpreting your data

Website A/B Testing: 4 tips to beat an unbeatable landing page

Online Cart: 6 ideas to test and optimize your checkout process

B2B Gamification: Autodesk’s two approaches to in-trial marketing [Video]

How to Discover Exactly What the Customer Wants to See on the Next Click: 3 critical…

The 21 Psychological Elements that Power Effective Web Design (Part 3)

The 21 Psychological Elements that Power Effective Web Design (Part 2)

The 21 Psychological Elements that Power Effective Web Design (Part 1)

Thanks for the article. I’ve been trying to wrap my head around this type of testing because I’d like to use it to see the effectiveness on some ads. This article really helped. Thanks Again!

Hey Lauren, I am just getting to the point that I have something to perform A-B testing on. This post led me to this site which will and already has become a help in what to test and how to test .

Again, thanks for getting me here .

Good article. I have been researching different approaches to writing testing hypotheses and this has been a help. The only thing I would add is that it can be useful to capture the insight/justification within the hypothesis statement. IF i do this, THEN I expect this result BECAUSE I have this insight.

@Kaya Great!

Good article – but technically you can never prove an hypothesis, according to the principle of falsification (Popper), only fail to disprove the null hypothesis.

Leave A Reply Cancel Reply

Your email address will not be published.

Save my name, email, and website in this browser for the next time I comment.

• Quick Win Clinics
• Research Briefs
• A/B Testing
• Conversion Marketing
• Copywriting
• Digital Advertising
• Digital Analytics
• Digital Subscriptions
• E-commerce Marketing
• Email Marketing
• Lead Generation
• Social Marketing
• Value Proposition
• Research Services
• Video – Transparent Marketing
• Video – 15 years of marketing research in 11 minutes
• Lecture – The Web as a Living Laboratory
• Featured Research

Welcome, Login to your account.

Recover your password.

A password will be e-mailed to you.

• Free Resources

A/B Testing in Digital Marketing: Example of four-step hypothesis framework

by Daniel Burstein , Senior Director, Content & Marketing, MarketingSherpa and MECLABS Institute

This article was originally published in the MarketingSherpa email newsletter .

If you are a marketing expert — whether in a brand’s marketing department or at an advertising agency — you may feel the need to be absolutely sure in an unsure world.

What should the headline be? What images should we use? Is this strategy correct? Will customers value this promo?

This is the stuff you’re paid to know. So you may feel like you must boldly proclaim your confident opinion.

But you can’t predict the future with 100% accuracy. You can’t know with absolute certainty how humans will behave. And let’s face it, even as marketing experts we’re occasionally wrong.

It’s not bad, it’s healthy. And the most effective way to overcome that doubt is by testing our marketing creative to see what really works.

Developing a hypothesis

After we published Value Sequencing: A step-by-step examination of a landing page that generated 638% more conversions , a MarketingSherpa reader emailed us and asked …

Great stuff Daniel. Much appreciated. I can see you addressing all the issues there.

I thought I saw one more opportunity to expand on what you made. Would you consider adding the IF, BY, WILL, BECAUSE to the control/treatment sections so we can see what psychology you were addressing so we know how to create the hypothesis to learn from what the customer is currently doing and why and then form a test to address that? The video today on customer theory was great (Editor’s Note: Part of the MarketingExperiments YouTube Live series ) . I think there is a way to incorporate that customer theory thinking into this article to take it even further.

Developing a hypothesis is an essential part of marketing experimentation. Qualitative-based research should inform hypotheses that you test with real-world behavior.

The hypotheses help you discover how accurate those insights from qualitative research are. If you engage in hypothesis-driven testing, then you ensure your tests are strategic (not just based on a random idea) and built in a way that enables you to learn more and more about the customer with each test.

And that methodology will ultimately lead to greater and greater lifts over time, instead of a scattershot approach where sometimes you get a lift and sometimes you don’t, but you never really know why.

Here is a handy tool to help you in developing hypotheses — the MECLABS Four-Step Hypothesis Framework.

As the reader suggests, I will use the landing page test referenced in the previous article as an example. ( Please note: While the experiment in that article was created with a hypothesis-driven approach, this specific four-step framework is fairly new and was not in common use by the MECLABS team at that time, so I have created this specific example after the test was developed based on what I see in the test).

Here is what the hypothesis would look like for that test, and then we’ll break down each part individually:

If we emphasize the process-level value by adding headlines, images and body copy, we will generate more leads because the value of a longer landing page in reducing the anxiety of calling a TeleAgent outweighs the additional friction of a longer page.

IF: Summary description

The hypothesis begins with an overall statement about what you are trying to do in the experiment. In this case, the experiment is trying to emphasize the process-level value proposition (one of the four essential levels of value proposition ) of having a phone call with a TeleAgent.

The control landing page was emphasizing the primary value proposition of the brand itself.

The treatment landing page is essentially trying to answer this value proposition question: If I am your ideal customer, why should I call a TeleAgent rather than take any other action to learn more about my Medicare options?

The control landing page was asking a much bigger question that customers weren’t ready to say “yes” to yet, and it was overlooking the anxiety inherent in getting on a phone call with someone who might try to sell you something: If I am your ideal customer, why should I buy from your company instead of any other company.

This step answers WHAT you are trying to do.

BY: Remove, add, change

The next step answers HOW you are going to do it.

As Flint McGlaughlin, CEO and Managing Director of MECLABS Institute teaches, there are only three ways to improve performance: removing, adding or changing .

In this case, the team focused mostly on adding — adding headlines, images and body copy that highlighted the TeleAgents as trusted advisors.

“Adding” can be counterintuitive for many marketers. The team’s original landing page was short. Conventional wisdom says customers won’t read long landing pages. When I’m presenting to a group of marketers, I’ll put a short and long landing page on a slide and ask which page they think achieved better results.

Invariably I will hear, “Oh, the shorter page. I would never read something that long.”

That first-person statement is a mistake. Your marketing creative should not be based on “I” — the marketer. It should be based on “they” — the customer.

Most importantly, you need to focus on the customer at a specific point in time — when he or she is in the mindspace of considering to take an action like purchase a product or in need of more information before they decide to download a whitepaper. And sometimes in these situations, longer landing pages perform better.

In the case of this landing page, even the customer may not necessarily favor a long landing page all the time. But in the real-world situation when they are considering whether to call a TeleAgent or not, the added value helps more customers decide to take the action.

WILL: Improve performance

This is your KPI (key performance indicator). This step answers another HOW question: How do you know your hypothesis has been supported or refuted?

You can choose secondary metrics to monitor during your test as well. This might help you interpret the customer behavior observed in the test.

But ultimately, the hypothesis should rest on a single metric.

For this test, the goal was to generate more leads. And the treatment did — 638% more leads.

BECAUSE: Customer insight

This last step answers a WHY question — why did the customers act this way?

This helps you determine what you can learn about customers based on the actions observed in the experiment.

This is ultimately why you test. To learn about the customer and continually refine your company’s customer theory .

In this case, the team theorized that the value of a longer landing page in reducing the anxiety of calling a TeleAgent outweighs the additional friction of a longer landing page.

And the test results support that hypothesis.

Related Resources

The Hypothesis and the Modern-Day Marketer

Boost your Conversion Rate with a MECLABS Quick Win Intensive

Designing Hypotheses that Win: A four-step framework for gaining customer wisdom and generating marketing results

Improve Your Marketing

Join our thousands of weekly case study readers.

Enter your email below to receive MarketingSherpa news, updates, and promotions:

Note: Already a subscriber? Want to add a subscription? Click Here to Manage Subscriptions

Get Better Business Results With a Skillfully Applied Customer-first Marketing Strategy

The customer-first approach of MarketingSherpa’s agency services can help you build the most effective strategy to serve customers and improve results, and then implement it across every customer touchpoint.

Get headlines, value prop, competitive analysis, and more.

Marketer Vs Machine

Marketer Vs Machine: We need to train the marketer to train the machine.

Free Marketing Course

Become a Marketer-Philosopher: Create and optimize high-converting webpages (with this free online marketing course)

Project and Ideas Pitch Template

A free template to help you win approval for your proposed projects and campaigns

Six Quick CTA checklists

These CTA checklists are specifically designed for your team — something practical to hold up against your CTAs to help the time-pressed marketer quickly consider the customer psychology of your “asks” and how you can improve them.

Infographic: How to Create a Model of Your Customer’s Mind

You need a repeatable methodology focused on building your organization’s customer wisdom throughout your campaigns and websites. This infographic can get you started.

Infographic: 21 Psychological Elements that Power Effective Web Design

To build an effective page from scratch, you need to begin with the psychology of your customer. This infographic can get you started.

Receive the latest case studies and data on email, lead gen, and social media along with MarketingSherpa updates and promotions.

• Your Email Account
• Customer Service Q&A
• Search Library
• Content Directory:

Questions? Contact Customer Service at [email protected]

© 2000-2024 MarketingSherpa LLC, ISSN 1559-5137 Editorial HQ: MarketingSherpa LLC, PO Box 50032, Jacksonville Beach, FL 32240

The views and opinions expressed in the articles of this website are strictly those of the author and do not necessarily reflect in any way the views of MarketingSherpa, its affiliates, or its employees.

How to formulate a smart A/B test hypothesis (and why they’re crucial)

By Josh Gallant and Michael Aagaard

Updated on June 27, 2024

A/B testing is a no-brainer for figuring out how you can improve your conversion rate.

And the more targeted and strategic an A/B test is, the more likely it’ll be to have a positive impact on conversions.

A solid test hypothesis goes a long way towards keeping you on the right track and ensuring that you’re conducting valuable marketing experiments that lead to performance lifts as well as learnings…

But how do you come up with a smart A/B test hypothesis?

TABLE OF CONTENTS

What is a hypothesis in a/b testing, what are the key components of an a/b testing hypothesis, how to generate a hypothesis for multiple tests, a/b testing hypothesis examples, 6 best practices for making a winning hypothesis.

Josh Gallant

Josh is the founder of Backstage SEO , an organic growth firm that helps SaaS companies capture demand. He’s a self-proclaimed spreadsheet nerd by day, volunteer soccer coach on weekends, and wannabe fantasy football expert every fall.

» More blog posts by Josh Gallant

Michael Aagaard

Michael Aagaard has been working full time with CRO since 2008. He has helped companies all over the world, including Unbounce, improve their online businesses. Michael’s approach to CRO revolves around research and consumer psychology. He is a sought-after international keynote speaker and is generally known as one of the most passionate and enthusiastic people in the industry. Get in touch with Michael via http://michaelaagaard.com .

» More blog posts by Michael Aagaard

In this article, you’ll learn:

• What makes a good A/B testing hypothesis
• The components of an A/B testing hypothesis
• Best practices for better hypotheses
• How you can start testing with confidence

Ready? Let’s dig in.

An A/B testing hypothesis is a clear, testable statement predicting how changes to a landing page or element will impact user behavior. It guides the experiment by defining what you’re testing and the expected outcome, helping determine if the changes improve metrics like conversions or engagement. Zooming out a bit, the actual dictionary definition of a hypothesis is “a supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation.” Wordy… Put simply, it’s an educated guess used as a starting point to learn more about a given subject.  In the context of landing page and conversion rate optimization, a test hypothesis is an assumption you base your optimized test variant on. It covers what you want to change on the landing page and what impacts you expect to see from that change. Running an A/B test lets you examine to what extent your assumptions were correct, see whether they had the expected impact, and ultimately get more insight into the behavior of your target audience. Formulating a hypothesis will help you challenge your assumptions and evaluate how likely they are to affect the decisions and actions of your prospects. In the long run this can save you a lot of time and money and help you achieve better results.

Now that you’ve got a clear definition of what an A/B testing hypothesis is and why it matters, let’s look at the ins and outs of putting one together for yourself. In general, your hypothesis will include three key components:

• A problem statement
• A proposed solution

The anticipated results

Let’s quickly explore what each of these components involves.

The problem statement

Your first task: Ask yourself why you’re running a test. 

In order to formulate a test hypothesis, you need to know what your conversion goal is and what problem you want to solve by running the test—basically, the “why” of your test.

So before you start working on your test hypothesis, you first have to do two things:

• Determine your conversion goal
• Identify a problem and formulate a problem statement

Once you know what your goal is and what you presume is keeping your visitors from realizing that goal, you can move on to the next step of creating your test hypothesis.

The proposed solution

Your proposed solution is the bread and butter of your hypothesis. This is the “how” portion of your test, and will outline the steps you take to run and achieve your test. 

Basically, if you’re looking at an A/B test, start thinking of the elements you want to test and how those will support you reaching your goal. 

This is the final component of a good A/B test hypothesis and is your educated guess at what results you anticipate your test delivering. 

Basically, you’re trying to predict what the test will achieve—but it’s important to remember you don’t need to worry about accuracy here. Obviously, you don’t want your results to be a total surprise, but you’re just trying to outline a ballpark idea of what the test will achieve. 

Putting these three components together is how you develop a strong hypothesis, outlining the why, how, and what that will shape your test. 

Generating a hypothesis for multiple A/B tests, or even multivariate tests, follows the same logic and process as a single test. You identify a problem, propose a solution, and predict your results. 

The challenge in this instance, though, is that it can be really tempting to create a hypothesis and try to apply it to multiple different variables on a landing page. If you’re testing multiple things at once, it’s harder to determine what’s causing the results.

But that doesn’t mean you don’t have options. 

Multivariate testing allows you to compare multiple variables across multiple pages (not just the single variable over two different pages you get with an A/B test). These tests are a bit more complicated than an A/B test, but they can still deliver incredible insights. 

Generating a hypothesis for a multivariate test follows the same principles as it always does, but you need to pay closer attention to how altering multiple elements on one or more pages will impact your results . 

Make sure to track those variables closely to ensure you can measure effectiveness. 

Let’s look at a real-world example of how to put it all together.

Let’s say there’s a free ebook that you’re offering to the readers on your blog:

Image courtesy ContentVerve

Data from surveys and customer interviews suggests you have a busy target audience who don’t have a lot of spare time to read ebooks, and the time it takes to read the thing could be a barrier that keeps them from it.

With this in mind, your conversion goal and problem statement would look something like this: 

• Conversion goal : ebook downloads
• Problem statement : “My target audience is busy, and the time it takes to read the ebook is a barrier that keeps them from downloading it.”

With both the conversion goal and problem statement in place, it’s time to move on to forming a hypothesis on how to solve the issue set forth in the problem statement.

The ebook can be read in just 25 minutes, so maybe you can motivate more visitors to download the book by calling out that it’s a quick read—these are your expected results . 

Now say you’ve got data from eye-tracking software that suggests the first words in the first bullet point on the page attract more immediate attention from visitors. This information might lead you to hypothesize that the first bullet is thus the best place to address that time issue. 

If you put the proposed solution and expected results together, you get the following A/B test hypothesis: 

“By tweaking the copy in the first bullet point to directly address the ‘time issue’, I can motivate more visitors to download the ebook.”

Great! Now, it’s time to work on putting your hypothesis to the test. To do this, you would work on the actual treatment copy for the bullet point. 

Image courtesy ContentVerse

Remember: until you test your hypotheses, they will never be more than hypotheses. You need reliable data to prove or disprove the validity of your hypotheses. To find out whether your hypothesis will hold water, you can set up an A/B test with the bullet copy as the only variable.

Working with test hypotheses provides you with a much more solid optimization framework than simply running with guesses and ideas that come about on a whim.

But remember that a solid test hypothesis is an informed solution to a real problem—not an arbitrary guess. The more research and data you have to base your hypothesis on, the better it will be.

To that end, let’s look at six best practices you can follow for making a winning A/B test hypothesis:

1. Be clear and specific 

The more specific you can get with your A/B test hypothesis, the stronger it will be. 

When you’re developing your hypothesis, avoid any vague statements. Make sure your hypothesis clearly states the change you’re making and the outcome you expect to see. 

As an example, a statement like “changing the button color will increase clicks” is broad and vague. Instead, try specifying the color you’re changing your CTA button to and creating a clearer estimate on how much you think clicks will increase by. 

Your amended statement could look something like: “Changing the CTA button to blue and updating the copy will increase the amount of clicks by 20%.” 

2. Focus on a single variable  

A/B testing is all about comparing the performance of a single variable across two versions. Introducing more variables changes the nature of the test and thus the nature of your hypothesis.

Don’t muddy the waters by trying to do too much in one test. This will make it much harder to spot what change led to any observed effect, let alone whether it confirmed your hypothesis. 

Remember, you can always continue to test additional variables—pick one to start and focus on getting results there before you dig in elsewhere. 

If you’re not sure where to start, we put together a list of 18 elements on a landing page you can test to help you get started. 

3. Base it on data and quantifiable insights 

Like we said above, the more data you have, the better . Any hypothesis you develop should be based on quantifiable data and insights to get the most out of your test.

Google Analytics, customer interviews, surveys, heat maps and user testing are just a handful of examples of valuable data sources that will help you gain insight on your target audience , how they interact with your landing page, and what makes them tick.

Look at past data, user feedback, and market research to inform your hypothesis. This ensures that your experiment is grounded in reality and increases the likelihood of meaningful results. 

As an example, if previous data suggests that users prefer blue buttons, your hypothesis might be that changing the button color from green to blue will increase engagement.

Not sure what data you should be focusing on? Here are a few metrics and KPIs to keep in mind: 

• Bounce rate
• Conversion rate
• Form abandonment rate
• Time on page
• Page load time
• Click-through rate

4. Make sure it’s testable

You need to be able to measure the results of your test. Defining specific metrics and KPIs—such as click-through rates and especially conversion rate—is a great way to make sure what you’re doing is testable and measurable. 

If the best practices you’ve read to this point are sounding familiar, that’s because they broadly align with the concept of SMART goals —that is, making sure that your hypothesis is specific, measurable, achievable, relevant, and timely. 

Making sure your hypothesis is clear and testable will also ensure you can easily interpret your results, removing or reducing the risk of subjective interpretation. 

5. Don’t ignore context

How does your test fit into your broader conversion rate optimization goals and strategies ? As much as you’re hypothesizing about the specific results of your test, a strong hypothesis also acknowledges how your experimentation will benefit your efforts at large. 

Whenever you’re developing a hypothesis, consider other potential interactions with elements on your website, your email marketing, or even the products and services you’re offering. Changing an element in one location may have different effects depending on where it sits in the overall buyer’s journey. 

6. Keep it realistic

The final best practice to keep in mind when developing a strong A/B test hypothesis is to aim high but keep a level head—no matter how ambitious you may be, it’s important to keep your expectations realistic and grounded. 

An overly ambitious goal could lead to disappointment at best and missed targets at the worst. This is why it’s critical to benchmark and refer to past performance. 

Even so, that doesn’t mean you need to be constantly doubting your efforts. Just because you’re being realistic about your expectations doesn’t mean you have to scale down tests and initiatives. 

Take, as an example, travel company Going , who ran a simple A/B test around CTA phrasing that led to a 104% increase in conversions . 

Don’t miss out on the latest industry trends, best practices, and insider tips for your marketing campaigns

Start testing with confidence

So there you have it—everything you need to get started with a strong A/B testing hypothesis. We’ve covered:

• The elements you need to include in a good hypothesis
• An example of an A/B testing hypothesis you can use to inspire your own hypotheses
• Six key best practices to keep in mind when developing a hypothesis

Buf if there’s one thing you should take away from this article, it’s that A/B testing isn’t some huge, intimidating topic that only seasoned professionals can tackle.

Far from it—one of the best ways to get comfortable with A/B testing is to run some experiments of your own. 

If you’re not sure where to start, don’t worry—we’ve got you covered. First thing’s first, check out our guide to the essential of A/B testing . Then once you’re ready to go, the Unbounce A/B testing tool can help you figure out your best-performing pages so you can get the most out of your landing page experiments.

Related articles

A/B Testing

How a three-word a/b test led to triple-digit conversion growth.

March 5, 2024 . 7 minute read

Landing page optimization

How to score double-digit conversion rates—a marketing hero’s journey.

May 27, 2021 . 9 minute read

10 A/B testing examples and case studies to inspire your next test

May 1, 2024 . 17 minute read

What to A/B test: 10 A/B testing ideas to inspire your experiments

May 13, 2024 . 13 minute read

Conversion optimization

12 must-read cro case studies to inspire your next campaign.

April 19, 2024 . 20 minute read

What is CRO testing? A step-by-step guide to running efficient tests

April 16, 2024 . 17 minute read

Are you getting the most out of landing page testing? Here’s the best way to get results

March 18, 2024 . 28 minute read

The 16 best A/B testing tools and software alternatives to Google Optimize

March 19, 2024 . 23 minute read

(Guide) how to choose & customize a landing page template

July 29, 2020 . 10 minute read

What is A/B testing? A step-by-step guide with ideas & best practices

March 16, 2024 . 29 minute read

Explore our resource library

Get actionable insights, expert advice, and practical tips that can help you create high-converting landing pages, improve your PPC campaigns, and grow your business online.

Landing pages

Digital marketing, lead generation.

Learn / Guides / A/B testing guide

Back to guides

6 A/B testing examples to inspire your team’s experiments

A/B testing seems simple: put two different product versions head-to-head to see which one works better for your users.

But in reality, A/B testing can get complicated quickly. Your website has so many different elements—buttons, inputs, copy, and navigational tools—and any one of them could be the culprit of poor conversion rates. You want to ensure you have the right tools and processes to solve the case.

That's why you need to analyze A/B testing examples—to see what kind of strategies and tools other companies used to successfully carry out their experiments.

Last updated

Reading time.

This article looks at six A/B testing examples and case studies so you can see what works well for other businesses—and learn how to replicate those techniques on your own . You’ll walk away with new ways to test website improvements that boost the user experience (UX) and your conversion rates.

Conduct A/B tests with confidence

Use Hotjar’s tools to see how users experience different versions of your product

6 brilliant A/B testing case studies to learn from

Product and website design is not just an art; it’s also a science. To get the best results, you need to conduct A/B testing: a controlled process of testing two versions of your product or website to see which one produces better results.

A/B testing, also known as split testing , follows a predictable structure:

Find a problem

Create a hypothesis of how you could solve it

Create a new design or different copy based on your hypothesis

Test the new version against the old one

Analyze the results

But within this structure, you have many choices about the A/B testing tools you use, the types of data you collect, and how you collect that data. One of the best ways to learn and improve is to look at successful A/B testing examples: 

1. Bannersnack: landing page

Bannersnack , a company offering online ad design tools, knew they wanted to improve the user experience and increase conversions —in this case, sign-ups—on their landing page.

Unsure where to start, Bannersnack turned to Hotjar Heatmaps to investigate how users interacted with the page. With heatmaps, the company could visualize the areas with the most clicks and see spots website visitors ignored .

With A/B testing, Bannersnack discovered that a larger, higher-contrast call-to-action button made a huge difference. Check out the heat difference on these before-and-after click maps!

With this data, Bannersnack could hypothesize how to improve the experience and then create an alternate design, or variant, to test side-by-side with the original. 

Bannersnack completed multiple rounds of testing, checking heatmaps each time and getting incrementally closer to their desired results. Ultimately, they realized they needed a larger call-to-action (CTA) button with a higher contrast ratio—and sign-ups increased by 25%.

💡Pro tip: optimize your landing page by breaking down drivers, barriers, and hooks. 

Drivers are the reasons a lead came to the page

Barriers are the reasons they’re leaving

Hooks are the reasons they convert

Once you fully understand customer behavior on your landing page, you can develop—and test—ideas for improving it. 

2. Turum-burum: checkout flow

Digital UX design agency Turum-burum aimed to optimize conversions for their customer Intertop, an ecommerce shoe store based in Ukraine. 

In the UX analysis phase, Turum-burum used Hotjar Surveys —specifically, an exit-intent pop-up —to gather user insights on Intertop’s checkout page. When a user clicked to leave the page, the survey asked, “Why would you like to stop placing the order?” Out of the 444 respondents, 48.6% said they couldn’t complete the checkout form.

The next step was to develop hypotheses and A/B test them. Turum-burum tested changes like reducing the number of form fields, splitting the webpage into blocks, and adding a time-saving autofill feature.

Each time they tweaked a page, the company used Hotjar Recordings and Heatmaps to see how users experienced the change. Heatmaps revealed trends in users’ click and scroll behavior, while Recordings helped the team spot points of friction, like rage clicks, users encountered during the checkout flow.

The final result? Intertop’s conversion rate increased by 54.68% in the test variant. When they officially rolled out the changes, the average revenue per user (ARPU) grew by 11.46%, and the checkout bounce rate decreased by 13.35%.

Hotjar has flexible settings for heatmaps and session recordings, which is especially useful when you’re A/B testing and want to see how users experience each version of your design.

3. Spotahome: new features

A/B testing doesn’t have to be stuffy or stressful. Online booking platform Spotahome keeps it casual and fun with Hotjar Watch Parties .

Right now, people in product and engineering at Spotahome use Hotjar on a daily basis. We’re always running A/B tests and using Hotjar to see how the new feature performs.

Developers gather virtually, over a video call, and watch recordings of users interacting with new features.

For example, when watching recordings of users experiencing their new sign-up flow, developers noticed a broken button. 

While they might have grimaced and groaned when they spotted it, the moment allowed them to catch a problem that would’ve cost them conversions.

💡Pro tip: don’t be afraid of negative results when running A/B tests. 

Johann Van Tonder , CEO at ecommerce CRO agency AWA digital , says, “A test with a strong negative result means you’ve identified a conversion lever. You’ve pulled it in the wrong direction, now just figure out how to pull it in the opposite direction.”

Johann says he often gets even more excited about negative results because they showcase how valuable A/B testing actually is. 

“We tested a redesigned checkout flow for a famous car rental company,” he says. “It would’ve cost them £7m in annual revenue if they’d just made it live as is.”

Even though negative results are sometimes inevitable, there are some common A/B testing mistakes you need to be aware of, so you can get maximum results from your experiments. Check out the top A/B testing mistakes chapter of this guide (coming soon!) to learn more.

4. The Good: mobile homepage

Ecommerce CRO experts The Good took on the task of achieving higher conversion rates on mobile for client Swiss Gear, a retailer of outdoor, travel, and camping supplies.

To uncover any existing issues or bottlenecks, The Good turned first to Google Analytics to determine where, when, and why visitors left the website . 

With this quantitative data as a starting point, the company cued up Hotjar Heatmaps, which are free forever , to highlight users’ click and scroll patterns. Then, they used Hotjar Recordings to determine the why behind user behavior — the qualitative data —and form their hypotheses about how to make improvements. 

The Good tested their hypotheses, using heatmaps and recordings again after each test to see how changes impacted user behavior.

The Good used Hotjar Heatmaps to understand how users interacted with content filters, and used this data to redesign client Swiss Gear’s mobile menu to be more user-friendly.

The Good discovered that users were getting confused by the iconography and language on Swiss Gear's mobile site. The process led the team to design a simple, visually appealing menu-driven user interface (UI) for the mobile homepage.

This interface streamlined the browsing experience by promoting top filters—a move that led to big results: Swiss Gear’s mobile bounce rate dropped by 8% and time on site increased by 84% .

💡Pro tip: use Hotjar Engage for even more insights when optimizing your mobile site. 

Engage lets you source and interview real users about how they experience your site on their phones. Then, you can filter these interviews by type of phone, like Android or iPhone, to look for usability trends.

Recruit from Hotjar’s pool of 175,000+ verified participants and automatically screen to make sure you’re speaking to the right people

5. Re:member: application form

Re:member , a Scandinavian credit card company, knew something was wrong with their funnel. Google Analytics showed that many qualified leads arrived from affiliate sites—and quickly left before they signed up for a credit card.

Using Hotjar filters , re:member’s Senior Marketing Specialist, Steffen Quistgaard, pulled up recordings and click maps of sessions from affiliate sites only. 

While studying these sessions, Quistgaard noticed users scrolling up and down, clicking to the homepage, and hovering over—and attempting to click on—the benefits section.

Putting together these behavioral trends, Quistgaard hypothesized that leads were hesitant and needed more persuasive information on the form.

Re:member redesigned their credit card application form with more visual organization on the right side for users: three distinct content sections, checkmarks instead of bullet points, and icons in the rewards program section.

Re:member’s team redesigned the form, using visual and web design hierarchy cues to call attention to the credit card’s features and benefits. Then, they conducted split testing.

The result? Form conversions went up 43% among users from affiliate sites and 17% overall.

💡Pro tip: use filters to spend less time searching and more time analyzing. 

If your site experiences high traffic volume, you could rack up many recordings in a short time. (No worries! You get 1,050 session recordings for free every month on the Hotjar Basic ‘free forever’ plan. 💪) 

To make the most of your time, you need to sort through your recordings in the most efficient way.

Hotjar offers several filters that you can use, depending on your goals: 

Finding broken elements or bugs: sort recordings by rage clicks , errors , or u-turns (when a user returns to the previous page in under seven seconds).

Test a new feature: verify your assumptions about how a new button or link is performing with the clicked element filter. This lets you refine your sessions to only see those sessions where users actually clicked on the featured element.

Compare two versions of your page: filter by events to better understand your A/B test results. By setting up each page variant as a separate event, you can easily separate the recordings before watching them.

6. Every.org: donation flow

Dave Sharp, Senior Product Designer at charity donation site Every.org , was watching session recordings when he noticed something interesting: a surge of rage clicks, or a series of repeated clicks in a short time, on their donation form.

After watching many sessions, Dave hypothesized that the form’s two CTAs were confusing and frustrating visitors.

Every.org created a new version of the donation flow, splitting it into two pages, each with only one CTA button. Then they tested it against the original version.

By the end of the A/B testing process, conversions had increased by a whopping 26.5%.

💡Pro tip: when running tests, save time with Hotjar features and integrations. 

While fine-tuning Every.org’s donation flow, Dave used three time-saving tricks to narrow down the number of recordings he was watching: 

Filter by URL: this filter meant Dave could focus on user activity on certain pages—he could skip sessions of users on the blog, for example, and focus on activity closer to the point of conversion

Sort by relevance: instead of watching users’ sessions chronologically, Dave chose to sort them by relevance . That meant Hotjar’s algorithm did the heavy lifting, finding the most insightful recordings for him.

Set up alerts: to save even more time, Dave used Hotjar’s Slack integration to get an alert each time new recordings surfaced of users trying the updated donation flow.

Every.org gets thousands of visitors each month, but with these strategies, Dave made quick work of a task that could otherwise seem daunting.

Get closer and closer to what users need

You can’t just rely on gut instinct when making changes to your website. To create a site visitors enjoy (and one that gets results), you need to collect real evidence and user insights. 

By looking at A/B testing examples, you’ll have a clear roadmap of how to identify a problem, create a hypothesis, and test variants of your site. In time, you’ll have a site that delivers exactly what your target audience wants—and keeps them coming back for more.

FAQs about A/B testing examples

What is a/b testing.

A/B testing is a controlled experiment in which you run two different product or website versions simultaneously and see which one performs better. For example, you might run your current sales page against a new version with a section that addresses objections. Then, you’d gather and analyze data to see which one resulted in more conversions.

Why is A/B testing important?

With A/B testing, you become data-driven instead of relying on your instincts when making improvements to your website design. It helps you identify and address issues like bugs, confusing layouts, and unclear CTAs to create a more satisfying user experience that decreases bounce rates, increases conversion rates, and gets you return customers. 

What can I learn from A/B testing examples?

A website is packed with content, images, organizational features, and navigational tools, so it’s sometimes hard to know where to start to make improvements. Looking at other companies that have had success with A/B testing can spark ideas as you develop your own approach. Here are a few A/B testing case studies we recommend checking out:

Bannersnack boosted its landing page conversions

Turum-burum improved shoe retailer Intertop’s checkout flow

The Good redesigned Swiss Gear’s mobile menu

Spotahome looked for bugs in new features

Re:member increased application form conversions

Every.org made its donation flow better for would-be donors

A/B testing framework

Previous chapter

A/B testing tools

Next chapter

Like our new look? Read on .

• Start for free

A/B testing explained

A copywriter at Funnel, Sean has more than 15 years of experience working in branding and advertising (both agency and client side). He's also a professional voice actor.

Whether you love it or find it suspect, A/B testing is a concept that every marketer should be familiar with. So, what is it exactly? 

Well, A/B testing refers to a specific form of experimentation that marketers often use to determine the effectiveness of various efforts. And while some work with it every day, others wonder what the difference between A and B are. 

Join us as we explain what A/B testing is, walk through some real-world examples and explore the limitations of the methodology. 

What is A/B testing? 

At its core, A/B testing determines which of a set of variables has the most significant impact on a given metric. In marketing, A/B testing can be used to identify which call-to-action on a web page generates more conversions or which copy or image in an ad is most effective.

A real-world example

To give this definition further context, we can use an example from our own homepage. We wanted to test which media type drove more conversions: an image of a happy customer or a video explaining our product.

Specifically, we wanted to see if either option had a greater influence on a visitor clicking the “Book a Demo” button than the other. To do so, we decided that we would show 50% of visitors option A and 50% of visitors option B. The choice was made at random. 

Note: You don’t need to split your test evenly. Depending on the variables, length of the testing period and other factors, you may decide to weight the test differently. 

Why is A/B testing important for marketers?

A/B testing offers marketers a simple and straightforward way to determine the effectiveness of certain design and messaging choices by gathering real-world data. Without testing, well, you’re just guessing. 

Let’s revisit our homepage example. Let’s say this whole project started off with a meeting about how to drive more “Book a Demo” conversions from the homepage. One person may raise their hand and say we need to put a compelling video at the top. Another person may then say that it’s best practice to show a happy customer. 

How do you decide which option to go with? Both people are steadfast in their opinion. Do you go with the democratic route and just vote on which to go with? The elected option may be less effective, and that choice may even cause conversions to go down. 

Additionally, if you do happen to notice an uptick in conversions, you won’t necessarily be able to claim that the higher performance is due to the selection you made. Rather, it could be due to a spike in ad performance, a celebrity mentioning your brand, seasonality or a host of other reasons.

Instead, by testing the options with A/B methodology, you can gain a clearer picture of which choice is right. 

What’s needed for A/B testing? 

So, you understand the importance of A/B testing at this point and you now want to start testing for yourself. Let’s quickly run through the core components of a good test. 

Control group

A/B testing isn’t just about comparing two new variables. You need some sort of baseline (or norm) to test against. This is called your control group. 

You might remember this concept from science class or the pharmaceutical industry. It measures what happens when no change is applied at all. It’s this control group that helps you account for those unrelated spikes in conversions like seasonality, etc. 

In the case of our homepage, the control group would see the current version of the homepage without option A or B. 

Any good experiment is trying to prove or disprove something. Otherwise, you are just doing stuff for no real reason at all. 

This means that you should have some sort of claim in your mind ahead of time. In our example, one person claims that a customer image will drive more conversions, and another person claims a video will drive more conversions. 

Those hypotheses give us the variable to test (image or video) as well the means by which to measure them (homepage conversions). 

Additionally, the colleague favoring the video may have an even better hypothesis: using a video on the homepage will lead to a 30% increase in conversions. This kind of detail in your hypothesis is great for later on when you need to think about test length and volume required so you can see a statistically significant signal in the data. 

A good format to use for your hypotheses goes: If_____, Then_____, Because______. For example, if I add a video to our homepage, our conversions will increase by 30%, because the video will effectively demonstrate the value of our product. 

You know we love a good metric, and they are critical to setting up your test effectively. In our example, we will want to primarily view conversions. However, another test may require you to examine engagement, time on page, click-throughs, acquisition cost and more. 

Your choice of metrics should be driven by your detailed hypothesis. Additionally, you need to be sure that you can accurately measure that metric. For instance, trying to A/B test something like brand awareness between two options of an ad creative could be very difficult to accurately measure — and it could take a very long time to do so. 

Great A/B testing tools:   Survey Monkey and  ABTasty

More A/B testing examples

Let’s put our homepage test aside for a moment and look at other ways marketers can employ A/B testing. 

Email optimization

First, think of your email marketing program. If you're an online retailer, you may have a customer retention or loyalty program that reaches out to shoppers who’ve made a purchase from your store within the last six months. All of those emails will need a subject line, which provides any opportunity to test different options. 

Again, though, you’ll need a hypothesis to test. Are you trying to test the open rate? If so, how much of an increase do you envision? And why are you focusing on the open rate at all? 

Then, once a customer opens your email, there are a whole host of elements that can be A/B tested. If you’re a fashion brand, you may want to test highlighting new seasonal releases versus related products based on the customers’ past purchases. 

But beware: the plethora of options that can be tested requires a strong framework and even stronger data governance . 

Improving app revenue

Second, let’s imagine you have an app through which users can be monetized. You may want to test different interfaces, colors, button styles and more against the rate of purchase conversion. 

It’s important to note that A/B testing calls for running separate tests for each of these variables, with each test having a control group. If running each of those tests may take too long, you may want to think about multi-variate testing — though that’s a subject for another day.

What’s the deal with A/B testing?

While we’ve spent a significant portion of this piece extolling the virtues of A/B testing, that’s not to say it’s without limitations. Welcome to our hot takes on this methodology.

Statistical significance

In order to gain usable data from any A/B test, you need to hit statistical significance. In other words, there can be a lot of random fluctuations in your data (called noise). Any “results” from your testing need to noticeably stick out from this noise. The level at which it needs to stick out is mathematically calculated based on several factors like test volume, hypothesis clarity and more. 

That can be a very hard thing to achieve for certain companies trying to run an A/B test. For instance, if we had a high degree of conversion irregularity (say, +/- 10% day to day), we would need to measure a large increase in conversion rate over that control group, perhaps by 30% or more, to make any determinations from the resulting test data. 

If your results fall within the range of day-to-day fluctuations, you probably can’t make a determination from the data. Let’s explain using the 10% conversion irregularity. 

Imagine we tested the video on our homepage. One day, we saw a 5% uptick in conversions. The next day, we saw a 5% decrease. And the following day, we saw a 12% increase in conversions. 

The first two days can be easily understood as falling within the “noise” of everyday operations. However, the third day could perhaps show a signal to us, though it’s not that really far off from our 10% daily fluctuations, and certainly not enough to draw any insights. 

However, imagine that, over the course of a year and with a representative amount of web traffic, that 12% increase in conversions became the daily average. We may be able to begin making inferences, though it may also signal that we need to test more. 

That’s all to say that this sort of testing requires volume and time, meaning websites with low visitor volume will likely struggle to run these sorts of tests. 

And for those whose websites have insufficient visitor volume? Find other hypotheses that can be tested that will have the largest impact on your business. 

A/B tests can slow you down

To be fair, taking the time to run any experiment can be a slower alternative to just following your gut instinct right away. For some companies and situations, though, A/B tests can be particularly cumbersome and may not result in any worthwhile insights. Let’s explain what we mean. 

Imagine you’re working on a brand identity refresh for your company. You’ve spent a lot of time, energy and money with your team and an external agency. You’ve gone through several concepts and iterations, and you are now down to your final two options. 

Perhaps you’d like to perform an A/B test with a sampling of current and/or prospective customers to see which identity refresh really moves the needle. 

Except… you’ve just run into a few fundamental problems. First, brand identities are highly subjective from person to person. After all, some people just hate blue. 

Second, and perhaps more importantly, you’re lacking a clear hypothesis or rational metric. In theory, you may want to know that brand refresh A may lead to higher revenues than refresh B (and especially more than keeping your brand as it is). Except brands don’t really work like that - nor do you have a specific revenue increase goal in mind. 

Plus, there’s no easy way to launch both identities, and the control group, to the same audience subset as we did in our homepage example. That would cause incredible market confusion. 

Now, I know what you’re about to suggest: focus groups! Again, see above regarding subjective perspective. And without objective data (rather than opinions) you’re sort of throwing your money and time away. 

Additionally, A/B testing can quickly seem like the best way to determine every single element of your website’s UI and UX. Yet, implementing all of those tests to all of those elements would take so long that your website would be out of date by the time you launch anything. In many instances, you need to rely on the expertise of your team to launch projects in a timely manner. 

There is another… test

Let’s say you have a solid hypothesis that can be tested through quantifiable means. That still doesn’t mean A/B testing is the right course of action. You may want to test multiple variables all at the same time. In that case, you’ll want to explore multi-variate testing. 

Or, perhaps you want your testing to inform and improve a specific product. In that case, it may make more sense to layer your tests and continually iterate – learning as you go. 

What we’re trying to say is that, while A/B testing can feel like it provides you with a black-and-white answer to the marketing world’s most pressing questions, there is always an alternative route you can follow. Oftentimes, these alternatives offer a better solution for your specific needs. 

That’s not to say that A/B testing isn’t valuable (it is!), it’s just that it’s not a panacea that many marketers make it out to be. 

What do you think? 

So, are A/B tests manna from heaven for marketers, are they a waste of time or are they somewhere in between? We fall into the third category. They can provide rich insights (when executed properly), but we also don’t want to be led around by those by test data. Sometimes, human instinct can figure out innovative solutions that lead to even happier customers. 

Let us know what you think. Drop us a line on LinkedIn , or get into the comment section on our YouTube channel . 

A multivariate test (MVT) is an experimental technique used in conversion rate optimization (CRO) and marketing to evaluate multiple variables simultaneously to determine the most effective combination. It extends the concept of A/B testing, which typically compares two versions of a single element, by testing multiple combinations of several elements at the same time.

An A/B test should run long enough to reach statistical significance, which typically means collecting data for at least one full business cycle (e.g., one week) to account for daily variations, and until a sample size is reached that provides confidence in the results. Using an online calculator can help estimate the necessary duration based on your traffic and expected effect size.

Solving agency cash flow issues

How to work with marketing data across international borders

What does data quality mean?

• Integrations

A/B Testing Plan Template

An A/B testing plan template to help you execute A/B tests more effectively

A/B testing (also called split testing) is an important part of conversion optimization. While it can be tempting to trust your intuition when it comes to creating landing pages, email copies, call-to-action banners, if you simply make decisions based on “feelings”, you might be losing a lot of conversions that you might otherwise be able to get.

By running A/B tests, you can test out your hypotheses and use real data to guide your actions. This template helps you plan for an experiment in a more structured way. This ensures an experiment is well thought out and it also helps you tos communicate it more effectively with designers, developers and others who will be involved in implementing the test.

How to use this A/B testing plan template

1. hypothesis.

The key part of a A/B test is formulating your hypothesis as this basically guides the whole A/B test plan.

• What problem are we trying to solve?
• Its impact? (e.g. how big this problem is to our customers?)

Define the problem

In formulating the hypothesis, first you need to define the problem you want to solve. For example, you are an SaaS that offers free trial and you want to improve the traffic-to-lead conversion ratio (i.e. attracting more website visitors to actually sign up for a free trial). But that problem might be too broad to form an A/B test as you can simply test one variable in an A/B test to be effective (otherwise you won’t know which variable is causing the change).

So to narrow down the problem you want to solve, you need to find out the bottle-neck in the conversion funnel – where do people drop off the most? Are there any key information or call-to-action buttons that you expect people to read/click but they didn’t? You can use heatmaps and session recording tools like Hotjar and Fullstory to help you identify bottlenecks more easily.

Formulating the hypothesis

After narrowing down the problem you want to solve, you then need to make a hypothesis as in what causes those bottlenecks and what you can do to improve.

For example, you noticed most of the visitors will visit your “Features” page but very few of them will actually scroll past even half of the page so many features that you think are important are not actually viewed by the visitors. To improve this, one hypothesis might be using tab or toggle list design to make your page shorter and visitors can select to dig deeper into content that they are interested in by expanding the content.

Remember when formulating your hypothesis, change only one variable so that you will know it’s really that variable that is causing the change in conversion.

Result and success metrics

Now you have your hypothesis, the next is to plan how you are going to measure your results. Defining your success metrics carefully beforehand is important. Otherwise, if there is not enough tracking done during the experiment, it might be hard to draw conclusions and next steps at the end of the experiment.

2. Experiment setup

To communicate clearly to the implementation team, detail out the experiment setup that you will use to test your hypothesis. This include

• Location – where will this experiment be set up? Provide the URL of the page that you are going to set up your test with annotated screenshots on which parts you would like to change
• Audiences – will all visitors or users be able to view the experiment? Or you will only allocate x% of traffic to the experiment? Lay out the details with the rationale behind
• Tracking – with the success metrics that you have defined, what tracking needs to be set up? Try to provide more explanation on how you will use this metrics to analyse the results to ensure the implementation team will set up the tracking the right way.

3. Variations design

In this section, describe what variations you would like to test.

Layout the design work related and add diagrams, mockups and designs related to the confirmed variation that you’d like to test. Gathering all these in one place helps your development team understand the context much better.

4. Results and learnings

So at the end of the planned experiment period, you get all the stats but does a better conversion rate for one variation really conclude that variation is really better? You need to run a test of statistical significance to see whether your results are really statistically significant. You can use this A/B Testing Calculator by Neil Patel to check the results easily by inputting the sample size and conversion numbers of the variations.

If one variation is statistically better than the other, then you have the winner and can then complete the test by disabling the losing variation.

But if neither variation is statistically better or the original version is still better, then you might have to run another test.

Document any learnings you got from this experiment so that it can help you better plan your future ones.

5. Next actions to take from this experiment

From the results and learnings section, list out the action items that you would need to do after the experiment. Is that you would need to disable the losing variation? Is there more elements on that page that you want to test to further improve conversion rate?

A/B testing is a continuous process. Hope this template can help guide you in executing better split tests.

Related templates

Standard Operating Procedures (SOPs) Template

OKR Template (Objectives & Key Results)

Interview Scorecard Template

Unleash your teams’ full potential with a knowledge base they actually use every day.

14-day free trial. No credit card required.

OUR INVESTORS

• What's new
• Cases studies
• Zendesk Grader
• Help center
• [email protected]
• +1 415 871 0545

OTHER LANGUAGES

Example A/B test hypothesis

Shana Rusonis

Imagine you set out on a road trip. You packed the car, made a playlist and set out to drive 600 miles—but you don’t actually know where you’re headed. You remember to top off the gas tank before you leave and pack snacks. But you arrive at a destination, and it’s not at all what you imagined it would be.

Running an experiment without a hypothesis is like starting a road trip just for the sake of driving, without thinking about where you’re headed and why. You’ll inevitably end up somewhere , but there’s a chance you might not have gained anything from the experience.

“If you can’t state your reason for running a test, then you probably need to examine why and what you are testing.” — Brian Schmitt , Conversion Optimization Consultant, CROmetrics

Creating a hypothesis is an essential step of running experiments. Although you can set up and execute an experiment without one, we’d strongly advise against it. We’d even argue that a strong hypothesis is as important as understanding the statistical significance of your results (see our sample size calculator for more info).

‘Hypothesis’ defined

A hypothesis is a prediction you create prior to running an experiment. It states clearly what is being changed, what you believe the outcome will be and why you think that’s the case. Running the experiment will either prove or disprove your hypothesis.

Hypotheses are bold statements, not open-ended questions. A hypothesis helps to answer the question: “What are we hoping to learn from this experiment?”, while ensuring that you’ve done due diligence in researching and thinking through the test you’re planning to execute.

The components of a hypothesis

A complete hypothesis has three parts. The statement follows: “If ____, then ____, because ____.” The variable, desired result and rationale are the three elements of your hypothesis that should be researched, drafted and documented prior to building and setting an experiment live.

Let’s look at each component in more detail and walk through an example:

The Variable

A website or mobile app element that can be modified, added or taken away to produce a desired outcome.

Tips to select a variable : Try to isolate a single variable for an A/B/n test , or a select handful of variables for a multivariate test. Will you test a call to action , visual media, messaging, forms or other functionality? Website or app analytics can help to zero in on low-performing pages in your website or user acquisition funnels and inform where you should be looking for elements to change.

Example : Your website has a primary call to action that’s above the fold on your homepage. For an experiment, you’re going to modify this variable and move it below the fold to determine if conversions will improve because the visitors have read more information.

The predicted outcome. This could be more landing page conversions, clicks or taps on a button, or another KPI or metric you are trying to affect.

Tips to decide on the result : Use the data you have available about your current performance to determine what the ideal outcome of your experiment will be. What is the baseline metric that you’ll measure against? Is the change to the variable going to produce an incremental or large-scale effect?

Example: Maybe your desired result is more conversions, but this may not always be the result you’re aiming for. Your result might be to reduce bounce rate by testing a new navigation or recommended content module.

Demonstrate that you have informed your hypothesis with research. What do you know about your visitors from your qualitative and quantitative research that indicates your hypothesis is correct?

Tips to craft the rationale : Show that you’ve done your experiment homework. Numerical or intuition-driven insights help formulate the “why” behind the test and what you think you’ll learn. Maybe you have input from customer interviews that helped formulate the hypothesis. Maybe you’ve seen an application of the change being tested work well in other experiments. Try using qualitative tools like surveys, heat maps and user testing to determine how visitors interact with your website or app.

Example: The rationale for testing a new headline on a landing page might be: removing your company’s name from the homepage headline will improve conversions because I’ve conducted surveys that indicate our language is confusing. Borrowing customer language from feedback surveys will improve our performance.

What are the outcomes of a strong hypothesis?

A thoroughly researched hypothesis doesn’t guarantee a winning test. What it does guarantee is a learning opportunity, no matter the outcome (winner, loser, or inconclusive experiment.)

Winning variation? Congratulations! Your hypothesis was correct. If your variations were inconclusive or lost, the hypothesis was incorrect, which should tell you something interesting about your audience.

“When a test is based upon a thorough research and a clear hypothesis to test, you learn about your audience with every test. I always segment the testing results per device type, browser, traffic source and new/returning visitors. Sometimes the average uplift isn’t the best metric to examine. By segmenting the results, you can find the true winner.” — Gijs Wierda , Website Conversion Specialist, Catchi Limited

Maybe you crafted a hypothesis based on ‘conventional wisdom,’ or read and A/B testing case study and decided to replicate it on your own audience. The variation lost, but you and your team learned that what works for other sites and apps doesn’t work for you. Go forth and craft a new hypothesis, and uncover your own best practices!

Tip : Document both your research and your hypotheses. Remember to share a hypothesis along with the key experiment metrics when publicizing experiment results within your team. Your library of experiment hypotheses will become a valuable reference point in creating future tests!

How does a hypothesis fit into your experiment workflow?

According to Kyle Rush, Head of Optimization at Optimizely, a hypothesis is a key component of every test and should be tackled right after you identify the goals of the experiment. Here’s his experiment process:

• Identify goals and key metrics
• Create hypothesis
• Estimate test duration with a sample size
• Prioritize experiments with projected ROI
• QA the experiment
• Set test live
• Record and share results
• Consider a retest

Steps 1 and 4 of this process are just as important as the hypothesis creation. Keep in mind that not all hypotheses are created equal . Your team may have an interesting idea, or there may be a disagreement that you’re trying to settle—but that doesn’t mean it’s the most important thing to test.

Prioritize and test based on parts of your site or app that have high potential for business impact (revenue, engagement or any other KPI you’re trying to improve.) Use your analytics to identify these areas, and focus on crafting hypotheses that can support improvements in these areas. Resist the urge to test just for the sake of testing, and focus on high-impact changes to your variables.

“Everything starts and ends with the hypothesis. Ask, ‘What business or customer experience problems do we think we can solve for mobile and why do we think those changes will impact a certain metric?’ Ultimately, time is the most valuable asset for any company … so we start by crafting hypotheses we believe in and then prioritize those hypotheses against all other opportunities we have to test. [T]he success of your optimization program is most correlated to your ability to identify test hypotheses that will move the needle and your ability to tell the story with the resulting test data.” —Matty Wishnow, Founder & CEO, Clearhead

How you can get started

Here’s your actionable cheat sheet:

Hypothesize for every outcome:  Make every experiment a learning opportunity by thinking one step ahead of your experiment. What would you learn if your hypothesis is proven correct or incorrect in the case of a variation winning, losing, or a draw?

Build data into your rationale: You should never be testing just for the sake of testing. Every visitor to your website is a learning opportunity, this is a valuable resource that shouldn’t be wasted.

Map your experiment outcomes to a high-level goal: If you’re doing a good job choosing tests based on data and prioritizing them for impact, then this step should be easy. You want to make sure that the experiment will produce a meaningful result that helps grow your business. What are your company-wide goals and KPIs? If your experiments and hypotheses are oriented towards improving these metrics, you’ll be able to focus your team on delving into your data and building out strong experiments.

Document your hypotheses: Document all of the experiments you run. This habit helps to ensure that historical hypotheses serve as a reference for future experiments and provide a forum for documenting and sharing the context for all tests, past, present, and future.

Crafting great hypotheses is a skill learned over time. The more you do it, the better you’ll get.

About the author

Content & Beyond 2024: The 7 key takeaways for marketers everywhere