What is A/B testing?
Introduction
We live in an era of data-driven marketing, far away from the times when marketers simply made decisions based on guesswork and intuition and hoped for favorable outcomes. The modern day marketer has a scientific approach and relies on data. And A/B testing is the best way to remove uncertainty and gut feel when making marketing or design decisions for websites, ads or other digital campaigns.
Embrace the experimentation mindset.
By basing your strategy on data and A/B tests, you’ll be agile but most importantly you’ll have guaranteed feedback on what works and what doesn’t. You’ll be better placed to make sound business decisions and to invest time and money in what your visitors actually want.
A/B testing to drive marketing success
75% of websites with more than 1 million monthly visitors already run A/B testing programs.
But successful A/B testing requires preparation and education, and time and effort to put into practice. You’ll have to create a process, put a framework in place, learn about statistics, set up and learn a new tool and make sure you’re actually getting accurate results. But the effort and time is worth it, given the potential to achieve your marketing goals.
To help we’ve brought together in-depth content on A/B testing, from the best blogs and experts.
What is A/B testing (or Experimentation/Split testing)?
A/B testing is also known as split testing, which can be either exactly the same thing as A/B testing or mean split URL testing. For a classic A/B test, the two variations are on the same URL. Alternatively with split URL testing your changed variation is on a different URL (although this is hidden from your visitors).
What about multivariate testing (MVT)?
Sometimes, you want to test several changes on a page, for example the banner, header, description and video. To test all of these elements at the same time, you use multivariate testing (or MVT).
In this case you have multiple variants generated to test all the different combinations of these changes to determine the best one.
The big downside of multivariate testing is that it requires an enormous amount of traffic to be statistically accurate. Before starting a multivariate testing project, you need to check that your audience is high enough to provide representative results.
Dynamic traffic allocation or multi-armed bandit testing
Multi-armed bandit testing (or dynamic traffic allocation) is when your algorithm automatically and gradually redirects your audience toward the winning variation.
A/B/n testing
A/B/n testing is when you test more than two variations of an element or page. For example, you could test six versions of a page and do an A/B/C/D/E/F test.
Introductory resources on A/B testing to help you get started
What are the benefits of A/B testing?
Why should YOU do A/B testing? Put simply, it will help maximize the investment you’ve made in bringing traffic to your site. Once you have traffic, increasing your conversions is much less expensive than going out again and finding more. Optimizing the experience for your visitors therefore has a high potential ROI. But A/B testing also delivers other benefits:
- Learn deeply about your audience with every test: what they like, how they react, their needs and habits.
- Remove gut decisions from your marketing strategy by adopting an experimentation culture and testing everything.
- Focus your time and money on what your visitors respond to best, thanks to the results of your A/B tests.
With A/B testing you’ll be able to confidently answer these key questions:
- Which elements drive sales, conversions or impact user behavior?
- Which steps of your conversion funnels are underperforming?
- Should you implement this new feature or not?
- Should you have long or short forms?
- Which title for your article generates more shares?
How does A/B testing work?
You compare the current version (control) of a page/element against one (or more) variations of it with the changes you want to test. This could be a website page, an element in a page, a CTA, a picture, or bigger changes to the customer journey.
You divide your traffic into equal portions, and visitors are then randomly exposed to one or the other variation during the set period of time when the test is running. Then, their relative performance (in terms of metrics such as conversions or sales) are compared and analyzed to determine if the change(s) are worth implementing.
GOING DEEPER INTO THE INNER WORKINGS OF A/B TESTING
Why Conversion Rate Optimization (CRO) is key to your A/B testing success
Simply having lots of traffic doesn’t necessarily lead to success – you have to do something with it. That’s where CRO is invaluable.
How to optimize your conversion rate with A/B testing?
- Why experimentation is at the heart of success in digital marketing
- CRO best practice
- The Beginner’s Guide to Conversion Rate Optimization (ConversionXL)
- Building an optimization culture for conversion ROI
- The Definitive How-To Guide For Conversion Rate Optimization
- The Advanced Guide to Form Conversion Optimization
- Case studies from experimentation agency Widerfunnel
How to run A/B tests: frameworks and methodology
MEASURE, STUDY, ANALYZE YOUR WEBSITE DATA. IDENTIFY THE PROBLEMS AND OPPORTUNITIES
FORMULATE HYPOTHESIS (A GREAT WAY TO FORMULATE AN HYPOTHESIS BY CRAIG SULLIVAN)
PRIORITIZE YOUR TEST IDEAS: ONE OF THE MOST USED PRIORITIZATION FRAMEWORKS IS PIE, FIRST CREATED BY WIDERFUNNEL
With this framework, you rank your test ideas by looking at three criteria to determine which you should run first:
- Potential ./10: How much room for improvement is there on this (these) page(s)?
- Impact ./10: How valuable is the traffic on this (these) page(s)?
- Ease (of implementation) ./10: How easy will this test be to implement on your site?
You then average the three and you’ll know which tests to do first. There are multiple other frameworks available so try them and work out which best suits your needs.
Test your highest priority hypothesis
Analyze your test results and learn from them
COMMUNICATE RESULTS TO STAKEHOLDERs
Repeat
Great A/B testing processes and frameworks
What to A/B test: ideas for your experiments
You can basically test everything on your website:
But sometimes, you need inspiration. So here are examples of multiple A/B testing ideas to look through.
Note : Do bear in mind that what worked for others might not work for you. Don’t blindly apply other people’s ideas - make sure you thoroughly analyze to see if it’s relevant for you and how/if you can adapt it for your business.
Dozens of A/B tests ideas for you to get inspired
A/B testing best practices and common mistakes
A/B testing can be hard, and there are plenty of ways it can go wrong. So it’s good to be aware of potential problems and to put safeguards in place.
Make sure you are set up for success by studying best practices and possible issues that can arise. But as with test ideas, don’t take everything at face value. Put things to the test and see if it applies to your circumstances.
Best practices to help you win
- Highlighting the link between digital experimentation and growth
- Is web personalization an extension of A/B testing?
- Why testing and personalization are key to meeting changing consumer needs during COVID
- How to create an optimization culture
- Multivariate testing: save time and refine your analysis
- Dynamic traffic allocation: optimize your A/B tests
- How Intelligent Tracking Prevention (ITP) impacts A/B testing - and how marketers can overcome its restrictions
Note: We produce a fortnightly newsletter with in-depth content on A/B testing, personalization and CRO, simply subscribe here.
A/B testing mistakes
- [INFOGRAPHIC] 19 Ways A/B Testing Is Ruining Your Site (And How To Fix It)
- Should you run an A/A test?
- Why your brain is your worst enemy when A/B testing
- Are you misinterpreting your A/B tests results?
- Warning! Is the world sabotaging your A/B Tests?
- Are you stopping your A/B tests too early?
- 7 Mistakes most beginners make when A/B testing
- What is the flicker effect and how can you get rid of it in your A/B testing?
- 11 ways to stop FOOC’ing up your A/B tests
A/B testing reporting & results
A/B testing statistics and how to understand them
A/B testing is based on statistical methods. While you don’t need to know all the math involved in analyzing your results, having a basic knowledge of statistics will improve your chances of success.
There are two main statistical methods used by A/B testing solutions. One isn’t better than the other, they simply have different uses.
Frequentist approach
This allows you to see the reliability of your results thanks to a confidence level: if this is at a level of 95% or more, you have a 95% chance of it being accurate. But this method has a downside: it has a ‘fixed horizon’, meaning that the confidence level is valueless up until the end of the test.
Bayesian approach
This provides a result probability as soon as the test starts, so there is no need to wait until the end of the test to spot a trend and interpret the data. But this method also has challenges: you need to know how to read the estimated confidence interval given during the test. With every additional conversion, the trust in the probability of a reliable winning variant improves.
A/B testing skills & management processes
To tip the odds in your favor, there are a number of skills and management processes you can focus on, such as web analytics, UX design and communicating results.
Sharpen your skills for better A/B tests
Project management
Useful A/B Testing blogs to read
There are some great blogs available, containing useful A/B testing, experimentation and Conversion Rate Optimization-related content that will help you to learn, get inspired, and run better A/B testing programs.
Partner with these A/B testing agencies to outsource your programs
Outsourcing your A/B testing can be a great way to run programs if you don’t currently have the necessary resources, helping you learn at the same time. Here are some of the best ones out there.
Note: we sell an A/B testing tool but we can also support you with your testing programs, learn more on our customer success page.