Multivariate testing or A/B testing? Experts weigh in
You’ve recently launched a new product, but your conversions aren’t meeting your expectations and you have no idea why. Or maybe you do.
Maybe it’s the call-to-action (CTA)—it doesn’t stand out. Maybe it’s the form visitors have to fill out—there are too many fields. Or maybe it’s the hero image at the top of the page—it’s hard to read the headline over the image.
All these questions provide you with a starting point to hypothesize what is causing the drop in conversions. You can get the answer with testing—either multivariate or A/B testing.
Both types of tests require a hypothesis—a question you’re trying to answer. The hypothesis is based on what you think will happen if you change one or several elements. The goal is to learn about the customer in order to better serve their needs—testing should be the catalyst that gets you there.
“When it comes to multivariate testing, as with all types of tests, the focus should be on what you’re trying to test (the question you’re trying to answer) and the complexity of the test. Amount of traffic is just one factor in a much bigger picture. Other factors that are equally (if not more) important are level of maturity of the team, the complexity of the tests, and your tracking.”—Talia Wolf, Founder and Chief Optimizer, GetUplift
In the earlier example, a multivariate test would look at studying the CTA (colors, copy, etc.), the website layout (putting the CTA at the center of the page or on the side) or the image and text in the header block. An A/B test would only test one of these three elements at a time.
The two testing methods yield invaluable insights to help improve your customer experience based on quantitative analyses, taking the guesswork and assumptions out of improving your site or platform.
1 What is multivariate testing (MVT)?
MVT is used when testing multiple elements on a page, site, or digital ad. It provides insight into how those elements interact with each other for the best results. Experts agree that multivariate testing is about understanding how every piece fits into the larger puzzle of CX optimization and experimentation.
MVT examines several different variables at the same time and provides insight into how these elements work together to increase conversions. This method is better when you have more time and want to test more elements together.
2 What is multivariate testing good for?
Multivariate Testing finds the right combination of variables
MVT is all about finding out which combination of the variables, taken together as a whole, yields the best results for customer experience.
An example is experimenting with the alignment of a form to download gated content: Centred or right-aligned with the copy? Does this form have a blue or green CTA button? And does the CTA say “Download Now” or “Get the eBook”?
While one CTA might outperform the other in an A/B test, there are cases when another variable, when changed—in this case, positioning—negates that uplift. MVT will tell you which of the combinations will get the most downloads.
“Multivariate testing can be helpful in cases where you want to find out what combination of elements works best rather than which element is most effective on its own. For example, if one headline gets more clicks and another gets higher conversion rates, but only when used alongside a particular image, then using this approach would allow you to determine how all three work together as a unit — instead of simply measuring each piece independently like an A/B test would show.” —Ankur Goyal, Digital Marketing Evangelist, Website Pandas
Multivariate testing uncovers relationships between elements
Here’s a simplistic example to consider: The color of your CTA button (blue, red, green) and the copy of the button (Apply Now, Submit, Send) may seem like two separate elements, but when you’re testing each of these elements, they may complement one another. So finding the right combination (from the nine possible) will definitively tell you which gets more clicks (or conversions). Multivariate testing is the only way to determine the relationship between elements and how they work to drive conversions.
“Multivariate testing offers a tremendous deal of insight into the relationships that exist between various components. The multivariate test approach allows you to test several variables in parallel to discover which combination provides the most overall value for the organization. Multivariate tests are essential for determining the interrelations between independent components to determine which combination is the most effective.”—Andriy Bogdanov, CEO and Co-Founder, Online Divorce
3 What is A/B testing?
A/B testing only tests one element on your page, site, or digital ad at a time. The quantitative results that come from an A/B test allow the marketer to make an informed decision on what element gets the most conversions.
A/B testing only studies two variables (the control and an alternate) to find out which single element drives more conversions. Note that you can do A/B/n testing for more variables. Many prefer A/B tests to focus on one element and a clearly defined hypothesis for your experiment.
What is A/B testing good for?
A/B testing enables faster experiments
With only two elements to test (the control and the variable), the test can be conducted in a shorter time frame. That’s why your hypothesis should be on elements that have a greater impact on conversions, such as the CTA button or the number of fields to fill out in a form.
“An A/B test is when a singular yet prominent variable is changed and then tested to see if an increase in conversions has occurred. As an example, companies can A/B test the location of an add-to-cart button on a product description page. This requires far less time to test, as the change is singular and controlled.”—Jonathan Zacharias, Co-Founder, GR0
A/B test data is easy to interpret
Unlike MVT, A/B testing only takes into account one variable, which is tested against a control (usually what is already live). Since you’re only changing one element, it’s much easier to interpret the results since only that one element is the reason for the change—whether positive or negative.
“Use A/B testing when you need fast, meaningful results. It takes advantage of the changes from page to page, which are stark, making it easier to notice which page is most effective.”—Harriet Chan, Co-Founder and Marketing Director, CocoFinder
Knowing what works, what doesn’t work, and what has no effect on your conversions can help you continuously improve your digital assets.
4 When should you use A/B vs. multivariate testing?
A/B testing and multivariate testing examine different things and will give you different results — but that doesn’t mean you can only use one or the other. In fact, combining the two methods is a powerful tool for a more nuanced analysis.
Use multivariate testing to experiment with large changes and big ideas
MVT allows you to test a few elements at once — so potentially larger overall changes. These can be the layout of the page with a hero image, a video embedded, and different colors for the CTA. A/B testing, on the other hand, is better for seeing what single elements perform best — smaller changes. So only one of the elements from the MVT, like the position of the video on the page.
“Multivariate testing isn’t conditioned by the number of visits. You can get results even with fewer visits. The key difference is that multivariate testing shows you many different elements at once, while A/B testing is great for testing a single page element and its performance. In my experience, multivariate is great when you have two completely different pages for the same offer and you want to see which concept works better. On the other hand, if you know the offer is great, A/B testing is great for tweaking it—finding the best order of elements, the best CTA, UI, and UX, and so on.”—Adam Hempenstall, Founder and CEO, Better Proposals
Use A/B testing when you don’t have resources to look beyond conversions
MVT is best used when considering macro (or larger) changes, like page layout. But these macro changes can not only affect your conversions, but other analytics such as click-through rates, bounce rates, and cart abandonment.
For instance, if you’re testing the number of fields in a loan application page and whether the field labels are inside the field or outside, it could hypothetically deter people from filling out your form because they can’t read the label easily. This isn’t a reason to avoid MVT, but this is why you need to know the possibilities ahead of testing so that if these things do start to occur consistently, you know to change those elements.
“In broad-brush terms, multivariate tests tend to be better suited for tweaking smaller page elements like CTAs and image choice and A/B for more macro changes like layout, color schemes, and general page element symmetry. Multivariate testing is also better suited for analyzing the combined effect of several smaller changes to see how the interplay affects things like time on page, click-through rates, and shopping cart abandonment.”—Caroline Hoy, Head of PR and Marketing, Concord
For broad experiments, start with MVT and optimize with A/B testing
When your hypothesis is broad or if you’re not sure what to expect, starting with MVT might be a great place to start since it’s all about the relationship between different on-page elements. Once you’ve gotten your results and can form a more solid hypothesis, you can then move to testing those one or two final elements with A/B testing and receive a more complete picture.
“You should use A/B testing as a sort of final check between two options you have come down to; that can be after multivariate testing or not. But testing many variables at once can be complicated, but gives you much more intricate data, so depending on your use case, amount of traffic, and how long you want to test for, this may be the way to go.”—Bryan Philips, Head of Marketing, In Motion Marketing
5 Should you run multivariate or A/B tests for a specific amount of time?
There is a lot of misinformation out there regarding the minimum amount of traffic and how long it is required to conduct either test. The truth is, every website or digital ad is different, and so the answer is it all depends.
Decide your minimum number of conversions
You want to ensure you receive a minimum number of conversions (at least 150 per variation) in order to truly know if the results are reliable. Consider working backward, starting with how many conversions each variation can potentially happen in order to determine how many variations to test.
“Generally speaking, when running tests, we put a lot more emphasis on amount of conversions per variation vs amount of traffic we need to run a test. Our go-to is a minimum of 150 conversions per variation (during a two-week period) for us to know the test is significant. This means that when running a test, we will decide how many variations to include according to the amount of conversions each variation will receive.”—Talia Wolf, Founder and Chief Optimizer, GetUplift
Determine your timeline
Do you have a firm deadline as to when you need to implement the results? Since MVT takes longer, A/B tests are best for shorter timelines, this way you don’t run the risk of changing a few variables that may negatively impact your conversions.
“Multivariate tests are slower because they require multiple steps before results reach statistical significance (due to needing more significant sample sizes). A/B testing will allow us to make an informed decision quickly since no risk of changing multiple variables at once could skew results by confusing cause-and-effect relationships. So, while these methods have distinct pros and cons, both should be considered depending on your goals, timeline, resources, and risk tolerance.”—Ankur Goyal, Digital Marketing Evangelist, Website Pandas
Calculate the traffic you’ll need to each variant for statistical significance
Depending on how many combinations you’re testing, you could only be receiving 5% to 15% of your usual traffic to any test page. That’s because when you’re running a MVT, if there are six different versions to test, your testing platform will automatically divide your traffic into six equal parts to get results. That translates to 15% of your traffic will see each version during the test. And if you have more versions, then your traffic gets cut per version even more.
“A high number of daily visitors are required for multivariate testing because almost 5% to 15% of your audience is assigned to each variant. Therefore, you need enough visitors for each variant to have reliable results.”—Chris Westmeyer, President, Caring Advisor
The more variables you’re testing, the more traffic you need
Since MVT receives a much smaller proportion of the traffic (5% to 15%) than A/B testing (50%), MVT is best for sites with greater traffic. If you know how much traffic your site or specific page gets on average, you can use this sample size calculator to estimate what you need.
“Multivariate testing requires a great deal more traffic than A/B testing. Exactly how much traffic you need will depend on how many variants you are testing at once and the sample size you need to get significant results. Going back to statistics class, we can remember that it’s important to have the right sample size to get a significant result. If you’re doing A/B testing, you can use [a sample size calculator] to do the hard work for you. The amount of time you run your test depends on the amount of traffic you have — the more traffic, the more quickly you’ll reach the minimum sample size.”—Nicholas Tippins, Director of Digital Marketing, Beyond PhD Coaching
Should You Test in Regulated Industries?
Marketers in highly regulated industries, such as banking, healthcare, insurance, can rest assured that either form of testing can be conducted securely while also remaining compliant. In fact, most multivariate and A/B tests don’t require any Personally Identifiable Information (PII) in order to provide actionable results since it’s really based on conversions (i.e., did they sign up: yes or no?).
Testing is limited to content (not PII)
One advantage of either testing methodology is that it’s limited to content, messaging, and design, and it has nothing to do with a user’s PII. So whether you’re conducting an A/B or multivariate test, you can be sure that you’re in compliance.
“After analyzing the test results, you can make an informed decision and promote the version that performed the best. The idea is not to put all of your eggs in one basket when marketing to a large audience. Both A/B testing and multivariate testing are great for regulated industries due to limitations placed on content and messaging.”—Lauren Kennedy, Founder and Chief Marketing Nerd, Coastal Consulting
Your digital products should be tested
Regardless of your industry, you should always consider testing your digital products. This is to ensure you’re offering your clients or patients exactly what they want and need, and optimizing them as you go.
“Whether you do multivariate or A/B testing depends on which stage your product is in its life cycle. It doesn’t have anything to do with your industry. Therefore, your product might need multivariate testing or A/B testing regardless of whether you’re in the healthcare, insurance, or finance industry.”—Sally Stevens, Co-Founder and Head of Marketing, FastPeopleSearch.io
6 Next steps for multivariate and A/B testing
Conducting multivariate and A/B tests doesn’t have to be overwhelming; in fact, it should make your job as a marketer that much easier. Ultimately, your testing decision should come down to resources, including time and budget. If you still have questions about either testing method or you’re ready to get started, contact us today to book a demo with Kameleoon.
