7 Mistakes Most Beginners Make When A/B Testing

A/B testing can be extremely beneficial for all your marketing and design decisions. Knowing exactly what works, what doesn’t, testing before implementing anything so you don’t lose money. You're most likely already doing split testing.

A staggering number of people doing split tests get imaginary results. How come? Because you can be mistaken in so many different ways, and because maths are involved. In this article, we’ll cover about every Split Testing mistakes you could make, and how to avoid them.

1 You start doing Split Testing with complicated tests

For your first tests ever, start simple. Being successful at Split Testing is all about process. So it’s important that you first go through the motions.

Choose obvious and simple sticking points for your first split tests

See how theory computes with reality, what works for you and what doesn’t. Where you have problems, be it in the implementation of the tests, coming up with ideas, analyzing the results, etc… Think about how you’ll scale your testing practice, or if you’ll need new hires for example.  Starting with A/B Testing is a lot like starting weight training seriously. You don’t start with your maximum charge and complicated exercises.

Exact same thing with Split Testing. You start simple, focus all your attention on each step of the process, set up safeguards and adjust as you go so you don’t have to worry about it later. Another benefit with starting with simple tests is that you’ll get quick wins.

Getting bummed out when your first tests fail (and most do, even when done by the best experts out there) is often why people give up or struggle to convince their boss/colleagues that split testing is indeed worth the time and investment.

Communicate with quick wins split tests

Starting with quick wins allows you to create momentum and rally people to the practice inside your team/company. So you got the message, doing complicated tests right off the bat could kill your efforts in the egg. You could be overwhelmed, get fake results and get discouraged. Here are a couple examples of things you could test to start with:

• Test copy on your offers, product pages, landing pages (Make it focused on benefits not features, be sure that what you mean is crystal clear)
• Removing distractions on key pages (is this slider really necessary, or all these extra buttons?)
• Test the hero images of your pages to see if they are catchy enough and really fit to the content your visitors expect

2 You don’t have a hypothesis for each split test

Every test idea should be based on data, articulated through an informed hypothesis with an underlying theory.

If you’re not doing this, you’re mostly shooting in the dark without even knowing in what direction the target is. Not having a hypothesis (or a flawed one) is one of the most common reason why split tests fail. Let’s take our initial statement apart piece by piece to understand what this means.

Every split test idea should be based on data

Be it quantitative or qualitative, every test is induced by an analysis of your website data and the identification of a problem or element to improve. Gut feelings or “I read that…” won’t do. Here are several sources to build your hypothesis on:

• Analytics
• Heatmaps
• Surveys
• Interviews
• Usability tests
• Heuristic analysis

Put yourself in the shoes of your customers while Split testing

Why didn’t they do what you wanted them to? What was their intent at this stage? What would make you leave/not do what was expected?

Example:“I think that if I’m a customer, I’d be more inclined to go through with this step if the form had less fields”. Don’t skip this. When you’re deep in your CRO practice, you sometimes tend to forget about your users. You focus on numbers and design. It’s paramount to take a step back. The difference between a bad split testing hypothesis and a great one could be staring at you in the face. Let’s not forget that we are beings doted with empathy.We’re humans creating value for humans.

Formulate informed split test hypothesis

With data and a theory, you can now craft your split test hypothesis.

You can use this format or something along those lines: By {making this change}, {KPI A, B, and thus primary metric} will improve {slightly / noticeably / greatly} because {reasons (data, theory, …)}.

Working this way will not only improve the quality of your tests, but also your global conversion optimization efforts. Good, you are now testing to confirm or disprove a hypothesis, not shooting from the hip hoping to hit something eventually.

3 You don’t have a Split Testing roadmap

If you want to succeed at A/B Testing, and more largely at Conversion Rate Optimization, you need 2 essential elements: a roadmap, and a process.

1. Why do you need a split testing roadmap?

A roadmap will help you test what matters and work toward a clear end goal. It’ll guide your efforts and prevent you from doing aimless testing. In your roadmap, you should have:

• Business goals: the reasons you have a website. Be concise, simple, realistic.
• Website goals: how will you achieve these business goals through your site. What are your priorities?
• What are your most popular pages? Which ones have the highest potential for improvement?
• What does your conversion funnel look like, step by step? Where are the friction points?
• Key metrics: how will you measure success?
• A North Star: What’s the one metric—correlated with Customer satisfaction, that if you focus exclusively your efforts on will guarantee your success? (Ex: Facebook=Daily Active Users, AirBnb= Nights Booked, Ebay=gross merchandise volume)

Put all these down and make sure everyone in your team/company is on the same page and can get behind them.

2. Why structuring your split testing with a process?

It’s BECAUSE you have a process that you’ll be able to be creative. Split Testing is a science experiment. You need to be rigorous, have a set of parameters making sure what you’re doing is measurable, repeatable and accurate. It’s easy to fail. So you need safeguards. Steps that you will go through each and every time without having to think. No need to reinvent the wheel every time.

You need a process so you can focus on crafting the best hypothesis possible and optimize learning. The details of your process will be specific to you and your company. But it will look something like that:

1. Measure
2. Formulate hypothesis
3. Prioritize ideas
4. Test
5. Learn
6. Communicate
7. Repeat

When you feel like you lost your focus, use Brian Balfour question: “What is the highest-impact thing I can work on right now given my limited resources, whether that’s people, time, or money?”

4 You don’t prioritize your split tests

With a process, a roadmap and the development of a testing culture inside your company, you’ll have a list of test ideas longer than your arm.

You need a system to prioritize tests, to stay on top of things and make sure you do what matters. And to not waste time and money on stupid tests.

Once you're confident in your ability to Split Test, focus on tests with the most impact/potential. Target your pages with the most traffic and influence. You can use something like the PIE Framework to rate your tests so you know which one to do first. Each test will be rated following three criteria:

1. Potential gain (../10): How much room for improvement is there on this(these) page(s)?
    
2. Importance (../10): How valuable is the traffic on this(these) page(s)?
    
3. Ease of implementation (../10): How easy will this test be to implement on your site?

Average the three for each test, and you’ll have a backlog of ideas ranked objectively.

5 You don’t optimize for the right KPIs

There are two types of conversions. Micro and macro. You should measure both, and optimize for the macro conversions.

How are micro conversions useful?

A micro conversion is a step (click, social share, newsletter subscription, add to cart, …) on the path leading to a macro conversion, which is an outcome that impacts your bottom-line (check out, free trial, …), in other words, the main conversion goals of your website.

​

Why is it important that you know the difference? Because you need to make sure that you measure both for every test, but that you don’t optimize for micro conversions only.

There are two types of micro conversions according to the Nielsen Norman Group:

1. Process Milestones are conversions that represent linear movement toward a primary macro conversion. Monitoring these will help you define the steps where UX improvements are most needed.
    
2. Secondary Actions are not the primary goals of the site, but they are desirable actions that are indicators of potential future macro conversions.

Measuring micro conversions allows you to know where your friction points are and help you paint a holistic picture of your entire conversion funnel.

Why micro conversions are not enough?

But you shouldn’t optimize for them. You want to set your test goals as close to revenue possible. You could get more traffic to your landing page through a given variation of your homepage, but have fewer form completions even though more people arrived on it.

So if you were optimizing only for the micro conversion of arriving on the landing page, you would have lost money. Track micro conversion, but don’t optimize solely for them. And before starting any test, go back and make sure you measure everything that matters.

6 You ignore small lifts in your split tests

"Go big or go home.” It’s true, as we’ve just said, that you should focus your tests on high impact changes first.

What you won’t see us say anywhere, though, is “if your test results in a small gain, drop it and move on.” Why? Because maths, that’s why. Let’s take a quick example. If a test give your variation winning with 5% more conversions, and each month you get similar results. That’s an 80% improvement over a year. How’s that for small! Also, the more you test, the more you'll improve your website, the less you’ll have big results.

Don’t be sad if you just get small lifts. It could mean your website is good. It’s pretty rare to get big lifts on “normal” site. Don’t be disheartened by small gains; it’ll pay off big time over time.

7You’re not testing at all times

Every day spent without an experiment running is wasted. Opportunities missed.

Why are you testing? Because you want to learn more about your visitors, make informed decisions, get more customers and earn more money. When wouldn’t you want all that? Never? So as they say, Always Be Testing. Testing properly takes time, you better not lose any.

“There’s a way to do it better—find it”, T.Edison.

How you say? Test, test, test! And when in doubt, test again!