A/B testing: overcoming cognitive bias!
Humans don’t operate by the rules of statistical reasoning. We’re flawed, irrational and subjective. Why?
Because we’re influenced by many different cognitive biases. While you can go through life without giving these a second thought, when you’re running an A/B test they can be a real hindrance. After all, A/B testing is a scientific experiment that requires objectivity as well as actionable, measurable and pertinent data, meaning there’s no room for guesswork or hunches.
1 A/B testing, a formidable weapon against cognitive bias
Cognitive bias, the nemesis of the marketer
Cognitive biases are opinions, beliefs or preferences that influence your ability to reason and to remember or evaluate information objectively. Whether you like it or not, your choices when optimizing a web page can be skewed by certain cognitive biases.
This influence that you unintentionally exercise over your decisions can impact your visitors’ experience. For example:
- Your choices in terms of UX may considerably impact your bounce rate
- Your content may not address your visitors’ actual questions
- Your call-to-actions or any other inducements to convert may be poorly positioned or incomprehensible to anyone but you.
In each of these cases, you’ve made assumptions that are based on your convictions and that don’t necessarily correspond to your visitors’ expectations. This can have a detrimental effect on the overall conversion rate on your website.
A/B testing, the solution to overcome cognitive bias
To optimize your website, you must rely on actionable, measurable and objective data. This is why A/B testing is the best solution for making decisions that set aside cognitive bias.
With A/B testing, you submit different variations of your website content (from whole pages to navigation details) to visitors. Analyzing the results of these tests then shows you which version was most appropriate to your visitors. For example, you might have thought that your CTA was very visible in red, but your visitors actually preferred it in green. It is vital that you accept these results, since they are reliable and reflect visitors’ desires, rather than relying on your, potentially biased, opinion.
Nonetheless, even when setting-up A/B tests, new cognitive biases may creep into your method.
2 The 9 cognitive biases that may alter your A/B testing process
In terms of A/B testing, it is essential to follow a rigorous methodology and leave aside the cognitive biases that could lead you to draw the wrong conclusions.
Cognitive bias 1: Finding (non-existent) relationships between events
With the illusory (or misleading) correlation bias, you are likely to create a relationship between two unrelated events.
You could, for example, assume that it’s the new pink color of your CTA that increased conversions for your variation, whereas in reality people are rushing to buy your products because of another factor such as the weather.
Sometimes, our brain jumps to hasty conclusions, so you must scrupulously analyze your results to ensure you don’t misinterpret the data.
Cognitive bias 2: Not understanding sample size
For a test to be reliable, you need to have a large enough sample size. With cognitive biases, our brain struggles to accurately estimate the size of the sample and underestimates the underlying variations of small samples.
Let’s take an example from Daniel Kahneman's excellent book, Thinking, Fast and Slow: there are two hospitals in a city. 45 children are born every day in the largest one, and 15 in the smallest. Let’s imagine that there is exactly a 50% chance of having a boy. The exact percentage varies every day and sometimes there are more boys than girls, sometimes fewer.
Over a year, each hospital recorded the days on which more than 60% of newborns were boys. Which hospital do you think recorded the most?
- The large hospital
- The small hospital
- Both were more or less identical (give or take 5%).
During this survey, 56% of respondents chose answer 3, 22% chose answer 1 and 22% chose answer 2. The correct response, however, is the small hospital. With a smaller sample to work with the variations are automatically greater.
This is the same with your test, meaning it is essential that you understand this concept of sample size. However, as the question above shows that our brain often forgets it when spontaneously solving problems. A small sample can give you ideas for tests. Don’t draw any conclusions from them though, as the results won’t have any statistical value. Now you know the risks: the smaller the sample, the less reliable it is.
Cognitive bias 3: Interpreting information to confirm a preconceived idea
It is also important to be aware of the confirmation bias, which is when you focus on a particular piece of information simply to confirm a preconceived idea.
We can also add the congruence bias into the mix. This is when you only test or challenge what YOU think is the problem, without even considering an alternative: the result of this is a subjective test.
For example, if you think that the color red increases conversions, your brain is only going to focus on information that seems to confirm this idea.
When you think you’re right and the data seems to confirm your idea, ask yourself if this proves your hypothesis objectively or if there are other factors that might have influenced this increase in conversions. You must not test to prove you’re right but rather to learn and to challenge all assumptions. In fact, a test that fails and proves you wrong is equally relevant.
Cognitive bias 4: Identifying imaginary patterns in your data
The clustering illusion is when you see coincidences in data picked randomly or think that random events follow a specific logic. Just because you are noticing similarities between events doesn’t mean that they are necessarily related.
For example, if you toss a coin 39 times and you get tails 39 times, what is the probability that you will get tails again on the 40th toss?
It’s 50% because it is simply a new toss that is independent of the others, which all had a 50% probability too. This is also what we call the gambler fallacy, which consists in believing that if we have obtained a similar result several times during a random experiment, it will happen again.
Set your intuition aside and be wary of your brain, which is wired to try to find links between all the information that it processes. Don’t stop a test because “you think” you’ve spotted a trend: you may have over-interpreted your results.
Cognitive bias 5: Thinking that what is in front of you is enough to draw conclusions
Let’s take another example: A bat and a ball together cost $1.10. The bat cost $1.00 more than the ball. How much does the ball cost?
The answer is not $0.10. In fact, if the bat costs $1.00 more, then the ball costs $0.05 and the bat costs $1.05 (i.e. $1.05 + $0.05 = $1.10).
Your brain is designed to draw conclusions with the information it possesses. Except sometimes (perhaps even often) it is too quick off the mark. Just because you seem to have all the pieces of a jigsaw in front of you doesn’t mean you’re ready to put them all together and finish it.
Cognitive bias 6: Basing all reasoning on the first information received
The anchoring bias is when we often allocate most importance to the first piece of information we’re given.
Here’s an example from a study conducted by Fritz Strack and Thomas Mussweiler, who asked two groups of people about Gandhi’s age when he died.
The first group was asked if he died before or after the age of 9.
The second group was asked if he died before or after the age of 140.
Both answers were pretty obvious. However, what is interesting is the answers given by both groups when they were asked to guess Gandhi’s age when he died. In the first group, the average answer was 50; in the second, it was 67.
Why was there such a difference? Because the respondents were influenced by the first question they’d been asked.
Our brain is designed to look for links between the information at its disposal (i.e. what’s in front of our eyes), and to draw conclusions from this. We don’t have the reflex to step back and consider the entire situation, either.
Be very careful with the results provided by your tests. When they confirm what you thought, don’t move to the next stage. Stop, take a step back, and check your test—or check that you are not just in the process of confirming your opinions via a biased test. Run a second test if necessary.
Cognitive bias 7: Abandoning reason as soon as your emotions are involved
This is probably the hardest to do. You go all out working on a design, you spend hours on it and are even really proud of your work. You run an A/B test to compare performance with the previous design… And it’s a complete flop…
It’s tough, but if you’re running A/B tests, it’s so you don’t lose money on decisions based on emotions or hunches. You have to give your visitors what they want. Forget your pride, go back to the hypothesis that led you to this new design and use the data obtained to try to understand what didn’t work.
Cognitive bias 8: Not second guessing yourself
Functional fixedness is when you’re stuck in linear thinking. When you see an iron, you automatically associate it with clothing (linear thinking). You don’t think about using it as a toaster. That is what we call lateral thinking or thinking outside the box.
You can experiment with lateral thinking by repeatedly asking “why” as soon as something seems obvious or surprising. You’ll end up getting back to the source by confirming (or disproving) your hypothesis. Other ideas for triggering your lateral thinking:
- Is there a problem? Try to solve its opposite
- Think of the most obvious and stupid solution
- Break your problem down into a series of smaller more precise problems
- Don’t be content with a single solution
- Change your perspective. How would you address this problem if you were an engineer, a scientist, or even an (absolute) beginner?
Cognitive bias 9: Overestimating the unanimity of your opinion
“Everyone hates pop-ins”. Well, YOU hate them. They generally disturb customers, but if used well they can also increase conversions.
This bias is known as the false consensus effect: Sometimes we can firmly believe something and think that everyone else has the same idea when they don’t. So, when you think that something can’t be improved or that a change won’t work because “everyone” hates it, stop right there and question your assumptions.
Test the elements that will have the most impact, but also test what already seems to be working well. You can always improve.
To run a conclusive A/B test, it is vital to focus on the figures, analyze the results objectively and overcome cognitive bias. As you’ve seen, it can be difficult to remain objective in all situations, so don’t hesitate to test and re-test!
