How to build a high-performance culture of A/B testing

Experimentation and A/B testing are proven to accelerate growth and performance. Yet many brands struggle to create a culture of experimentation within their business. We interviewed Stephen Pavlovich of experimentation agency Conversion.com and Martin Harrison of Kameleoon to get their views on the subject and advice for digital marketers.

1 Why do companies need to create a culture of experimentation? What benefits will they see?

Stephen Pavlovich (SP): Experimentation is a way to make better decisions. It means changing from an opinion-led approach to an evidence-based method of deciding what you do - and then assessing whether decisions have actually delivered the promised results. It covers everything from customer experience to wider business challenges. Collecting evidence through experimentation allows people to quantify results and gives clarity on investment decisions. Every result gives insight into how you as a business can do a better job.

Martin Harrison (MH): Customer experience is now the key battleground and differentiator for brands, unless they want to fail by competing solely on price. Creating the experience that customers want means continually assessing their needs and improving your offering - and to do that over the long-term you need a culture of experimentation.

2 Generally, how well are UK-based brands doing when it comes to experimentation?

SP: There is a gap between those that are doing it well, and those that are lagging behind. While this isn’t exclusively the case, there are differences in how digital-first companies and more traditional brands perform. Research from the Go Group, which Conversion.com is part of, found that fast-growth companies have a much greater engagement from senior stakeholders in experimentation - essentially they are much more willing to test everything, from commercial models to smaller details.

MH: There are differences between the best performing and lagging brands. However, from what I’ve seen, the gaps are not necessarily between traditional and digital businesses - they are more about the people within them. A digital native in a traditional business, provided they have the support they need, can do great things. That means there are traditional brands that understand the importance of embracing experimentation, see the need to invest to drive growth and put plans in place to reap the benefits.

3 What are the obstacles to creating such an experimentation culture?

SP: I think the biggest challenge is a misconception on what experimentation actually is. A lot of people think A/B testing is solely changing button colors, but it should be much, much more. Brands need to look to experiment more widely, around business models, personalization, pricing and other key issues.

Additionally, there is what I’d describe as fear of transparency. Most companies are not used to operating in a way that all their decisions are open to scrutiny by the rest of the team, and can be challenged with data. This can be particularly uncomfortable in hierarchical organizations, which are built on managers making decisions without expecting those reporting to them to be able to question what they’ve done using data. There’s been a lot of work done on what makes a successful team, and psychological safety is a key factor - people need to trust that trying something reasonable that then fails is not a black mark against them.

MH: The idea of meritocracy versus autocracy is key one. People need to be happy to say my idea didn’t work, and how do we learn from that, rather than feeling they need to justify and defend their decisions to preserve their status.

4 Are skills gaps holding back the spread of experimentation?

MH: There are skills gaps, particularly inside brands. That’s where external support, such as agencies, can augment and accelerate learning and progress in order to deliver a wider impact. It is important to stress the distinction between tactical experimentation and bigger, more complicated experiments. You can gain value at a tactical level, but to really transform your business with a more strategic approach - that requires a level of maturity, skills and experience.

SP: The spread of skills in digital marketing is definitely improving. There are a lot more people around with the knowledge to successfully use analytics and create experimentation strategies. Five years ago I don’t think these skills were so widespread.

What’s interesting is that these skills are spreading bottom-up rather than top-down. So as people move forward in their careers I’d expect more use of experimentation at a strategic level. Adding to this, there is a growing body of evidence that companies doing experimentation well are out-pacing their rivals - that’s making brands sit up and realize they need to put the right skills and investment in place.

MH: Stephen raises a very important point. The fact that we’re seeing a ground-up push on skills is really exciting - with people empowered to make huge changes in traditional businesses. We are creating solid foundations that the industry can build on going forward, with the only caveat being that understanding is needed from the executive level in client businesses.

5 What processes do brands need to have in place for experimentation to succeed?

SP: Strong processes are critical to the success of the experimentation program. Ideally there’d be defined (and optimized) processes for generating and prioritizing ideas, for delivering experiments, and for analyzing and iterating on winning and losing concepts. It’s easy to do all of these processes – but it’s hard to do them well. Just like anyone can come up with an idea for an experiment – but knowing the right experiment to do next is critical.

MH: I’d agree that in many cases processes are not very mature. I know of organizations where they are only able to, or are comfortable, running a single test at once, though this is lessening. What I have seen more recently is smaller companies upskilling very quickly, particularly in retail where competition is cut-throat. As well as processes, companies need the right technology platform to help them become more mature.

6 What advice do you have for teams looking to justify experimentation investment?

SP: Experimentation is one of the few activities that can easily justify its investment – the return is clear to see. To get greater investment, I’d recommend taking either a push or a pull approach.

Either you “push” your results to other parts of the business and senior stakeholders – getting greater visibility for the experiments you’re running and the impact you’re having. If you’re showing strong uplift, you’d hope to get greater budget.

But many get better results with a “pull” approach. Instead of trying to make the C-suite interested in the latest landing page test, find out what they care about first. What are the choices they’re making, and how can experimentation help them?

MH: Big experiments delivering big impacts require skilled people and sophisticated technology if you are going to create changes that excite an executive level audience. So don't be afraid to bring in the skills you require to help accelerate the process of structured experimentation programs that deliver an amazing ROI. You’re driving success that will then help justify even more investment and grow your teams and their processes internally.

7 Finally, where should teams start?

SP: Start by defining the goal – what are you trying to achieve? Then identify how you’re going to measure it. With the goals and KPIs agreed, you can define the key audiences and areas that contribute towards the KPIs – and gather data on volume and performance. Next, identify the levers – the positive and negative factors that drive a user’s behaviour. With this in place, you can start a structured experimentation program – validating the levers, and doubling down on the ones that work.

MH: Start with the data. Understand it and identify the moments of truth, the micro conversions, and not only on a basket page. You’ll see where failures in the customer journey are impacting revenues. giving you clarity in where to build different experiences through experimentation. Few clients have an understanding of the potential impact, where I’ve seen double digit revenue uplift with big experiments that drive real, measurable value to the business.

Kameleoon and Conversion.com held a joint event to explore how brands can build their own experimentation culture. Find out more from experts, including Facebook and TUI, in our event report blog.