Web experimentation is still relatively niche — in part due to the volume of web traffic needed to conduct experiments. But what's the excuse for those with high traffic or product usage and an appetite to grow? 

Practitioners cite reasons such as inertia and locking horns with leaders over experimentation, as reasons why buy-in is hard. Yet a 2023 study found that: 

• 44% of business leaders want help from data to make better decisions.
• 41% want data to help them reduce risk. 
• 39% want data to help them make faster decisions. 
• 37% want to use data to make more money. 

The same study noted that most (85%) business leaders have “suffered from decision distress – regretting, feeling guilty about, or questioning a decision they made in the past year.” 

So, it seems there’s an apparent want and need from business leaders for experimentation. The benefits derived from doing so would relieve them of decision regret, so why is it still so hard to convince executives to adopt the practice?

Often, blame is attributed to “having the wrong culture.” This intangible concept leaves people grappling with how to solve the problem. So, here’s a fix-your-culture free post that provides practical things you can do to get the buy-in you need. 

A mismatch between needs and benefits 

There’s a spectrum of experimentation benefits, from the more idealistic ‘customer learnings’ to hard benefits such as increasing revenue. But without understanding who we’re speaking to and their challenges, we might offer more problems than solutions. For example, leaning too heavily on experimentation to “learn about customers” might put off an executive who needs to make a specific decision on pricing. We know that 70% of people have given up on making a decision because the data was overwhelming. So, in this instance, presenting experimentation as being able to “inform specific decisions” would resonate better with the executive. 

From a practitioner's perspective, presenting why a business should experiment can be challenging because it involves holding back. What most enthusiastic folk do is list all of the benefits and metrics experimentation could help with, in an attempt to ‘wow’ their executive audience — rather than focusing on one potential benefit to a specific challenge an individual faces. While we get not wanting to reduce experimentation to a decision-making factory where the ability to explore transformative ideas is restricted, being fluid in how we present the practice to match different use cases at any given moment can help gather initial momentum. 

Deborah O'Malley suggests mimicking the language executives use to make the benefits easier to understand:

If we face this mindset, we can focus on promoting the upsides they don’t understand, e.g., providing examples of how experimentation can increase the speed of project execution or share horror stories of the perils of waterfall projects. No one wants their name attached to a failed project. Leaders understand this, and a solution that extinguishes this fear can be desirable. 

But while we’re framing benefits in terms of loss aversion or avoiding bad outcomes, it’s important to ensure we aren’t shaming anyone. 

Now we’ve established that throwing every potential benefit at executives hinders, not helps, in the quest to get buy-in, let's cover some practical alternatives.

Practical tips for mastering buy-in

Identify a promising test bed and budget 

Identifying a promising and easy test bed can help you show executives how experimentation works in the real world. Elise Connors shared a great example of how she does this by tapping into an existing team's budget.

Align with specific motivations and goals 

Once you’ve identified a suitable team to trial experimentation, you need to understand what matters to them and their bosses. For example, the advertising team might not be as motivated to understand why customers churn as they are about driving more effective spending due to budget pressures. However, the CMO will have different goals, KPIs, and pressures/motivations. 

Deborah O'Malley provides us with a handy list of questions to help identify what’s important to executives whose buy-in you’re ultimately after.

As Deborah said, it’s essential to align with the goals and KPIs that matter to decision-makers, but how can you do this if your test bed results don’t answer the meaty problems management is facing? Michael Morgan has a suggestion to overcome this; ‘Provide proper historical use cases oriented to tackle "like problems" faced in today's moments. It's always easier to orient to data vs. opinion in moments of 'what should we do.' So, for example, show how experimentation helped the advertising team spend their budget more effectively and how this can be applied more broadly to drive company profitability. 

Justin Früh goes beyond just suggesting you use the same language:

Justin’s example reminds us of the path of least resistance. Make it stupidly easy for people to see what you want to show them. If it means presenting your data in tools they use vs. making them sign up for something new, the former will always be more successful. 

Influence stakeholders; a formulaic approach

Use a formulaic approach to document stakeholders' decision-making power, what motivates them, and whether they are supporters or detractors to help plan how you can influence them.

A stakeholder map can help you to set out the who, where, and relationships between key people in your organization. Then, use a Power-Interest Matrix to understand who you must influence and how. It’s worth mapping out each stakeholder's attitude toward experimentation to tailor your approach and messaging. 

Image

A great place to start building support is among the people who already “get” experimentation. It’s also worth noting here that despite this systematic approach, we’re dealing with humans who, like a free lunch; as Rich Page suggests, “Try to find A/B testing champions in your business who understand the great impact of it and get them to help influence senior leaders to do more A/B testing. Finding champions at the senior level like managers or directors, will help even better. You could even incentivize these champions by offering to buy them lunch or coffee regularly.” 

Talk about money 

The final suggestion, presenting revenue calculations based on experiments, is often debated. Whichever side of the debate you side with, there’s no doubt that budget and resource allocation are influenced by a department's impact on the bottom line. 

Rich Page says you should present the value you’ve achieved. The following metrics are particularly valuable when trying to scale the experimentation practice. 
 

Get ready to influence 

We’ve managed to cover how to get buy-in for experimentation without relying on the impractical advice of needing to create a ‘culture of experimentation.’ By focusing on practical approaches to influencing individual decision-makers, you can actively move towards achieving buy-in for your work. 

If you’d like to learn more about mastering executive buy-in, watch the recording of HiPPO Love: How to Get Exec Buy-in in 2024, where you will; 

• Hear how Shagun Aulakh, Director of Experimentation at American Express, has faced a lack of buy-in across her career and how she overcame it.
• Understand how experimentation can solve goals execs care about from André Morys, founder of Growth Marketing Summit and konversionsKRAFT.
• Why Ruben de Boer of Online Dialogue thinks we should apply the experimenter skills of customer research vs. sharing our test performance to garner buy-in.

Massive thanks to the following experimentation experts who provided their thoughts for this article; 

1. Lex Guest, Chief Marketing Officer at Aparito 
2. Deborah O'Malley, Founder of GuessTheTest
3. Florent Buisson, Experimentation Principal at Cars.com  
4. Merritt Aho, Digital Analytics at Breeze Airways
5. Elise Connors, Chief Consultant at Kinnective Digital Strategies
6. Ekaterina Eby, Director, Marketing Strategy & Analytics at Time4Learning 
7. Michael Morgan, Manager, Conversion Rate Optimization at Sumo Logic
8. Justin Früh, Director of Product Strategy & Research at Kameleoon 
9. Rich Page Conversion Rate Optimisation Expert at Rich Page: Website Optimizer