Web experimentation is still relatively niche — in part due to the volume of web traffic needed to conduct experiments. But what's the excuse for those with high traffic or product usage and an appetite to grow?

Practitioners cite reasons such as inertia and locking horns with leaders over experimentation, as reasons why buy-in is hard. Yet a 2023 study found that:

44% of business leaders want help from data to make better decisions.
41% want data to help them reduce risk.
39% want data to help them make faster decisions.
37% want to use data to make more money.

The same study noted that most (85%) business leaders have “suffered from decision distress – regretting, feeling guilty about, or questioning a decision they made in the past year.”

So, it seems there’s an apparent want and need from business leaders for experimentation. The benefits derived from doing so would relieve them of decision regret, so why is it still so hard to convince executives to adopt the practice?

Often, blame is attributed to “having the wrong culture.” This intangible concept leaves people grappling with how to solve the problem. So, here’s a fix-your-culture free post that provides practical things you can do to get the buy-in you need.

A mismatch between needs and benefits

There’s a spectrum of experimentation benefits, from the more idealistic ‘customer learnings’ to hard benefits such as increasing revenue. But without understanding who we’re speaking to and their challenges, we might offer more problems than solutions. For example, leaning too heavily on experimentation to “learn about customers” might put off an executive who needs to make a specific decision on pricing. We know that 70% of people have given up on making a decision because the data was overwhelming. So, in this instance, presenting experimentation as being able to “inform specific decisions” would resonate better with the executive.

From a practitioner's perspective, presenting why a business should experiment can be challenging because it involves holding back. What most enthusiastic folk do is list all of the benefits and metrics experimentation could help with, in an attempt to ‘wow’ their executive audience — rather than focusing on one potential benefit to a specific challenge an individual faces. While we get not wanting to reduce experimentation to a decision-making factory where the ability to explore transformative ideas is restricted, being fluid in how we present the practice to match different use cases at any given moment can help gather initial momentum.

Deborah O'Malley suggests mimicking the language executives use to make the benefits easier to understand:

Tap into the metrics that matter most to management, then communicate the value CRO delivers using their terms. Don't talk about conversions or lift. Talk in ways they'll understand and what will resonate with them most. Your job is to figure out what makes management tick and then present your data to them in this language. And, while revenue is likely very important to your company, it’s not always everything to management.

Deborah O'Malley

Founder of GuessTheTest

Another aspect that helps promote the most attractive benefits to the right executives is understanding their leadership style and level of understanding about experimentation. This can help us get into their mindset and understand what motivates them. As Florent Buisson says:

Today's senior leaders climbed the ranks before Internet and Email, and execution was the name of the game. Getting hundreds or thousands of employees across the country to execute the same plan at the same time was no small feat. Information was so slow to feed back that you couldn't really modify a project halfway through. The people who were promoted were those who could come up with a half-decent plan and then see it through no matter what. Now that they are senior leaders, they still reward and promote middle managers for getting something done and shipped. In such an environment, there are only downsides and no upsides to experimentation.

Florent Buisson

Experimentation Principal at Cars.com

If we face this mindset, we can focus on promoting the upsides they don’t understand, e.g., providing examples of how experimentation can increase the speed of project execution or share horror stories of the perils of waterfall projects. No one wants their name attached to a failed project. Leaders understand this, and a solution that extinguishes this fear can be desirable.

But while we’re framing benefits in terms of loss aversion or avoiding bad outcomes, it’s important to ensure we aren’t shaming anyone.

Avoid making people or their projects look bad.

We all know that the benefit of experimentation is being able to distinguish between crap initiatives and ones that move the needle, knowing that there are a lot more of the former. But when trying to go from a culture where people commit to projects without testing to a culture where people only commit to projects with testing, there's a real risk of becoming the enemy. People don't get excited (initially) about having projects they put blood, sweat, and tears into crapped-on (egos are often wrapped up in there, too).

There's a different entry point to a testing mindset that I call "mining for gold." It's the notion of testing different tactics to discover which ones work, and it tends to get people pretty excited. After they're excited about testing and after they've seen some ideas they were so certain were going to win but didn't, that's the time to challenge untested project commitments across the org.

Merritt Aho

Digital Analytics at Breeze Airways

Another option is to present experimentation as an independent arbiter. Lex Guest suggests that:

By framing experimentation through the lens of customer feedback, you aren't making the case for experimentation; your customers are, so the conversation becomes more objective than subjective. Based on our research, we understand that our customers would like X. Therefore, we should try Y, and the potential upside to leads/revenue could be as much as Z.

Lex Guest

Chief Marketing Officer, Aparito

Now we’ve established that throwing every potential benefit at executives hinders, not helps, in the quest to get buy-in, let's cover some practical alternatives.

Practical tips for mastering buy-in

Identify a promising test bed and budget

Identifying a promising and easy test bed can help you show executives how experimentation works in the real world. Elise Connors shared a great example of how she does this by tapping into an existing team's budget.

A good starting point is with advertising. For the clients I partner with, I usually work them up to a 10% testing budget. That budget is evaluated separately from the rest of their ad spend. The expectation is not to generate bottom line for the business. Instead, the goal of this 10% is to learn at least one new thing a month. This is a great starting point for building curiosity within the team. From there, larger experimentation “programs” are an easier “sell.”

Elise Connors

Chief Consultant at Kinnective Digital Strategies

Again, early on, we’re trying to demonstrate the benefits of experimentation, so don’t start with tests with a low likelihood of success. Instead, heed Ekaterina Eby’s advice.

Try to find a win that has the potential for high-impact but low-effort; document the process, be as transparent as possible, share learnings (what & the how), and over-communicate. Third-party validation works wonders.

Ekaterina Eby

Director, Marketing Strategy & Analytics at Time4L

Align with specific motivations and goals

Once you’ve identified a suitable team to trial experimentation, you need to understand what matters to them and their bosses. For example, the advertising team might not be as motivated to understand why customers churn as they are about driving more effective spending due to budget pressures. However, the CMO will have different goals, KPIs, and pressures/motivations.

Deborah O'Malley provides us with a handy list of questions to help identify what’s important to executives whose buy-in you’re ultimately after.

To measure and report the right KPIs, you must first be aligned with your company’s business objectives. To determine these goals, ask yourself and your management questions like:

1. What kind of data does my boss ask for again and again?
2. What metrics do executives communicate to the organization?
3. What metrics do stock analysts estimate and report on (for public companies)?
4. What kind of results do my bosses report to their bosses?
5. What is being tracked on management dashboards?

Assessing and answering these questions will enable you to present the data in a meaningful way that resonates most with management. When you "speak their language," you'll become aligned with the organization's business objectives and will more easily and effectively be able to communicate the value of experimentation.

Deborah O'Malley

Founder of GuessTheTest

As Deborah said, it’s essential to align with the goals and KPIs that matter to decision-makers, but how can you do this if your test bed results don’t answer the meaty problems management is facing? Michael Morgan has a suggestion to overcome this; ‘Provide proper historical use cases oriented to tackle "like problems" faced in today's moments. It's always easier to orient to data vs. opinion in moments of 'what should we do.' So, for example, show how experimentation helped the advertising team spend their budget more effectively and how this can be applied more broadly to drive company profitability.

Justin Früh goes beyond just suggesting you use the same language:

If you want C-suite attention, you need to use their tools. The data warehouse (BigQuery, Snowflake) with BI tools (Looker, Power BI, Qlik, Tableau) are used by senior managers such as CFOs. If you speak their language, you will have their attention.

Justin Früh

Director of Product Strategy & Research

Justin’s example reminds us of the path of least resistance. Make it stupidly easy for people to see what you want to show them. If it means presenting your data in tools they use vs. making them sign up for something new, the former will always be more successful.

Influence stakeholders; a formulaic approach

Use a formulaic approach to document stakeholders' decision-making power, what motivates them, and whether they are supporters or detractors to help plan how you can influence them.

A stakeholder map can help you to set out the who, where, and relationships between key people in your organization. Then, use a Power-Interest Matrix to understand who you must influence and how. It’s worth mapping out each stakeholder's attitude toward experimentation to tailor your approach and messaging.

A great place to start building support is among the people who already “get” experimentation. It’s also worth noting here that despite this systematic approach, we’re dealing with humans who, like a free lunch; as Rich Page suggests, “Try to find A/B testing champions in your business who understand the great impact of it and get them to help influence senior leaders to do more A/B testing. Finding champions at the senior level like managers or directors, will help even better. You could even incentivize these champions by offering to buy them lunch or coffee regularly.”

Talk about money

The final suggestion, presenting revenue calculations based on experiments, is often debated. Whichever side of the debate you side with, there’s no doubt that budget and resource allocation are influenced by a department's impact on the bottom line.

Rich Page says you should present the value you’ve achieved. The following metrics are particularly valuable when trying to scale the experimentation practice.

Show senior executives the likely ROI of running more A/B tests. Calculate the potential revenue impact from increasing your conversion rate faster with more A/B tests launched per month. You should also remind them of any particularly good previous A/B test results or if you have proof that your competitors are A/B testing.

Rich Page

Conversion Rate Optimisation Expert

Get ready to influence

We’ve managed to cover how to get buy-in for experimentation without relying on the impractical advice of needing to create a ‘culture of experimentation.’ By focusing on practical approaches to influencing individual decision-makers, you can actively move towards achieving buy-in for your work.

If you’d like to learn more about mastering executive buy-in, watch the recording of HiPPO Love: How to Get Exec Buy-in in 2024, where you will;

Hear how Shagun Aulakh, Director of Experimentation at American Express, has faced a lack of buy-in across her career and how she overcame it.
Understand how experimentation can solve goals execs care about from André Morys, founder of Growth Marketing Summit and konversionsKRAFT.
Why Ruben de Boer of Online Dialogue thinks we should apply the experimenter skills of customer research vs. sharing our test performance to garner buy-in.

Massive thanks to the following experimentation experts who provided their thoughts for this article;