10 Executives share what they really think about experimentation
The relationship between experimentation teams and the C-suite is often presented as “us” and “them” — portraying executives as lacking interest and appreciation for experimentation. But is this a true reflection of today's C-suite? We decided to speak to ten company executives to get their take on experimentation and explore how they view the practice.
Is experimentation valuable, or is it a waste of time and resources?
We know the way to most executives' hearts is to talk about revenue and the bottom line. So, do the executives we spoke to consider experimentation an overall net gain or a resource drain? And, if they see it as valuable, can they move past the money and recognize the non-financial benefits, too?
A way to drive exponential growth & incremental gains
While it’s often hard to attribute and forecast revenue gains from individual experiments, there is often an impact on KPIs. Typically, gains can be categorized into two buckets: exponential gains and small incremental growth, as Michelle recognizes:
Experimentation protects revenue through efficiency & better decisions
Several executives saw experimentation as a way to preserve resources by helping teams make better decisions and saving time from being spent on poor choices, as Charity discusses.
Another interesting framing shared by Joosep is how experimentation can help you “protect revenue."
Experimentation drives innovation
Several leaders see experimentation as being critical for driving innovation, whether that’s overall business innovation or specifically within product development or marketing. It’s worth noting that they see the innovations derived from experimentation as an overall gain, as Campbell says.
Andrew explains how he sees experimentation as a method for “intelligent investment."
Setting the company direction when it comes to innovation and growth is a hefty decision, which is likely why some executives see experimentation as a strategic approach to these areas, as Matt describes.
On the flip side, if you aren’t using experimentation to innovate, your company risks stagnating. This innovation can be driven not only in marketing or product development but processes as well, as Tim explains.
Yes, but data validity determines the value
The overwhelming response is positive about the benefits of experimentation, but a couple of executives rightly caveat their views based on the quality of experimentation. Evaldas shares why this is the case with a great example.
A/B testing isn’t the only method of experimentation, and executives must understand that this method isn’t always suitable. It’s also crucial that teams using experimentation do so with a rigorous process and the right impetus. Aleksandar explains why this is the case.
But if you're an established company, you can't make changes as you used to. Every big change that upsets the status quo is a risk that can affect the bottom line, so caution and formal testing is paramount. In the end, experimentation is important, but the form depends on where your company is. If you're a startup, it's better to iterate quickly and listen to feedback, and if you're an established company, you can be more data-focused.
What’s your initial reaction to feature testing & website experimentation?
All of the executives we spoke to appreciate the value of experimentation for various reasons, but we wanted to delve a little deeper into their opinions. We asked those interviewed about their immediate thoughts when they heard phrases like ‘feature testing’ or ‘website experimentation.’ Their responses show an exciting range of perspectives and go deeper into why they, as leaders, hold certain opinions about experimentation.
Tools for growth
Several executives we spoke with focus on how web and feature experimentation are tools or processes for growth. Joosep shared his view.
This view was shared by Matt, too.
A user-centric approach to product development & marketing
For marketing leaders, web or feature testing might conjure up thoughts around some of their main challenges, such as understanding the market and creating strategies that truly resonate with the right audiences. However, the benefits of experimentation go beyond just creating better products or marketing, as Charity shares.
Rolling out your new features to a smaller cohort is beneficial because it will uncover ways to improve them and how the market will actually implement them in real-life scenarios. It will even help you determine how to communicate the offering to your audience. This approach helps avoid a common risk companies run when heavily investing in new features that fall short, often due to the most simplistic components.
Absorbing real-time insights from the beta-testing group not only benefits your company but also the end user because, this way, the product is, in essence, built for and by them. This user-centric method helps build trust with your clients, fosters loyalty, and instantly creates a group of advocates for your brand.
Integral to product development & continuous improvement
In a world of agile product development, one-off rounds of customer research don’t cut it. Instead, constant feedback loops are needed to inform the product development process, with feature experimentation offering a method for achieving this for leaders like Tim.
Andrew shares a similar reaction.
Overly myopic
For leaders who have a good understanding of experimentation, terms such as web testing may be perceived as a simplified view of what experimentation practices truly encompass, as Michelle felt.
As an over-simplified example, “web experimentation” evokes the concept of button color testing in my mind, which, of course, may not always be the case. Great “web experimentation” can deliver much more, such as buyer journey experimentation to deliver on engagement, assist in providing visitors with what they are seeking, and improve performance against business objectives. Feature testing is an absolute must before rolling product enhancements out to general availability.
Results of experimentation
Finally, we asked executives to share the results of their experimentation efforts, whether good or bad. Below are some first-hand examples these leaders shared, which impacted their overall view of experimentation. So what struck them as a successful result or a noteworthy failure?
Learnings that drive demand
Experimentation can impact various metrics and improve a whole host of channels, experiences, and strategies. You can use the practice if you can form a hypothesis and apply an experimental method. What stood out for Michelle is how learnings from one test can apply to other business areas and the compounding nature of iterative tests.
The learnings were significant. We learned that we could serve a multi-touch interaction to continue the customer experience while people were in consumption mode, but early tests demonstrated pop-up experiences were blocked. We tested until we got the experience right, and the demand conversion metrics went up and up with every iteration! The output of this testing methodology was so successful, that the vendor we used to deploy it productized it, I learned years later.
Ecommerce checkout success that grows revenue
While many executives focus on revenue gains, it’s essential to recognize the savings from avoiding “mistakes.” For Matt, the combination of the two makes for a notable case study.
Even when experimentation doesn't yield the desired results, it still significantly informs our future strategies and prevents unnecessary mistakes.
Andrew shares a similar story that recognizes the value of both winning and losing tests to the business.
A test-and-learn approach to problems
As already discussed, experimentation can take a number of different forms and apply in a host of situations. For some executives, an experimental mindset can apply to any problem or challenge, as Megan shares in this interesting example;
To test this, we offered a full-time "trainer" to read the SOPs to the new hires and take them through their role. When this person quit, they said they were trained very well but made other excuses for quitting. Finally, we narrowed down the variables and realized it was a reason not given: lack of community. The type of people applying for this role were often extroverted, so we hypothesized that they might not succeed because of the lack of social interaction. We restructured the role to include them in a community so they weren’t working solo all day.
We need more data to conclude whether our hypothesis was correct, but that would require this person to leave the position, and so far, they're very happy and well-supported within the team.
When experimentation goes wrong
It’s not all rosy success stories. As some of the executives mentioned already, experimentation relies on valid data and robust processes. If those aren’t in place due to a lack of tooling or knowledge gaps, it can lead to teams making poor decisions based on faulty assumptions.
Sometimes, it’s not the experiment itself but the lack of one which proves a valuable lesson. Evaldas shares his story.
Without any testing, we decided to follow the competition and update our website in the same direction. After one week, it became evident that people didn't understand our proposal, and we were losing product registrations as a result.
Experimentation: A valuable asset
This article debunks the trope that C-suite executives are dismissive of experimentation. The discussions above reveal an appreciation for experimentation as a driver of growth, innovation, and informed decision-making, as well as avoiding the cost of poor decisions.
While the financial advantages of experimentation were important, it’s clear that value is also derived from testing assumptions, learning from failures, and being user-centric.
Finally, leaders recognized the importance of robust tooling, processes, and valid data in order to run experiments that can be relied upon.
Thanks to the following executives for sharing their thoughts on this topic.
- Michelle Killebrew SVP, Chief Marketing Officer - Americas at NTT Ltd.
- Charity Hudnall, CMO at Vagaro
- Joosep Seitam, Co-Founder at Icecartel
- Campbell Tourgis, Executive Vice President & COO at Wainbee Limited
- Andrew Grella, CEO of Formen Inc.
- Tim Green, COO of TeamUp
- Matt Aird, Chief Technology Officer at Custom Neon
- Evaldas Mockus, VP of Growth at Omnisend
- Aleksandar Olic, VP of Marketing at CAKE.com
- Megan Cox, CEO of Genie Supply
