16 PLG leaders on what separates good from great companies when it comes to experimentation

We asked 16 top product-led growth leaders what separates a good from a great company regarding experimentation.

Three main themes emerged regarding the companies that excel in experimentation and drive growth: 

• Clear objectives for experimentation.
• A solid process for running experiments.
• A culture of data-driven decisions.  

However, there is some nuance to the above. So, let’s explore what our 16 product leaders had to say and how you can apply these points to your experimentation program.

Walk before you run

Most of the following advice is aimed at teams who have already implemented experimentation to some extent.

If you are yet to do so, Kevin Anderson, Product Manager Experimentation at Vista, has some advice on getting started. 

“Before you can get great at experimentation, you need to start. I recommend starting by testing the changes you are already making to your product and measuring the impact on key metrics," says Kevin.

"Forget about doing more user research, writing better hypotheses, and prioritising new ideas. Start experimenting with the next feature on your backlog. You will quickly learn that some features are making an impact, but many are not. Some even have a negative impact on customer and business outcomes. Use that evidence to become smarter in shaping your roadmap and product." 

1 Clear experimentation objectives 

While many of the product thought leaders agreed that clear objectives were essential to reaching a ‘great’ level of experimentation, their opinions differed on the objective itself.

This perhaps reflects what’s important to the organizations they work with, the type of product, or their north star metric. 

Answer your company’s biggest problems

Patrick Buffum, Product Manager, Optimization at GoBankingRates, suggests that good to great diverges when it comes to the program's level of influence over the organization. 

Patrick goes on to say, "if you can ensure that your tests also solve your company’s biggest problems/answer its questions, then experimentation has earned a seat at the table during the most important discussions.” 

Ankit Goswami, Global Product Lead at Expedia, provides an additional point that “great” experimentation programs meet not only the company, but the customer’s needs too.

“Great experiment provides not only useful information and insights but also generates a significant impact on key business metrics," says Ankit.

Objectives should go beyond testing marketing tactics 

May Chin, Principal Product Manager - Growth Experimentation & Analytics at ZALORA explains the level of integration needed within a business if you want to use experimentation to its fullest potential. 

“Great experimentation is when A/B testing is an intrinsic part of the overall decision-making process, where everything from branding to P&L is informed by experimentation," says May.

"At the end of the day, experimentation is simply one of many R&D methods to better inform decisions and understand the potential risks around them."

Whatever the goal, it needs to be measurable 

Ben Carter, Global Head of Digital Product Management at IG, raises an important point, no matter what the goal or objective, it needs to be measurable. 

“Experimentation is fundamental to creating great products, but in the rush to experiment, it's easy to forget that the goal is what we learn from experimentation, not the experiment itself," says Ben.

Meaningful insights that lead to follow-up actions 

Results are good, but for great experimentation to occur, your experiments should give you insights and learnings you can act on. Janna Bastow, CEO & Co-founder at ProdPad, explains below. 

Good experiments might get you somewhere...though they might also lack a clear goal or have dodgy data and may not control for confounding factors. They may not give you meaningful insights or drive product development," says Janna.

She goes on to say, "Great experiments, on the other hand, will really shine a light on the problem! These experiments have a clear objective, are designed well to eliminate bias, have solid data, use proper analysis, give you actionable insights, and lead to follow-up actions that advance your product. In a nutshell, great experiments are well crafted, give you valuable direction, and push your product forward.” 

The objective isn’t to find winning tests 

For Jonathan Callahan, Senior Growth Product Manager, Games at Unity, the objective of running tests shouldn’t be to find winners.

“The big question is, are you prioritizing running winning experiments (which I see happen a lot), or are you prioritizing moving the business forward? If you are prioritizing winning experiments, you can get several years into a program and have a user experience and conversion rate that is largely unchanged," explains Jonathan.

2  Follow a rigorous process

While many good experimentation programs have a process in place, great programs have an elevated level of rigor that removes doubts about any test results. 

Rigour doesn’t mean rigid

Bhavik Patel, Founder & Lead Product Strategist at CAUSL says what separates good and great experimentation programs is, "Discipline in experimentation design. Teams that run great experiments are disciplined when it comes to planning experiments. They design each experiment meticulously." 

Not everyone agrees that a rigorous process is what separates good from great. For Henning Heinrich, Group Product Manager, Product Growth at Hootsuite, velocity is more important.

Having trust in results is key 

If your data isn’t accurate, experimentation programs can quickly lose support. What’s the point in running experiments if you can’t rely on the results?

Anthony Murphy, Founder at Product Pathways, goes on to specify.

Kalki Gillespie, Senior Product Manager - Experimentation at NerdWallet, adds what his version of a great process involves.

He goes on to say, "And finally, experiments are aligned and sequenced well so that after a set of tests are done, the business is informed about the grand hypothesis that is implicit in the product roadmap. A roadmap that hopefully contains pivot points that can change commitments based on experiment test data."

3  Create a culture that values data-driven decisions

‘Experimentation culture’ is discussed a lot in our industry, but what exactly does that look like for great experimentation programs? 

The decision-making process 

Observing how an organization makes decisions can be very telling, as Bandan Jot Singh, Product Strategy and Growth at Riverty shares. "In my experience working at experimentation-first companies like Booking.com, the most important element for "great experimentation" is the company culture itself." 

Luis Trindade, Principal Product Manager - Experimentation at Farfetch, goes on to illustrate what this looks like. 

“Great experimentation is when all test and learn principles are applied every single day for every single decision," he says.

Experimentation is a constant 

James Mayes, Cofounder & former CEO of Mind the Product, believes experimentation is a habit.

“I believe the crucial aspect to consider here is the nature of the experimentation habit," he says.

"Too often, an experiment is designed and conducted as a one-off. Our users, our markets, and the technology available to us is constantly changing, and therefore experimentation and discovery should be a continuing activity for the team." 

Experimentation is engrained in all aspects of the business 

John Ostrowski, Product Growth and Experimentation Consultant at Positive Experiments, gave a response to our question on what separates good from great that serves as an excellent summary of many of the points already discussed in this post (with a few of his own additions).

According to John, great experimentation needs:

• Automation that makes experimentation feel effortless.
• Has a strategy aligned with the organization. 
• Ideas are sourced at scale from the organization. Running a bunch of random A/B tests is just doing a bunch of tactical things with a chance of finding a winner. Spaghetti testing is bad testing.
• Has a diversified portfolio of experiments. Consider both optimization and innovation-based approaches. Great experimentation looks like a portfolio manager of business ideas. The portfolio evolves as businesses move from product-market fit to maturity.
• There is a well-defined process applied with rigour. There's a process for certain decisions, and not following them impacts performance. 
• Communication. Great experimentation will be able to effectively communicate the results of their experiments to stakeholders in order to drive business decisions.

Thank you to our experts for their contribution to this article!