16 PLG leaders on what separates good from great companies when it comes to experimentation

We asked 16 top product-led growth leaders what separates a good from a great company regarding experimentation.

Three main themes emerged regarding the companies that excel in experimentation and drive growth:

Clear objectives for experimentation.
A solid process for running experiments.
A culture of data-driven decisions.

However, there is some nuance to the above. So, let’s explore what our 16 product leaders had to say and how you can apply these points to your experimentation program.

Walk before you run

Most of the following advice is aimed at teams who have already implemented experimentation to some extent.

If you are yet to do so, Kevin Anderson, Product Manager Experimentation at Vista, has some advice on getting started.

“Before you can get great at experimentation, you need to start. I recommend starting by testing the changes you are already making to your product and measuring the impact on key metrics," says Kevin.

"Forget about doing more user research, writing better hypotheses, and prioritising new ideas. Start experimenting with the next feature on your backlog. You will quickly learn that some features are making an impact, but many are not. Some even have a negative impact on customer and business outcomes. Use that evidence to become smarter in shaping your roadmap and product."

Now that you are actually experimenting, you can start thinking about becoming great. Shape better processes, invest in better tooling, and get someone to lead the efforts to decrease the friction around setting up experiments.

Kevin Anderson

Product Manager Experimentation, Vista

1 Clear experimentation objectives

While many of the product thought leaders agreed that clear objectives were essential to reaching a ‘great’ level of experimentation, their opinions differed on the objective itself.

This perhaps reflects what’s important to the organizations they work with, the type of product, or their north star metric.

Answer your company’s biggest problems

Patrick Buffum, Product Manager, Optimization at GoBankingRates, suggests that good to great diverges when it comes to the program's level of influence over the organization.

The biggest difference to me comes down to the amount of influence the program has over the organization. Technically speaking, a ‘great’ program would do many of the same things as a ‘good’ program, like running tests to defined statistical thresholds, paying attention to guardrail and quality metrics (in addition to the primary metric), and having a full backlog.

Patrick Buffum

Product Manager, Optimization, GoBankingRates

Patrick goes on to say, "if you can ensure that your tests also solve your company’s biggest problems/answer its questions, then experimentation has earned a seat at the table during the most important discussions.”

Ankit Goswami, Global Product Lead at Expedia, provides an additional point that “great” experimentation programs meet not only the company, but the customer’s needs too.

“Great experiment provides not only useful information and insights but also generates a significant impact on key business metrics," says Ankit.

By embracing a proactive approach to experimentation and fostering a culture of innovation and risk-taking, product leaders can unlock the full potential of their product and drive meaningful improvements that meet the needs of their customers and their business.

Ankit Goswami

Global Product Lead, Expedia

Objectives should go beyond testing marketing tactics

May Chin, Principal Product Manager - Growth Experimentation & Analytics at ZALORA explains the level of integration needed within a business if you want to use experimentation to its fullest potential.

“Great experimentation is when A/B testing is an intrinsic part of the overall decision-making process, where everything from branding to P&L is informed by experimentation," says May.

"At the end of the day, experimentation is simply one of many R&D methods to better inform decisions and understand the potential risks around them."

Bad experimentation is when A/B testing is simply seen as a revenue generator or marketing tactic, rather than a powerful, innovation-steering tool that could potentially shape decisions at even the company level.

May Chin

Principal Product Manager, ZALORA

Whatever the goal, it needs to be measurable

Ben Carter, Global Head of Digital Product Management at IG, raises an important point, no matter what the goal or objective, it needs to be measurable.

“Experimentation is fundamental to creating great products, but in the rush to experiment, it's easy to forget that the goal is what we learn from experimentation, not the experiment itself," says Ben.

The difference between good and great experimentation, for me, is a clear articulation of what you are trying to learn and how you will measure it. That way, you know if you have achieved your goal or not and don’t end up in continuous experimentation cycles.

Ben Carter

Global Head of Digital Product Management, IG

Meaningful insights that lead to follow-up actions

Results are good, but for great experimentation to occur, your experiments should give you insights and learnings you can act on. Janna Bastow, CEO & Co-founder at ProdPad, explains below.

The difference between running good and great experiments can be the difference between running forward with lots of clear learnings versus just treading water in a murky soup of unknowns.

Janna Bastow

CEO & Co-founder, ProdPad

Good experiments might get you somewhere...though they might also lack a clear goal or have dodgy data and may not control for confounding factors. They may not give you meaningful insights or drive product development," says Janna.

She goes on to say, "Great experiments, on the other hand, will really shine a light on the problem! These experiments have a clear objective, are designed well to eliminate bias, have solid data, use proper analysis, give you actionable insights, and lead to follow-up actions that advance your product. In a nutshell, great experiments are well crafted, give you valuable direction, and push your product forward.”

The objective isn’t to find winning tests

For Jonathan Callahan, Senior Growth Product Manager, Games at Unity, the objective of running tests shouldn’t be to find winners.

“The big question is, are you prioritizing running winning experiments (which I see happen a lot), or are you prioritizing moving the business forward? If you are prioritizing winning experiments, you can get several years into a program and have a user experience and conversion rate that is largely unchanged," explains Jonathan.

If you are really trying to move the needle and actually answer big questions for the business, even if that means taking big swings and missing, you will be much more likely to actually have a tangible impact.

Jonathan Callahan

Senior Growth Product Manager Games, Unity

2 Follow a rigorous process

While many good experimentation programs have a process in place, great programs have an elevated level of rigor that removes doubts about any test results.

Rigour doesn’t mean rigid

Bhavik Patel, Founder & Lead Product Strategist at CAUSL says what separates good and great experimentation programs is, "Discipline in experimentation design. Teams that run great experiments are disciplined when it comes to planning experiments. They design each experiment meticulously."

Being disciplined doesn't mean that they are rigid and inflexible. It just means they have a compass that is pointed north for each experiment, and they can adjust their course as needed instead of aimlessly sailing the sea. That, to me, is what separates the great from the good.

Bhavik Patel

Founder & Lead Product Strategist, CAUSL

Not everyone agrees that a rigorous process is what separates good from great. For Henning Heinrich, Group Product Manager, Product Growth at Hootsuite, velocity is more important.

Experimentation is a flywheel that gets faster and more powerful as you run more experiments. To go from good to great, focus on velocity over perfection.

Henning Heinrich

Group Product Manager, Product Growth, Hootsuite

Having trust in results is key

If your data isn’t accurate, experimentation programs can quickly lose support. What’s the point in running experiments if you can’t rely on the results?

Running experiments is just a start. It gets awesome when you start running good experiments with solid hypotheses, trustworthy analysis, and valuable learnings that make you and your organisation smarter.

Denise Visser

Product Manager Experimentation, bol.com

Anthony Murphy, Founder at Product Pathways, goes on to specify.

Having a control group or, at the very least, having a baseline that you can properly compare against to see whether your experiment has had a clear improvement or not. As well as considering dimensions such as sample sizes, audience diversity, etc., for experimentation accuracy and reducing bias.

Anthony Murphy

Founder, Product Pathways

Kalki Gillespie, Senior Product Manager - Experimentation at NerdWallet, adds what his version of a great process involves.

What separates good from great from a product leader’s perspective is that there’s a justifiable and explainable reason for each experiment, prioritized by reach. Also, there’s a good mix between the types and scale of experiments, i.e., small vs. big.

Kalki Gillespie

Senior Product Manager Experimentation, NerdWallet

He goes on to say, "And finally, experiments are aligned and sequenced well so that after a set of tests are done, the business is informed about the grand hypothesis that is implicit in the product roadmap. A roadmap that hopefully contains pivot points that can change commitments based on experiment test data."

3 Create a culture that values data-driven decisions

‘Experimentation culture’ is discussed a lot in our industry, but what exactly does that look like for great experimentation programs?

The decision-making process

Observing how an organization makes decisions can be very telling, as Bandan Jot Singh, Product Strategy and Growth at Riverty shares. "In my experience working at experimentation-first companies like Booking.com, the most important element for "great experimentation" is the company culture itself."

A culture that values data-driven decision making and encourages continuous learning and iteration is crucial for ensuring the success of experimentation efforts. Without this, any product leader that pushes for experimentation would not find takers in the rest of the organization and hence would lack the necessary influence on decision making.

Bandan Jot Singh

Product Strategy and Growth, Riverty

Luis Trindade, Principal Product Manager - Experimentation at Farfetch, goes on to illustrate what this looks like.

“Great experimentation is when all test and learn principles are applied every single day for every single decision," he says.

A true product leader will be the first one to celebrate failure and incentivise their product managers to be bold. It’s this strong experimental mindset that will drive the desired outcomes from a business perspective in the long term — because failing leads to learning more about your customers' needs and how they fit the business.

Luis Trindade

Principal Product Manager, Farfetch

Experimentation is a constant

James Mayes, Cofounder & former CEO of Mind the Product, believes experimentation is a habit.

“I believe the crucial aspect to consider here is the nature of the experimentation habit," he says.

"Too often, an experiment is designed and conducted as a one-off. Our users, our markets, and the technology available to us is constantly changing, and therefore experimentation and discovery should be a continuing activity for the team."

Experimentation is a muscle that grows stronger with regular use - asking better questions, executing with greater rigour, and constantly driving learnings, innovation, and growth. Want to go from good to great? Spend the time building an ongoing culture around these aspects.

James Mayes

Cofounder & former CEO, Mind the Product

Experimentation is engrained in all aspects of the business

John Ostrowski, Product Growth and Experimentation Consultant at Positive Experiments, gave a response to our question on what separates good from great that serves as an excellent summary of many of the points already discussed in this post (with a few of his own additions).

According to John, great experimentation needs:

Automation that makes experimentation feel effortless.
Has a strategy aligned with the organization.
Ideas are sourced at scale from the organization. Running a bunch of random A/B tests is just doing a bunch of tactical things with a chance of finding a winner. Spaghetti testing is bad testing.
Has a diversified portfolio of experiments. Consider both optimization and innovation-based approaches. Great experimentation looks like a portfolio manager of business ideas. The portfolio evolves as businesses move from product-market fit to maturity.
There is a well-defined process applied with rigour. There's a process for certain decisions, and not following them impacts performance.
Communication. Great experimentation will be able to effectively communicate the results of their experiments to stakeholders in order to drive business decisions.

Great is when experimentation is so embedded in the process that it doesn't need to be discussed separately, and it becomes part of the product development lifecycle.

John Ostrowski

Product Growth and Consultant,Positive Experiments

Thank you to our experts for their contribution to this article!