What separates good vs. great experimentation programs?
Do you belong to a good experimentation program but, in quiet desperation, dream of greatness?
Both good and great programs know how to test well.
But instead of optimizing the process of experimentation itself, great programs ensure that tests are aligned to senior leader business goals and are actionable—according to our panel.
On May 28, Kameleoon spoke to 3 experimentation experts—Shagun Aulakh, Manuel Da Costa, and Ben Labay—to find the secret sauce that allows some programs rise above the rest.
Main takeaway: A great program goes beyond experimentation—it’s aligned with, and visible to, the organization’s key decision makers
Whereas a good team focuses on the details of their tests, a great one is able to take a step back and see the program through the eyes of the organization’s stakeholders.
To do this, they go beyond data, using emotional hooks, to communicate insights that are actionable and fit into broader strategic goals.
They don’t try to be management consultants and remake the organization to fit around experimentation from the start, but adapt their program to realities on the ground.
Finally, they have a North Star metric to keep everybody focused, and an orchestrator who allows them to scale without getting sloppy.
To watch the replay, click here.
How great programs influence change
1. Good programs communicate important insights. Great programs communicate insights in a way that ties back to C-suite goals.
The C-suite has their own goals, their own initiatives, their own KPIs, their own things that keep them up at night. While they’ll sit in a meeting and hear the insights your experiments generated, at the end of the day, it’s just another PowerPoint deck, spreadsheet or dashboard out of the barrage they’re already getting.
Practitioners should think about what C-level people want out of the results, and position their experiments to help solve real problems within the business.
2. Good programs talk about findings. Great programs make findings actionable.
As an experimenter, you have to understand the company and be where your stakeholders are. You need to find out the medium that they prefer to best communicate the insights and actions that need to be taken from your experiments.
Note that every company is different, and understanding the best way to reach the organization with your findings is crucial. For example, some companies want insights packaged up in the form of newsletters while others want formal documents. Some like the charts and the graphs and others want simple one pagers.
How great programs deal with silos
3. Good programs complain about silos. Great programs understand that silos will always exist and work around them.
There are silos in every company. Often, changing management and organizational structure is far bigger than the job of an experimentation program, or lone experimenter.
While experimenters have to affect change in a business, great programs know how to pick their battles and work within the silos and reality of the organization. This doesn’t mean throwing in the towel in all cases, but carefully prioritizing which silos absolutely need to be tackled, rather than becoming a management consultant.
4. But, some great programs break down silos too.
Some experimenters, like Manuel, lean away from calling it the job of an experimenter to break down silos. He believes that trying to get silos working together or even broken down is an organizational design challenge.
“It needs to be done from the top down,” he said. “It cannot be done from the bottom up, no matter how much you'd love to do that.
On the other hand, Ben believes experimenters, by definition, break down silos.
How great programs structure their teams and KPIs
5. Good programs have goals. Great programs have a North Star metric.
Goals, KPIs, objectives, and a well-aligned business strategy is what motivates people. Motivation is defined by having a clear understanding of that object in the future, which you can see. And you have a clear vision of the path to get there. Those are the two ingredients that a great experimentation program needs.Good programs have KPIs to judge the performance of their tests, but the test pipeline isn’t pointed toward a blindingly clear star. Great programs move like self-propelling machines to one well-defined big destination.
“Great programs have a metrics strategy that aligns to what leadership thinks is the North Star” said Ben. “What the leadership thinks is the values that they want to bring to the organization.”
6. Good experimentation teams have managers. Great teams have orchestrators.
Every robust experimentation program has contributors—those providing the inputs and outputs, like designers and development—and managers. The job of the manager is essentially oversight. While, at most teams, this role is a CRO experimentation specialist tasked with management, great teams understand that this manager needs to do more than keep the gears turning.
Manuel says that the orchestrator is more like a generalist who understands CRO and experimentation well, but has the distance to be on the outside looking in, and coaching the team to maintain best practices.
Don't miss the next Live Session. Follow Kameleoon on LinkedIn to learn about upcoming webinars with the world's leading experts on experimentation.
