What you should never do in your experimentation program, according to these 8 experts
In today's data-driven world, experimentation is essential for businesses looking to optimize their products and services. However, not all programs are created equal, and while the internet is full of how-to guides, few cover what not to do.
We asked eight experts to reveal what you should never do in your experimentation program. These experts have seen it all and there is a lot to learn from their wisdom - here are the top five things you should never do.
1 Never get blindsided by test outcomes
Companies may have different objectives when it comes to experimentation. Ultimately, however, testing can provide direction and insights into important business questions.
But, to use experimentation in this way, you need the correct focus.
Annemarie Dooling, VP of News Experimentation at McClatchy explains how to find the right questions.
“Never use the word failure. Having an experiment end as a "failure" means there was an assumption prior to experimentation that you were trying to achieve rather than leading with questions," says Annemarie.
2 Never test without data
Early-stage experimentation programs often start by testing the team's own ideas. For example, changing the button color to blue will increase conversions.
But this falls into the trap of not asking the right questions (see above). After all, knowing if customers convert better with a blue button isn’t as valuable as knowing what features your customers are willing to pay for.
So how should you identify the right questions? One word; data. Mohtashim Hashimi, Lead Product Manager, Growth at Zenefits explains what you're looking for in the data.
3 Never test without the right goals in place
A common theme mentioned by many experts was the need to have clear goals. Without goals, it's impossible to know if you've been successful.
Aditi Gupta, Senior Product Manager at Thomson Reuters, shares how you should set your goals.
Mohtashim goes on to share how you should define important metrics connected to any goals or objectives used in your program.
4 Never experiment without a rigorous process
What’s the point of testing if you can’t trust the results? Sandip Amlani, Global Optimisation Manager at Cisco says you should never run experiments unless you can execute them rigorously.
“Experimentation is a scientific methodology that, when used correctly, provides a very effective way to make decisions," says Sandip.
Anjali Arora Mehra, Senior Director - Analytics, Data Science, Experimentation & Instrumentation at DocuSign, shares what a rigorous process looks like.
"A strong hypothesis and definition of a primary KPI are absolute essentials to a successful experimentation program," says Anjali.
"You may think it would slow you down, but in reality, it takes much longer to drive any meaningful insights and learning without a good understanding of why you think an idea/feature may work and how you will measure the impact.”
Diving further into what elements of your experimentation program needs to have a rigorous process, we find ourselves at test statistics.
Bartosz Mozyrko, Senior Product Manager at GoDaddy, explains what you need.
"When the p-value is checked during an experiment, and the experiment is concluded right after it falls below alpha (e.g., 0.05), it violates the underlying assumptions of the controlled A/B experiment," says Bartosz. "As a result, the p-value becomes unreliable, leading to results that cannot be trusted."
"To prevent this, it's best to estimate the required sample size before starting your experiment. Let the experiment run until that sample has been reached. And don't forget to write down your estimate somewhere (e.g., Jira ticket or Confluence page) as a reference for following experiments!"
5 Never slow down decision-making by doing too much at once
Once you have all of the right inputs and processes in place, pressure can mount to run more and more tests. While a rigorous process can help you run multiple tests simultaneously, you must be mindful of running too many at any one time.
Pius Binder, Product Manager at Canva, explains why.
The five things you should never do
Here’s a neat little summary to remind you of what not to do when it comes to experimentation.
- Never get blindsided by test outcomes. Focus on asking the right questions by crafting solid hypotheses.
- Never test without data. Find the friction points in the user journey.
- Never test without the right goals in place. Experimentation goals should align with company goals.
- Never experiment without a rigorous process. For example, before starting any test, ensure you have a test plan which sets out the sample size.
- Never slow down decision-making by doing too much at once. Ensure you don’t run too many tests simultaneously, making analyzing the results challenging and time-consuming.
These five points provide an excellent cheat sheet for anyone running a testing program who wants to avoid common pitfalls. Follow the above advice, and you can’t go far wrong.
A massive thank you to the experimentation experts for sharing their wisdom!
