How can I encourage creativity when experimenting?

This interview is part of Kameleoon's Expert FAQs series, where we interview leaders in data-driven CX optimization and experimentation. Stephen Pavlovich is UK managing director at Conversion, an experimentation agency in Europe and North America, and Bunnyfoot, a leading UX and service design agency. He also works as chief strategy officer at parent company Sideshow Group.

How can I foster creativity when using a scientific and methodological approach such as experimentation?

Most people think about this in the wrong way — experimentation lets you be more creative, not less.

Without experimentation, everyone has to agree on one approach. There’s no better way to stifle creativity than throwing out the bold ideas and choosing the “safe option”.

I recommend creating an experimentation risk profile to ensure you’re running some experiments that push boundaries and others that scale previous winning experiments.

How can I use A/B testing to develop disruptive new digital products?

Experimentation is a big part of effective product development—finding what works and doubling down on it. So let’s focus on the “disruptive” part of this question.

I like Chamath Palihapitiya’s approach to disruption when leading Facebook’s growth team. He spoke about invalidating “lore” that existed, working to “disprove all of the random anecdotal nonsense that filtered around the company.”

There's this lore in every business and industry, the “that’s the way we’ve always done it” stuff. Disruption comes from questioning these assumptions. And experimentation gives us the perfect way to test them, with a safety net.

For example, If you want to see if customers want ‘X,’ you might run a painted door experiment, promoting it on the website, adding it to the navigation, or selling visitors on it, even though it doesn’t exist. Doing so allows you to test multiple new ideas and monitor the potential demand before investing.

The concept of disrupting lore applies to commercial strategy too. Domino’s Pizza has always offered free delivery, even when competitors charged. Using experimentation, we helped Domino’s understand the commercial impact of adding a delivery charge. With data, we could show the commercial implications of charging for delivery and how consumer sentiment varied by day of the week. Meaning Domino’s could make a data-informed decision, and Dom fans will know that they continue to champion free delivery.

Experimentation can discover a disruptive new product, determine the best price for it, and then optimize its marketing – all with confidence.

How can user psychology help to build sticky digital products?

Psychology affects all aspects of customer behavior, perhaps none more so than creating “sticky” products. Stickiness relies on a positive experience in the moment and a trigger or hook to get users back in the future, so the behavioral change has to last beyond the initial session.

We use a behavioral psychology framework taken from Stanford psychologist, BJ Fogg. We determine a user’s motivation, trigger, and ability to complete the initial action (e.g., visiting a SaaS website and signing up for a free trial or going to a news website with a paywall and registering, etc.)

Then we’ll apply the same framework to key user segments already active and engaged with the product – extending this to understand when and how they use it.

Next, we optimize the customer experience to ensure that new and existing users:

• Understand the product. Many companies fail this basic step and assume that just because someone has signed up means, they’re sold on what the product does and how to use it.
• Want to use the product. Again, many products make it almost too easy to sign up, so new users miss all the persuasive content and are dumped into a product they’re not sold on.
• Get meaningful value as quickly as possible. This is typically called the “aha moment” in SaaS. Essentially, it’s about reducing the time and effort between the user landing on the website and the initial point of customer satisfaction when using the product.
• Build habits to bring people back. What limits stickiness with engaged users is that people’s motivation has a short half-life. The behavior will drop off if you don’t actively build the habit.

One important consideration here is that when experimenting on digital products, we should consider that existing users may initially react negatively to a new experience.

Likewise, when adding new functionality, we may mistake users’ initial curiosity around a novel feature to be true engagement. That’s why it’s essential to build products in stages.

What metrics should be used to judge whether a new product experiment is successful?

It depends on the hypothesis and what you’re looking to affect. Here are a few examples:

• Suppose you’re running a painted door experiment. In that case, you’ll want to track interactions with an element as the primary metric, plus guardrail metrics to ensure you’re not negatively affecting the experience or customer sentiment.
• If you’re optimizing the product experience, you may want to measure the usage of key features plus retention metrics.
• If you’re testing new commercial products, you’ll want to measure sales, AOV, and, if relevant, LTV.

How can experimentation and product teams work efficiently together?

Experimentation enables the product team to move quickly and with confidence.

Having worked with many product-led companies (Gousto, Deliveroo, and Lookiero), I find the best model for collaboration is where the CRO team apply the experimentation framework to the product strategy. This should prevent you from spending a year building the wrong product. Instead, the product strategy is framed as a sequence of experiments that help to guide the product direction, getting concrete feedback from users at each point. Based on each experiment, the product strategy is refined and optimized. This ensures that it matches the needs of the user and the organization.

We heard you are really into baking. Have you applied any experimentation techniques to your baking?

Of course! I think anyone who works in experimentation has an optimization mindset and is constantly testing new approaches and trying to improve performance. Whether it’s A/B testing new commercial models or loading a dishwasher…

Baking is the same. I typically bake 4–6 loaves each week, so I have plenty of ‘variations’ to test different approaches. You can control so many variables: oven temperature, dough hydration, starter activity, etc., so there’s endless opportunity to test and learn.

My delicate hands have never done an honest day’s work in their lives, but this is the closest they get. :)