How can my team use feature experimentation to drive product-led growth?

This interview is part of Kameleoon's Expert FAQ series, where we interview leaders in data-driven CX optimization and experimentation. Shagun Aulakh has been in the experimentation space for nearly a decade, building and scaling experimentation programs for multiple companies, both in-house and consulting (Macy’s, Charlotte Russe, Optimizely, Tophatter, Gympass). She is currently the Director of Product Management at American Express.

How are the best companies using feature experimentation to drive product-led growth (PLG)? 

The best companies use feature experimentation as THE central engine to develop their product strategy. 

Average companies use the agile process to deliver as many products and features as fast as possible. But the best companies understand that to truly push innovation and get to the best-performing, most valuable product, you need constant experimentation.

Constant experimentation allows you to understand the statistical performance of the feature and enables you to answer why it’s performing as such. 

The best companies also understand that speed of delivery should not be at any cost. Experimentation may actually slow down product delivery, but the impact of product delivered through an experimentation process will be much greater and provide greater predictability of improvement. 

When companies put feature experimentation at the center of their growth flywheel, it allows them to learn more about where and what to build more quickly; therefore, they can pivot more efficiently, saving time, resources, and ultimately money in the long term. 

The best companies do a few common things with experimentation. 

They don’t necessarily think about experimentation

They think about continuous delivery and experiment consistently and strategically within their SCRUM processes. They experiment to determine their product backlog, build MVPs, incrementally build, and inform the next experiment.

This enables continual building and optimizing of products. 

They understand the tools in their arsenal are to gain insights

They use A/B testing, feature flags, canary releases, and staged rollouts for the appropriate situation and intended goal to learn and iterate as much as possible.

They understand that experimentation’s true purpose is deriving insights

They focus less on how each test pushes a metric and more on how experimentation produces a better understanding of user behavior. They use experiments to learn how the features they are building affect journeys and how learnings can influence strategies.

The goal to drive the velocity of insights is absolutely critical because focusing on a particular test metric movement can undermine a growth and innovation mindset. In such scenarios, teams only run tests that they believe will “win,” but if we know something will win, why test it?

So focusing on delivering insights promotes the idea that a losing variant, a winning variant, and even flat results all provide new understanding and can lead to new ideas.

They go beyond the superficial

Not only do they run experiments on the front end to test visual cues and interactions they also incorporate experimentation into engineering and data science teams. They experiment with algorithms, data models, back-end builds, and beyond.

This provides a total 360 feature testing approach, including all the variables that can affect feature performance. 

What are the pros and cons of using feature flags for experimentation?

Feature flagging is great for the risk-averse. If you only focus on experimentation, you could build a perfect feature, but you may not identify bugs or other failures that could occur. Feature flags provide a means to reduce risk when deploying features.

First, feature flags enable the CI/CD goal of fast and continuous deployment while allowing for risk mitigation by turning features on and off without having to deploy new code or roll back code, eliminating the need to manage multiple branches of code.

This independent functioning of feature rollouts allows teams to test and learn and not create risk or instability in production.

Additionally, feature flags provide you with control. Teams can build features behind a flag and keep the flag off until they are ready to make that feature visible. Feature flags allow teams to release features to a small base of users to measure and track stability, then gradually roll out to a larger user base once the feature is validated. 

Flags also enable teams to have an easy kill switch. Is the feature driving a drop in conversions? Just turn the flag off and kill the feature. 

Finally, flags enable product managers to determine release dates and segments of visitors that will be exposed to a feature and create a more constant feedback loop.

It is important to note that feature flags are not without cons—two pitfalls come to my mind: 

You can irritate some engineers. A common frustration is that feature flags can introduce technical debt and make it hard for engineers to understand the code. It could result in engineers deleting necessary code or having so many lines of code creating a mess to wade through.

You can hurt the performance of your website. Of the two pitfalls, this is the worst because it directly impacts your visitors. Sometimes feature flags are created and forgotten. Team members leave, code isn’t documented, and a feature flag can be left for a long time.

One major feature flag disaster was at a financial company where a feature flag was forgotten for eight years! Years later, a team made a deployment and unknowingly turned the feature flag on, which resulted in the company making unintended trades that caused a loss of half a billion dollars. 

You can easily overcome these risks with good governance, process, and documentation. Creating guidelines like keeping feature flags short-lived, creating easy-to-understand documentation on what the code does, and naming your feature flags something intuitive will help tremendously to prevent possible risks. 

How can I convince my organization to use feature flags for feature experimentation, not just engineering?  

To convince an organization is a challenging task. It depends on why the organization is not using feature flags or running feature experiments. 

Start with education and dispel myths from the top down. The hard truth is that change comes from the top down, or at least change happens more quickly when coming from the top down. Therefore, start educating the highest stakeholder you can, such as Product VPs or CTOs. 

Provide leaders with a vision of what is possible in the strategy and speed of innovation if feature flags and experimentation were used as a primary lever. The most powerful way to showcase this vision is with real examples.

If possible, start using feature flags and feature experiments in your team. Use the insights as tangible examples and extrapolate that to the change that could happen if used as a broader strategy. 

How do product/tech/marketing teams converge with regard to feature experimentation and PLG? 

I don’t think there is a singular answer. It depends on how your organization and teams are structured and their different needs. 

In my experience, experimentation is often managed separately. Product teams have their own experimentation practice that is separate and distinct from marketing, and then tech is somewhere in the middle supporting the setup of experiments.

We started this way in my previous role running experimentation at a large retailer. But an epiphany hit as marketing began sharing test plans and ideas with the product team. We realized if we partnered, we could accomplish more, eliminate duplicative work, and use each other’s data and insights to create more meaningful test plans that could drive ROI (the holy grail). 

Marketing began attending product roadmap shareouts, and we instituted experimentation forums to share challenges and recent learnings, discuss resource needs, and even “loan” headcount when needed.

What resulted was powerful insights into true product strategy and even the ability to influence critical leadership decisions around shipping promotions and policies impacting profit. It shifted our narrative from ‘this specific test drove this specific metric improvement’ to ‘if we shift our strategy to lowering our shipping fees, we will lose millions in profit, but offering a loyalty program with free shipping to high-spend customers may actually improve customer acquisition and LTV.’ That sure got the executive's attention! 

How should I manage PLG across multiple product teams? 

Again, this depends on your unique organization and how your teams are structured. 

Some companies set up product management as distributed, independent teams, each with its own approaches and programs. Others opt for a highly centralized PLG program where enterprise standards, templates, and processes are followed by all teams. And there’s a spectrum of approaches between these two. 

There is no right way, though there are benefits to some centralization of best practices and processes. It makes it easier for:

• Cross-team sharing.
• Prevention of errors as a result of complex, confusing code.
• Clarity on how to manage feature flag removal and cleanup. 
• Understanding the impact and challenges across teams.

Shagun, you’ve spoken at some of the biggest industry conferences over the years. What tips would you give someone giving their first on-stage talk?

My first speaking engagement at an industry conference was about seven years ago. I agreed to this session, assuming 20 or 30 people would attend. No big deal, I can manage that. 

When I walked into the room that morning, 500 people were waiting for me to present! It goes without saying I was officially in freak-out mode. But, in the end, the session was amazing, and I received an incredible amount of positive feedback from the attendees.

I think I was able to rise above my anxiety at the time because of three tips that anyone can use, whether presenting at a large conference or speaking in a small meeting with your stakeholders. 

Have something important to say

Don’t look to speak at conferences for accolades or attention. Focus your energy on understanding your unique strength and experience and what you want to share with other people. It’s not about you; it’s about sharing knowledge for the benefit of others in your industry.

When you do that, you will naturally do two things. You will speak with enthusiasm and passion, which always draws in audiences, and you will add value, making you memorable. 

Preparation 

It sounds cliché, but preparation is key. Before I presented at my first conference, I spent weeks determining the objective of my session and articulating what I wanted the audience to learn and take away.

I put a ton of time into developing content, modifying slides, and creating my narrative. I practiced with many people of varying levels of knowledge to see what resonated and what fell flat so that my presentation would apply to all levels and people from different functions.

Be human

We’ve all had at least one experience when we’ve watched a presenter, and it was a snore-fest, or the presentation was so complex that you felt lost. Most of us are looking to learn from those that know more than us, but we don’t connect with someone on a pedestal preaching at us.

We connect with those who reflect our own struggles and challenges, show vulnerability, and have a sense of humor and fun sharing what they know. When you mix those elements with a great story or a valuable lesson, it becomes inspiring.