The difference between good and great companies when it comes to experimentation
It may be a new year, but too many of the companies I meet are stuck in the past.
They have one testing solution for web, where marketers and experimentation agencies thrive, another for product managers and/or engineers to optimize product, and yet another tool for mobile app testing.
A few years ago, that may have made some sense. Today, leading companies (and their CFOs) scoff at the notion of spending capital on a strategy that enforces silos.
The world has changed.
Today, if you want to count yourself as among the leaders, you can’t just optimize acquisition channels with web experimentation leaving product adoption and retention to languish. Nor can product-led companies expect to reach IPO heights without optimizing marketing.
Sadly, where I see the most tool bloat is from profitable companies. (I’m looking at you financial services and cash cows.) Retailers (and increasingly more and more, SaaS companies) are 4x more likely than financial services to adopt a single, unified testing platform for their optimization needs. Their budgets aren’t as padded.
Research conducted in the 2023 Experimentation and Growth survey reveals that companies who believe they are a leader in their category have long abandoned this multi-tool approach. Nine out of ten leading organizations use one platform collaboratively with different teams.
Using a single, unified testing platform isn’t just a solution that pleases the bean counters. Leading companies share several common attributes compared to lagging companies, according to the 2023 Survey.
Here’s the main takeaway: Leading companies harness marketing-led growth (MLG) and product-led growth (PLG) experimenting with every technique in the book.
The difference between good and great companies when it comes to experimentation
Above all, they don’t choose between PLG and MLG.
90% of industry-leading organizations have product-led and marketing-led growth strategies that are strongly aligned.
Leading companies also don’t choose between web and feature experimentation.
Organizations with a high investment in web experimentation are 350% more likely to grow significantly this year. Feature experimentation? They are 270% more likely to grow significantly.
Investments in web and feature experimentation compound over time. Businesses with “well-established” programs that test across their full stack are 69% more likely to grow significantly. The likelihood for significant growth drops to 30% when testing programs are only “developing” or “emerging.”
Each team knows how their work affects the other.
Organizations that expect to “grow significantly” over the next 12 months are 2.3x more likely to report their product- and marketing-led growth teams share KPIs or common goals.
63% of organizations that are growing significantly have marketing and product teams that were designed from the ground up to work collaboratively.
Cross-team alignment on experimentation metrics is important, if not critical, to organizational success, according to 87% of survey respondents.
Leading companies buy and then build on top.
Organizations that invest in off-the-shelf experimentation technology have a 63% chance of growing significantly in 2024. These orgs buy best-of-breed experimentation technology and build bespoke features that fit their precise needs on top.
Leading companies use a single experimentation platform.
90% of leading organizations report that their company relies on a single digital experimentation platform for web and feature experiments. Data analysts, engineers, product managers, and marketers collaborate on the platform.
They know how and when to invite developers to experimentation.
59% of leading organizations report developers as critical to their experimentation needs.
High-growth organizations know how to nurture developer participation in experimentation by limiting when, where, and how developers are required.
Be the silverware, not the spoon.
Our tech stacks, our digital channels, our teams, and above all, every one of our customers is different. Why force a team to adopt one method over another?
Leading companies recognize that to achieve growth, they must foster collaboration between their two growth engines - product and marketing - while simultaneously offering them autonomy to do their experimentation work their way.
The ugly beautiful truth is that the best way to experiment depends on the context. Fortunately, today, technology gives all teams all the experimentation features and methods they need to thrive.
I’d love to hear your story about working (or not) with another team on experimentation. We’ll be featuring (anonymously, if you prefer) your stories in Kameleoon and the Test & Learn Community newsletter over the coming months. Get in touch! Put 15 mins in my calendar?
What is the “Experimentation and Growth Survey”?
Kameleoon wanted to explore the dynamics between marketing-led and product-led growth strategies and experimentation. It aimed to pinpoint what companies leading the way in growth are doing differently than companies growing at average rates or lagging.
Here’s a list of preliminary questions that Kameleoon wanted to help companies answer:
- Are companies that deliver a combination of MLG with PLG more successful than those that don’t combine these approaches?
- How and why do companies align MLG and PLG?
- How critical is the role of experimentation in MLG and PLG?
- How do growth-leading companies who use combined MLG and PLG succeed with experimentation?
- How important is it to these teams to have a platform with web experimentation, feature experimentation, or both?
- Are companies with a holistic experimentation program (e.g., allows business, marketing, e-commerce, product, and technical teams to work together on a single experimentation strategy/platform optimizing across all their stack) more successful than those without?
- Do companies feel more confident about their growth prospects with an off-the-shelf platform or an in-house tool?
- How valuable are developers to companies when it comes to experimentation and growth?
Survey methods
We invited the growth advisor, James McCormick, who formerly led Forrester’s research on experimentation, to conduct the 2023 Experimentation and Growth Survey.
Participants in the double-blind survey included 159 manager-level-and-above roles across the US (75%) and UK (25%). Participants were from five industry verticals, including BFSI (banking, financial services & insurance), healthcare, media/gaming, e-commerce, and SaaS, and they were distributed across product, marketing, engineering, and experimentation teams.
Participants were identified and sorted based on two attributes: 1) How they perceived themselves as leaders in their industry, and 2) Their expected growth over the next 12 months year (i.e., this year, right now, 2024). Around 20% of participants identified their company as being the one-and-only leader in their industry and they expected their company to grow more than their competition over the next year. The insights shared here are derived from the survey findings.
