23 Experts share bold predictions for experimentation in 2024
We sought the opinions of 23 leaders in our industry to get their predictions on where the industry is heading and the biggest trends likely to impact experimentation in the coming year.
Among the foreseen changes is how experimentation will transcend the confines of a siloed team and surpass the bounds of web experimentation—a notable departure from the myopic Conversion Rate Optimization (CRO) lens. The move means experimentation will become an integral part of product and marketing teams, among others.
The impact of data legislation, especially in the EU, is expected to reshape the industry, driving companies to adapt and prioritize customer-centric data approaches. This shift aligns closely with AI-driven personalization, which will put even greater emphasis on data.
These predictions foreshadow a fundamental transition from CRO to decision science, ushering in a new era of experimentation with a broader and more impactful perspective.
Amidst these predictions, there is hope for greater support for the next generation of practitioners, especially as job roles and expected competencies change with the increased use of AI and machine learning tools. 2024 promises to be a year of dynamic shifts and profound evolution for the experimentation industry.
Experimentation will continue to mature
1. Experimentation will extend across all teams
The Kameleoon 2023 Experimentation & Growth Survey found that businesses with well-aligned product-led and marketing-led growth strategies are expected to be 81% more likely to grow significantly in 2024. Experimentation breaking out of its silo will generate big wins for those able to follow this trend. But as Ruben discusses below, not all companies are ready for this shift;
2. Self-serve models will drive more teams to experiment
What is driving experimentation into other teams (specifically product and marketing) across companies? Sai believes this transition is being fueled by the popularity of self-serve sales models, which require a digital version of the sales and onboarding process. To achieve this, marketing, sales, and product teams need to be involved in experimentation to create highly converting experiences.
In e-commerce companies, this fusion has already been present, but we are now noticing it in other industries as well. Whether it's a software company offering a self-serve signup or an insurance company enabling individuals to initiate their coverage online, both will necessitate CRO experts who understand that the website is the product itself.”
The landscape of experimentation has surpassed what people are used to: testing on a website. Companies like Disney are constantly pushing the envelope of what is experimentation. They don't just test website changes, they're testing past the website.
Don't let experimentation learnings stop at your website. Technology has reached the point where full customer journeys can be produced all on the same device and other devices for known users. Engineers are already invested in the product and can be leveraged to help build experiments; they know the code best. Push the boundaries.
In 2024, teams should be trying to push innovation and insights outside of the four walls of a screen."
3. CRO will be seen as decision science
Kelly sees experimentation morphing into something different altogether: decision science.
Similar to Kelly’s prediction, Michiel suggests that as experimentation matures, teams will look at different methods to help make decisions, such as meta-analysis.
Gary also shares some of the areas where testing can help with business decision-making outside of website optimization.
4. Increasing industry maturity will change who is experimenting
The more mature experimentation is, the greater the impact our work has on the business. Interestingly, the recent Google Optimize sunset is having a lasting impact on the wider industry and its overall maturity, as Shiva explains.
This will combat an economy where CROs are amongst many being laid off, unfortunately (which will ding our maturity), but overall experimentation maturity across the board will increase, which will only beget more folks adopting it.”
It’s not only the absence of Google Optimize increasing industry maturity. Deborah examines the impact of a broader move to a pay-to-play model.
Plus, as we shift away from being able to freely collect 3rd and 1st party data, the cost to gather and analyze user information is becoming increasingly high. Data itself is becoming a commodity that comes at a premium. In essence, experimentation is moving from an activity that anyone used to be able to do for free to something only available to the world’s biggest and most elite corporations who can afford to pay to play.”
However, Manuel believes we’re still far from full maturity and provides a list of areas businesses need to fix before we can escape the “CRO hamster wheel” including the narrative told to senior management.
There is still a lack of understanding at the senior level about what experimentation can really do. Typically, they have bought into the narrative about big wins and revenue gains and ignore the true power of experimentation. Couple that with the fact there are still a lot of inexperienced CROs parading as senior experimentation leads or managers with no understanding of how to communicate well. There are a lot of bad practices festering at the practitioner levels that are kept well hidden from senior management. Selective reporting happens, too. So there's a lack of true oversight from the top down.”
It’s undeniable that experimentation has grown in popularity over recent years, but Sina has noticed a slightly less positive trend emerging.
Similar to SEO 10 years ago, there is a pressing need to educate consumers on "black hat" vs. "white hat" CRO approaches and to share resources on how to vet expert service providers from pretenders. Otherwise, legitimate agencies providing data-driven and research-based CRO will continue to face increasing consumer skepticism, fuelled by disappointing experiences with agencies that don't understand statistics or customer research and are just spaghetti-testing.”
AI & machine learning will disrupt all areas of the experimentation industry
5. The year of advanced customization
Many pundits have heralded previous years as “the year of personalization,” but to date, most businesses have only scratched the surface of what personalization can do. With the increasing sophistication of machine learning algorithms, online competition, and consumer demand for individualized content, will 2024 finally be the year of personalization?
Kelly has some reservations.
On the other hand, customers tell us they like flexibility and customization, such as filtering and navigation. Helping them get where they want to go faster with those capabilities will be key.”
Are we actually looking at a year of advanced AI segmentation and customization rather than “true” 1-1 personalization? Koutheir's prediction suggests so.
6. Innovations will focus on privacy-preserving personalization techniques
Rohan is a little more optimistic about the impact of personalization next year, along with some critical predictions on how this will impact the wider industry.
The impact will be twofold: firstly, it will lead to a more efficient and effective CRO process as AI can quickly analyze vast amounts of data to determine the optimal experience for each user. Secondly, it will shift the industry focus towards the ethical use of data and privacy compliance, as personalization relies heavily on user data. Companies will need to balance the drive for personalization with the growing public concern over data privacy, potentially leading to innovations in privacy-preserving personalization techniques.
This trend will likely elevate the importance of roles centered around data science and AI within CRO teams. It could lead to a new standard in user experience, where personalization is not just a feature but an expectation.”
7. The future of the web as a prompt machine
The potential of AI is so broad it’s fascinating to hear how our experts think things might turn out. While James has reservations about when we might see big changes, there’s no doubt it will cause a significant shift in our industry and the world at large.
The optimist in me believes every website could become a form of Google—a prompt machine to generate a single page of content for every individual person. Maybe personalization isn’t dead; it just needed a little AI boost?
The pessimist in me believes we’ll spend another couple of years frantically building AI features into every facet of our tech stack, only to have them collect dust years later.
How it plays out could determine whether we keep optimizing the front end of digital experiences or shift to optimizing prompts and what we return for prompts in the next few years.”
8. Companies with solid data foundations will excel with AI
To deal with some of James' pessimism, Nils suggests how to avoid getting blinded by the new shiny AI.
If I were to make a prediction for 2024, it would be that the companies that excel at getting the basics right won’t be blinded by shiny objects (as easily) but instead use technology like AI to build on solid foundations and with that outcompete their competition.”
Experimenters' relationship with data will get tougher
9. No more cookies means upping the ante on zero and first-party data
While the cookie phase-out has been on the cards for a while (and doesn’t directly impact A/B testing), it does affect the broader digital landscape and what data companies can collect and use about customers. As SIobodan explains,
10. New legislation will impact our relationship with data and tools
It’s not just cookies that will shake up how we use data; new legislation and geopolitical events will also affect our industry, as Lucas explains;
I expect more stringent legislation to be developed in 2024. Potentially, a far tougher ruling over Google Analytics restricting or even prohibiting its use, browser standards to be more privacy friendly, making it harder to do personalization and experimentation might require explicit consent. Due to the geopolitical climate, the country a tool comes from will become more relevant, with preferences rising for EU-based tools for businesses based in Europe. Outside of Europe, that could lead to a mirroring effect.”
Experimentation teams will have to focus on their human advantage
11. Employers will look for AI & machine learning tool competency
With all this talk of AI and machine learning, how many of us actually know how to use the plethora of tools that are emerging? The AI trend is predicted to introduce demand for a whole new set of skills, which need to be administered in compliance with regulations and measured for impact, as Tony suggests.
12. Designers will need to rethink their USP
While the “robots are coming for your jobs” panic isn’t exactly true, new technology will undoubtedly change how and what we work on. And Brian suggests designers might be impacted heavily.
Generative AI will prove to be as good or better than human designers, will allow for unlimited iterations, will deliver changes instantly, and will never get grumpy about criticism. Human designers will turn to behavioral research, data analysis, and experimentation to regain their ‘human’ advantage.”
13. Diversity improves experimentation, and more companies will finally catch on.
Diversity in the workforce is something every business should be implementing without incentive, but there are also measurable benefits, too, as Lucia has found.
As more companies and hiring managers push for greater diversity and inclusion across experimentation teams, we can expect digital experiences that cater to a broader spectrum of people.
Sadly, the trend of “synthetic data” might counter this positive development, as Jason Patterson, Founder of Jewel Content Marketing Agency, mentioned. Synthetic data is artificially generated data that could be used in place of real user research. Such data is trained on an initial data set, which could have inherent biases and not be truly representative.
14. We will need to offer greater support for the next generation of practitioners
It’s so easy to get caught up in the tools, tech, data, and processes of experimentation. Despite all the predictions for AI and machine learning in the coming year, we need to ensure we don’t forget the people who make it all happen.
Content (optimization) is king
15. Prioritizing genuine customer feedback & messaging will be top priority
In response to the emergence of synthetic data and AI-generated digital experiences, Loredana expects a renewed focus on customer-focused copy.
Companies ahead of the game will recognize the need to experiment with messaging that resonates most effectively with their customers and is based on real feedback.
This shift will place importance on customer-focused copy that is benefit-driven rather than feature-driven. There will be a strategic focus on aligning more closely with customer preferences and testing buyer psychology in copywriting on landing pages and websites.”
Similar to Loredana’s prediction, Nima sees content optimization as a big trend for 2024.
Top 5 predictions for experimentation in 2024
2024 is going to be anything but quiet. Here’s a summary of the biggest trends likely to impact our industry in 2024 according to 23 industry leaders;
Trend 1: Experimentation will continue to mature
- Experimentation will extend across all teams.
- Self-serve models will drive more teams to experiment.
- CRO will be seen as decision science.
- Increasing industry maturity will change who is experimenting.
Trend 2: AI & machine learning will disrupt all areas of the experimentation industry
- 2024 will be the year of advanced customization.
- Innovations will focus on privacy-preserving personalization techniques.
- The future of the web as a prompt machine.
- Companies with solid data foundations will excel with AI.
Trend 3: Experimenters' relationship with data will get tougher
- No more cookies means upping the ante on zero and first-party data.
- New legislation will impact our relationship with data and tools.
Trend 4: Experimentation teams will need to focus on their human advantage
- Employers will look for AI & ML tool competency.
- Designers will need to rethink their USP.
- Diversity improves experimentation, and more companies will finally catch on.
- We will need to offer greater support for the next generation of practitioners.
Trend 5: Content (optimization) is king
- Prioritizing genuine customer feedback & messaging is top priority.
A massive thanks to all of the experts who shared their predictions with us.
1. Ruben de Boer, Lead Conversion Manager at Online Dialogue
2. Sai Maddali, Digital Optimization Analyst at URBN
3. Kelly Anne Wortham, Founder of Forward Digital and Test & Learn Community (TLC)
4. Michiel Jansen, Head of Global Conversion Centre of Excellence at ING Bank
5. Gary Viray, Founder at Propelrr
6. Shiva Manjunath, Senior Web Product Manager, CRO at Motive
7. Deborah O'Malley, Founder GuessTheTest
8. Manuel Da Costa, Founder at Effective Experiments
9. Sina Fak, CEO of Conversion Advocates
10. Koutheir Somai, Analytics & Optimization Project Manager
11. Rohan Singh Rajput, Senior Data Scientist at Headspace
12. James Flory, VP of Delivery at Conversion
13. Nils Koppelmann, Founder at 3tech Gmbh
14. Slobodan Manic, Host at nohacks.show and growth strategist at Maksimer
15. Lucas Vos, Senior Conversion Specialist at RTL Netherlands
16. Tony Grant, Head of Marketing - Asia & Performance at INFINOX Global
17. Brian Massey, Conversion Scientist at Conversion Sciences
18. Lucia van den Brink, Lead Consultant & Founder at Increase Conversion Rate.com
19. Jason Patterson, Founder of Jewel Content Marketing Agency
20. Loredana Principessa, Founder at MAR-CO
21. Nima Yassini, Director & Co-Founder at White Maven
22. Tracy Laranjo, Fractional Head of CRO & Experimentation at Aftergrow
23. Eddie Aguilar, Founder, Blazing Growth
