How to use AI for A/B testing

This article explores generative and predictive AI and how they can be used in experimentation. Whether you’re new to A/B testing or running a global experimentation program, this article will show you how AI can help achieve goals and overcome common challenges. Discover ten practical applications for AI in A/B testing, with real-life examples from various brands and industries. Understand what future trends are emerging in AI-powered experimentation and how to get quality outputs from AI while keeping your data secure.

What is AI-driven A/B testing?

AI-driven A/B testing refers to the act of applying generative or predictive AI into the A/B testing process. When generative or predictive AI is used in the A/B testing process, it is known as AI-powered or AI-driven A/B testing. AI can be applied to tasks and processes across all workflow stages.

Here are four key areas leveraging AI-driven A/B testing:

• Test ideation: AI-generated hypothesis or copy/design ideas for test variations.
• Data analysis and modeling: Building propensity models, analyzing test data, or quantitative and qualitative user research.
• Personalization: Running real-time predictive targeting or creating personalized experiences.
• A/B testing process: Summarizing key themes within large data sets, prioritizing the test backlog, or refining manual processes or workflows.

Generative and predictive AI are both forms of artificial intelligence based on machine learning. Generative AI produces new text, images, videos, audio, code, or data in response to a prompt. Predictive AI focuses on forecasting outcomes based on historical data.

Predictive AI has been used in experimentation to run personalization and propensity modeling for some time. But generative AI has exploded recently thanks to easy-to-access tools such as Google’s Bard (now called Gemini), DALL-E, Midjourney, and ChatGPT. Generative AI offers a range of use cases for experimentation, product, and growth practitioners, at every stage of the testing process — from creating scripts for user research to designing test variations and interpreting results.

Reasons to integrate AI with A/B testing

AI can help at all stages of the experimentation process, from research to test automation, as Craig Sullivan, a recognized AI and experimentation expert and founder of Optimize or Die, shared in our recent webinar.

The reasons for integrating AI within A/B testing vary depending on your current challenges. Here are some common scenarios, their challenges and how AI can help overcome them:

Large experimentation teams who want to scale

When companies are looking to scale, one of the biggest challenges is orchestrating multiple teams, sometimes across multiple countries. These teams need efficient workflows and streamlined processes to help increase test velocity. 

Increase test velocity 

Identify bottlenecks by asking AI to assess your workflow, processes, or project management data to identify sticking points. You could also ask for help with scheduling and planning resources across multiple teams to get work done more efficiently.

Another area where generative AI can help you speed up your processes is taking the busy work away from your team. For example, it can produce a range of reports, each customized for different stakeholders, or write a ticket for the dev backlog to build and deploy a winning test.

Move beyond low-hanging fruit and test more impactful ideas for bigger wins

As experimentation programs scale and mature, wins become more challenging to find. In these cases, many opportunities are still to be had, but they require more advanced data analysis, multiple types of user research, advanced technical implementation, and radical thinking. AI can help tick these boxes, from advanced data analysis, modeling, and predictive targeting to suggesting creative exercises to help you develop new test ideas.

Craig Sullivan recommends using AI to help with problem exploration. ChatGPT, and other generative AI tools, can provide a structured output of user issues based on the data you provide. With a hierarchy of problems identified, you can ask AI to produce ideas to solve each problem. What’s unique is that you can set it answers each problem, e.g., conservative ideas or wacky ideas. Starting with unusual or novel ideas can encourage your team to engage in radical thinking in ideation sessions too, helping you move beyond your local maximum.

Enable product teams to offer the best customer experience

Department silos create a disconnect between teams, which ultimately prevent companies from providing a top-notch (some call it ‘unified’) customer experience. Getting growth, marketing, sales, customer support, and product teams in sync when each has its own (often conflicting) work, goals, and objectives can feel insurmountable.

Draw insights from across the customer journey and departments

AI really shines when it’s tasked with finding patterns or themes in large datasets and triangulating data from multiple sources, which helps you to generate high-potential feature ideas. So, gather customer service transcriptions, product analytics, and paid ad data and get AI to analyze, synthesize and identity themes across datasets. You’ll quickly find recurring themes across the customer journey that might otherwise have been missed with a single channel view.

Craig Sullivan provides the following prompts to help draw insights from data:

Find commonalities between teams

Task generative AI with finding commonalities between team goals or metrics. Understanding how metrics influence one another can help you gather support for experimentation and customer experience initiatives across functions that might otherwise be a battle.

Offer the best customer experience

Most customers want personalized experiences, and with AI, it’s never been easier to create them. From real-time copy changes to images that resonate better, we can now use AI tools to produce hyper-personalized experiences.

Launch experimentation programs with new or limited resources

Getting an experimentation program off the ground without a dedicated team can be a grueling task. And in the early days, it’s usually down to a single person to show the rest of the business the value and impact experimentation can have.

Launch your first test

Creating a test often requires support from several departments, but until you have buy-in, you might have to fight for resources. Not anymore. You can call on AI to help you turn paper-sketched wireframes into high-fidelity designs, AI-generated test variants based on text prompts, and test results without the business analyst.

Cover any knowledge gaps

Without a full team of specialists covering everything from statistics to buyer psychology, it’s hard to know what you don’t know. Use generative AI as your copilot to guide you through the testing process. You can ask it to form a hypothesis based on research data, weigh up the pros and cons of different prioritization frameworks, or check for errors in your test setup. You can also train AI on specialist subjects by “feeding” it academic papers and training material, and then asking it questions as if it were your colleague.

Gain support for experimentation

When experimentation is new to a company, you need some early wins. Predictive AI can help you identify the best opportunities for a winning test and the best segments to target. Once you get your winning test, you can use AI to write a report to share with your stakeholders.

Examples of how AI can be used in A/B testing

Using AI in A/B testing for e-commerce

• Use AI to test personalized product review summaries like those used by Amazon.
• Test to see whether conversational commerce, virtual shoppers, and AI-generated product recommendations improve conversions, e.g., eBay’s ShopBot feature.
• Establish whether customers want virtual try-on features, e.g., Anthropologie and H&M using Google's “virtual try-on” technology. Or test 3D modeling and image generation to “try out” products vs. the cost to produce real product images.
• Copy tests using AI-generated product description as used by StitchFix or feature comparison information generated by AI, e.g., Taobao: Ask. 

Using AI in A/B testing for travel

• Test AI-generated travel itineraries vs. human written guides, such as the Expedia Project Explorer feature.
• Experiment with chatbots to help with customer service issues around the clock vs. human teams.
• Test the use of personalized homepages or whole experiences based on customer segmentation, such as Club Med.

Using AI in A/B testing for education

• Evaluate highly personalized lessons that adapt in real-time vs. structured curriculums. For example, Duolingo Max has features such as role-playing with AI-powered personas to practice language learning.

Using AI in A/B testing for entertainment, gaming, and social media 

• Test content created by AI across media formats and measure consumption and engagement compared with planned and produced content. This could be everything from full TV series, games, BuzzFeed quizzes, and music to social media avatars, filters, and videos.
• Test the relevance of personalized video recommendations, e.g., Canal+ with a personalized “more videos” feature.
• Test feature usage of AI editing and subtitling.

Using AI in A/B testing for healthcare

• Test user engagement with AI diagnoses via images or descriptions, e.g., AI patient triage used by Ada.

How to use AI in A/B testing

Below are ten real-world ways to apply generative and predictive AI in A/B testing:

1. Streamline customer journeys with AI-powered opportunities.

Most customer journeys produce endless amounts of data, making it increasingly difficult to identify problems and opportunities for optimization. But with AI, you can easily analyze swathes of data to determine where to focus your efforts, whether that's in your paid ads or your retention emails.

2. Generate data-backed hypotheses

Alongside human analysis, experimentation practitioners can leverage AI to increase the diversity of ideas, create research documents, analyze customer research, conduct sentiment analysis, and summarize key findings. This is often one of the most resource intensive activities in A/B testing, so there is big cost and resource savings to be made.

With tools like Sprig, for example, you can automate the creation of surveys and then use generative AI to synthesize the results alongside other behavioral data in real-time. This helps you create hypotheses based on the user problems identified.

What’s amazing here is the use of multimodal AI, meaning you can input different data formats for analysis. For example, audio from customer service recordings alongside product analytics data. Combined, you can then surface data-driven insights that appear across any dataset.

3. Automate performance-based test prioritization

Use AI to prioritize test hypotheses based on the potential impact. This adds an unquestionable level of objectivity to prioritization, as predictive models can be used to assess which variant is most likely to win. To do this, use AI to conduct a meta-analysis of previous test outcomes to identify where tests tend to produce positive outcomes. Prioritization is critical to ensure you are using your testing bandwidth on the biggest opportunities. So, using AI to help you prioritize your experiments can have a massive impact on the effectiveness of your program.

4. Make copy variations optimized for conversions

Unless you’re a skilled copywriter, creating copy tests can be tough. However, generative AI is built on LLMs (Large Language Models), which means it can produce natural-sounding copy that’s creative and contextually relevant. You can ask for your copy suggestions to be in the style of other brands or people, use different types of tone of voice, and translate them into various languages. Copy is a major factor in convincing customers to buy, persuading them of key features and alleviating their concerns, so using AI to help write better copy will help improve conversions and reduce abandonment.

5. Create hyper-personalized experiences

Combining predictive and generative AI takes personalization to another level. Based on user behavior and first party data, you can create hyper-personalized versions of your website in real-time.

For example, you can use generative AI to personalize product descriptions to address any concerns you know the user has, or write copy that speaks directly to their individual needs.

6. Discover high-value audiences

A negative test doesn’t always mean the test wouldn’t perform well for all of your users. When you investigate test results, you can often find that some ‘losing’ tests perform well for a specific sub-segment.

Use AI to help with this type of opportunity detection by looking at different sub-segments within the test’s audience. Seems like a small deal, but teams who conduct this type of AI opportunity detection find an average 15% uplift that would have otherwise been missed.

Additionally, AI can help with advanced propensity targeting, which finds segments to target for tests based on their propensity to convert. For example, Kameleoon provides web visitors with AI Predictive Targeting which can help improve conversions rates as Julien Descombes of Toyota explains;

7. Create AI-powered test workflows

Generative AI can help make decisions that move your test workflow forward. This includes helping produce test variations, putting tests live when other tests have completed, and analyzing and presenting the results once a test has concluded. While decision-making should still be augmented with humans, there are some binary or non-critical decisions where AI can take the pressure off your team so they can run higher-velocity programs.

Alongside this, AI can help you allocate resources to ensure your team, their time, and your testing bandwidth are used most effectively.

8. Troubleshoot test-launch challenges

AI can help you troubleshoot issues, such as finding and diagnosing errors found during the QA process. It can also help find issues in coded tests.

With generative AI, you can also use a conversational interface to provide instant answers to queries about setting up experiments or understanding results. This is particularly useful if you are scaling your experimentation program across your company. Typically, as testing scales, you will have individuals with varying levels of testing experience and knowledge. Depending on how big your core team of experimentation practitioners are, it can become overwhelming to try to manage all incoming inquiries and long form guides are often unread. In these situations, a trained AI assistant can help individuals across the business get answers to their questions instantly.

9. Support KPIs with AI-sourced test insights

Use AI to understand how your test results relate to, and impact, your KPIs and business goals. Generative AI can help you query experiment results using natural language to view what's behind the data. This is a particularly useful advantage for teams who struggle to get time with the company data scientist. Data analysis and interpreting the results inform the next steps, such as whether you iterate or implement a solution, so using AI to help here can have a big impact on overall test outcomes.

10. Improve A/B testing impact

If you’re looking to improve the impact and performance of your A/B testing program, there’s no better way than using AI to help you better predict, ideate, prioritize, launch, troubleshoot, and personalize your tests. Here’s more on the impact of AI on CRO according to 8 experts.

Future and emerging trends in AI-driven experimentation

With billions of people already using AI and many platforms integrating it into existing products, the technology is rapidly evolving.

Here’s what’s coming down the line with regards to AI and A/B testing.

Advanced predictability and intelligence

Advances in context, relevance, and accuracy of responses will make generative AI more reliable for decision-making. This, for example, might improve the reliability of sentiment analysis based on visual expressions in session recordings, helping to produce more accurate insights from a range of data formats.

Future advancements in predictive targeting and modeling capabilities will also lead to more intelligent optimization and personalization opportunities than are currently possible.

Native integrations

AI will seamlessly integrate into experimentation tools and platforms, as well as more integrations between systems and data sources to improve the outcomes AI can provide. It’s also expected that AI will be used to break down data and communication silos between teams to help all team experiments — a critical factor for peak performance.

Changing shopping behavior

AI will impact what and how customers buy. For example, rather than ecommerce companies offering customizable products, customers could use generative AI to design their own product from scratch, which could then be made-to-order, helping to reduce returns and inventory risk.

As Craig Sullivan suggests;

No matter what the future holds, one thing is clear, now is the time to start using AI in A/B testing.

How to ensure quality and compliance when using AI in A/B testing

When choosing to use AI within any A/B testing program, it's important to introduce checks and balances. AI is subject to hallucinations and there are potential data compliance risks depending on which tools you use.

Here’s how to ensure AI is compliant and producing quality outputs;

Ensuring data compliance for AI in A/B testing

Ask what happens to the data you input into an AI tool? How will it be used, recorded, and stored? Does this meet your data compliance requirements? Consider the nature of the information or data you input. Is it confidential, PPI, or publicly available? Get your legal and data team to review the tool policies for any security, privacy, and legal implications.

According to Craig Sullivan, if you are worried about uploading sensitive data to AI tools, there are several ways to tweak the settings (these suggestions are specifically for ChatGPT);

Be sure to ensure any A/B testing tool, including those which integrate AI, are compliant! Learn more about Kameleoon's commitment to ensuring data compliance at our Security Portal.

Integration into processes

AI tools should be seamlessly integrated into existing processes rather than treated as standalone entities, so think about how you will integrate AI into existing processes and workflows. Consider which processes could be adapted or replaced with AI, and which will need a hybrid approach.

Human quality assurance

Currently, AI output requires a human quality check. You can explore the concept of “alignment,” which involves sending the AI-generated output to another AI system to evaluate its alignment with the organization's values and standards. However, human QA is still strongly recommended. When it comes to outsourcing QA to AI, we’re still a ways off. While AI can identify and flag experiments with questionable logic, or spot errors in coded tests, full QA of test variations should be conducted by a human tester.

Craig Sullivan also provided some great dos and don'ts when it comes to prompts, which you should use to improve the quality of the output.

DO

• Ask AI to cite sources, books, and articles used to give an answer.
• Read the primary sources AI cites to ensure they exist and that they are relevant and credible.
• Feed and prime the AI with data, background information and context relevant to the task.

DON'T

• Ask for market statistics, rankings, ratings, or other quantitative metrics or comparisons.
• Ask for math, concrete facts or answers to research questions. Large language models (LLMs) are great with words, but are prone to hallucinations (although this has reduced since early 2024).
• Ask for quantitative counts for large amounts of test data. LLMs aren’t built to answer this type of question. Expect magic with one prompt. You’ll need to iterate.
• Trust what you are given implicitly.
• Use AI in place of UX research. It can’t replace the real voice of customer data.

The bottom line on AI integration into A/B testing

Generative and predictive AI should serve as complimenting solutions to your existing suite of A/B testing solutions. An effective AI integration should enable more streamlined workflows and higher rates of productivity, not replace that human touch. It’s this fine line that needs to be addressed when assessing how and when to use AI in your experimentation program.

With Kameleoon’s AI Copilot, you get the best of both worlds. Our AI-infused solution offers a practical way to harness AI for enhancing online experiences, with a focus on supporting real ‘jobs to be done.’ In that vein, we built AI Copilot to be more than just a tool - it’s an integrated, AI-driven approach to web and feature experimentation.

Learn more about how AI Copilot can help your team leverage AI to achieve more with your experimentation programs.