How to write AI prompts for A/B testing teams

Mastering effective AI prompts is crucial for maximizing AI's potential in experimentation. Because AI isn't just a tool—it's a valuable team member. And, by asking the right questions in the right way, you can significantly boost its speed, consistency, and quality.

Get 50 AI prompts for A/B testing here.

8 tips for getting the most out of your AI prompts

An AI prompt is a question or instruction you use to generate a response from an AI solution like ChatGPT, Gemini (formerly Bard), or Midjourney.

Here are eight tips for getting the most out of your AI prompts:

Include source information and context. Do you have links, text, data, or other material you can provide to give context to the question or problem? Make sure to set this out in your prompt.
Give the AI time to think. The AI may offer an incorrect guess if a task is too complex to complete in the allocated time. To avoid this, you can try “chain of thought” prompting. By providing a chain of thought prompt, you are modeling how to solve the problem, leading to faster and more accurate answers.
Figure I: Example of a chain of thought prompting. Source: Google Research
Break down complex instructions. Create a step-by-step instruction for AI to follow to reduce the chance of errors. You can start prompts with “Let's think step by step” to get the AI to approach the problem this way or specify the steps yourself. Use delimiters (triple quotation marks, XML tags, or section titles) to signify different sections of text and help you write more precise prompts.
Be clear on the output you expect. From tone of voice, length, and response style to the format, set out precisely what you want the output to look like. Don’t leave anything to guesswork. Providing examples of the output you expect can also help.
Repeat important information. AI models have a fixed context length, so the information within the chat isn’t held indefinitely. Therefore, you should reiterate relevant details. Similarly, if you provide extensive source material, the AI may stop without considering all the information. Use a follow-up prompt such as “Are there more relevant excerpts?” to ensure a response considers all the source material.
Iterate. According to HubSpot research, 55% of people said the most effective way to use generative AI was to experiment with different prompts to see what works best for specific use cases. Also, 43% recommended using follow-up prompts to expand on previous outputs. This is to say, don’t expect the first response to be perfect. Build on what worked and didn’t, and keep iterating until you find which prompt results in the output you’re after.
Fact check. As mentioned, AI models might not provide correct answers in some scenarios. They are also prone to hallucinations—fabricated or false information presented as fact. This is why your human team members should check all outputs. You can ask for citations to verify source material or ask for the reasoning behind an answer to help establish its accuracy.
Consider privacy and legal implications. For example, ChatGPT uses your data to train its AI model. This is unless you switch off ‘chat history & training,’ in which case the data will be kept for 30 days before it is deleted.

How to use AI in A/B testing

Here are some of the broader areas where generative AI can add value to your team - helping save time and generate better experiment ideas for more effective results.

Summarizing data and creating reports

You can save time doing busy work, such as inputting data into A/B test reports, so you can spend more time analyzing the results and what they mean for the business. Large language models are also very good at summarizing large amounts of data, which can be particularly useful for analyzing user research data.

AI gives you time to think. Do more thinking, less doing. I now outsource CRO analysis tasks to AI routinely. It saves time and, frankly, does a better job. That frees up mental space to think strategically. Work on the "why" of a test, and let AI do the "what."

Johann Van Tonder

CEO at AWA Digital

Exploring problems and guided thinking

You can ask AI to assume a specialist role, such as a data scientist, to help fill in any knowledge gaps you have and sense-check your approach or thinking. AI can also act as a sounding board by asking you questions that help you dive deeper into a problem. If you want to perform a new task for the first time, you can ask AI to guide or teach you how to approach the task.

As individual practitioners, it is important for us to understand our own creative process and find ways to enhance it using AI. I like to do this by seeking new perspectives on problems, as it enhances my creativity and allows me to connect new ideas.

However, obtaining 10 different perspectives from different experts would be impractical and would result in discarding numerous brilliant ideas before they’ve had the time to develop.

Instead, I use ChatGPT to describe my problem and ask it to use the Six Thinking Hats framework to give different perspectives, or I consult expert perspectives on Delphi (an AI tool that “clones” the minds of experts).

Ultimately, creativity is limitless, and in order to leverage AI to enhance it, we must commit to continually evolving our understanding of it.

Sim Lenz

Director of Experimentation at Conversion.com

Develop innovative ideas

When it comes to solving user problems, there are often hundreds of different approaches you could take. Sometimes, teams can fall into a rut, proposing the same solution over and over. To help you generate novel, innovative solutions, you can get AI to generate unconventional ideas.

Generative AI makes an astoundingly good creative assistant. The important thing is to remember that everything that gets published must, must, must go through human review first. With that in mind, we have yet to plumb the depths of the value of using generative AI as a brainstorming colleague. It is exhibiting new mind-blowing capabilities every day. Things are about to get very exciting.

Jim Sterne

President at Target Marketing of Santa Barbara

Finding opportunities

It can be difficult to spot opportunities with large data sets. However, this isn’t the case for AI models, which can predict behavior or identify new segments to target based on an individual's propensity to convert.

Use AI tools to analyze large sets of data quickly and identify trends and patterns. AI can help you uncover insights that might be difficult for humans to discern in massive datasets, providing a data-driven foundation for experimentation.

Encourage your team to brainstorm and generate creative ideas. Humans can offer unique insights, empathy, and creativity that AI lacks. These ideas can be used to formulate hypotheses for testing.

Combine AI's data analysis capabilities with human-generated hypotheses. AI can help prioritize which hypotheses to test based on data, while humans can provide critical thinking and creativity in crafting those hypotheses.

Abhi Agarwal

Leader - Digital Transformation at IBM

Coordinating and supporting teams

Teams often get assigned goals, which create friction or conflict with other teams. Ask AI to map goals across the company and highlight where they can be better aligned so everyone is pushing in the same direction. Additionally, AI can help teams share data and insights previously hidden in silos. Sharing this data helps all teams understand their users better and improve decision-making. For example, AI can help query current or past tests, informing experimenters if a similar hypothesis has already been tried or if there’s overlap with concurrently planned experiments.

AI makes a great additional team member. It can help bridge knowledge gaps in experimentation. Whether you’d like AI to act in a specific role, such as a data scientist, or answer questions on various topics, from how to set up complex experiments to questions about buyer psychology, AI can support individuals without needing knowledge bases or courses.

To help get you started, we’ve put together 50 AI prompts for A/B testing teams to use in their experimentation programs. Get the full toolkit here.