How to write AI prompts for A/B testing teams

Mastering effective AI prompts is crucial for maximizing AI's potential in experimentation. Because AI isn't just a tool—it's a valuable team member. And, by asking the right questions in the right way, you can significantly boost its speed, consistency, and quality.

Get 50 AI prompts for A/B testing here.

8 tips for getting the most out of your AI prompts

An AI prompt is a question or instruction you use to generate a response from an AI solution like ChatGPT, Gemini (formerly Bard), or Midjourney. 

Here are eight tips for getting the most out of your AI prompts:

1. Include source information and context. Do you have links, text, data, or other material you can provide to give context to the question or problem? Make sure to set this out in your prompt.

2. Give the AI time to think. The AI may offer an incorrect guess if a task is too complex to complete in the allocated time. To avoid this, you can try “chain of thought” prompting. By providing a chain of thought prompt, you are modeling how to solve the problem, leading to faster and more accurate answers.

   

   Figure I: Example of a chain of thought prompting. Source: Google Research

3. Break down complex instructions. Create a step-by-step instruction for AI to follow to reduce the chance of errors. You can start prompts with “Let's think step by step” to get the AI to approach the problem this way or specify the steps yourself. Use delimiters (triple quotation marks, XML tags, or section titles) to signify different sections of text and help you write more precise prompts.
4. Be clear on the output you expect. From tone of voice, length, and response style to the format, set out precisely what you want the output to look like. Don’t leave anything to guesswork. Providing examples of the output you expect can also help.
5. Repeat important information. AI models have a fixed context length, so the information within the chat isn’t held indefinitely. Therefore, you should reiterate relevant details. Similarly, if you provide extensive source material, the AI may stop without considering all the information. Use a follow-up prompt such as “Are there more relevant excerpts?” to ensure a response considers all the source material.
6. Iterate. According to HubSpot research, 55% of people said the most effective way to use generative AI was to experiment with different prompts to see what works best for specific use cases. Also, 43% recommended using follow-up prompts to expand on previous outputs. This is to say, don’t expect the first response to be perfect. Build on what worked and didn’t, and keep iterating until you find which prompt results in the output you’re after.
7. Fact check. As mentioned, AI models might not provide correct answers in some scenarios. They are also prone to hallucinations—fabricated or false information presented as fact. This is why your human team members should check all outputs. You can ask for citations to verify source material or ask for the reasoning behind an answer to help establish its accuracy.
8. Consider privacy and legal implications. For example, ChatGPT uses your data to train its AI model. This is unless you switch off ‘chat history & training,’ in which case the data will be kept for 30 days before it is deleted.

How to use AI in A/B testing

Here are some of the broader areas where generative AI can add value to your team - helping save time and generate better experiment ideas for more effective results.

Summarizing data and creating reports

You can save time doing busy work, such as inputting data into A/B test reports, so you can spend more time analyzing the results and what they mean for the business. Large language models are also very good at summarizing large amounts of data, which can be particularly useful for analyzing user research data.

Exploring problems and guided thinking

You can ask AI to assume a specialist role, such as a data scientist, to help fill in any knowledge gaps you have and sense-check your approach or thinking. AI can also act as a sounding board by asking you questions that help you dive deeper into a problem. If you want to perform a new task for the first time, you can ask AI to guide or teach you how to approach the task.

Develop innovative ideas

When it comes to solving user problems, there are often hundreds of different approaches you could take. Sometimes, teams can fall into a rut, proposing the same solution over and over. To help you generate novel, innovative solutions, you can get AI to generate unconventional ideas.

Finding opportunities

It can be difficult to spot opportunities with large data sets. However, this isn’t the case for AI models, which can predict behavior or identify new segments to target based on an individual's propensity to convert.

Coordinating and supporting teams

Teams often get assigned goals, which create friction or conflict with other teams. Ask AI to map goals across the company and highlight where they can be better aligned so everyone is pushing in the same direction. Additionally, AI can help teams share data and insights previously hidden in silos. Sharing this data helps all teams understand their users better and improve decision-making. For example, AI can help query current or past tests, informing experimenters if a similar hypothesis has already been tried or if there’s overlap with concurrently planned experiments.

AI makes a great additional team member. It can help bridge knowledge gaps in experimentation. Whether you’d like AI to act in a specific role, such as a data scientist, or answer questions on various topics, from how to set up complex experiments to questions about buyer psychology, AI can support individuals without needing knowledge bases or courses.

To help get you started, we’ve put together 50 AI prompts for A/B testing teams to use in their experimentation programs. Get the full toolkit here.