Six Groundbreakers Give Practical Advice on Using AI in Experimentation
The hype around AI has hit a feverish crescendo. According to Gartner, we’ve reached the ‘peak of inflated expectations,’ which explains why many people feel overwhelmed and unsure about how to really use AI in experimentation.
But it’s worth considering that within the next 2-5 years, technologies such as generative AI and decision intelligence will reach mass adoption. Today’s early adopters will get a head start on their competition.
With that in mind, and to ensure we stay firmly in the realms of what’s practical, we spoke to six AI groundbreakers about where and how AI can help in day-to-day optimization work, including a particular focus on how to balance human creativity alongside AI capabilities.
Here’s what they shared.
Delegating data analysis
A fully staffed experimentation team is the dream for many working in the industry. But even if you are lucky enough to have plenty of resources, you can still find yourself constantly busy. Johann suggests what tasks to delegate to AI;
While AI is known for lightning-speed data analysis on incomprehensibly large data sets, (right now) it can only take you so far. What’s needed is some of our uniquely human qualities, as Abhi explains;
Encourage your team to brainstorm and generate creative ideas. Humans can offer unique insights, empathy, and creativity that AI lacks. These ideas can be used to formulate hypotheses for testing.
Combine AI's data analysis capabilities with human-generated hypotheses. AI can help prioritize which hypotheses to test based on data, while humans can provide critical thinking and creativity in crafting those hypotheses.
Automate away the mundane tasks
Imagine a world where you could click your fingers, and all of those niggly little jobs were wiped clean off your to-do list.
Okay, automation doesn’t sound sexy, but you can’t deny taking busywork off your plate is an attractive prospect. Especially when you consider there’s no hiring process, training, or costs involved in getting your new AI intern up to speed, as Jim says;
A brainstorming sounding board
One area that several individuals mentioned was AI acting as a creative assistant to support you with the brainstorming and ideation process. The pertinent word being “assistant,” as Jim stresses;
So, how can you practically use AI to help you be more creative, whether designing variants to tests or coming up with new ways to solve user problems? Sim provides some ideas on which tools and prompts to use;
However, obtaining 10 different perspectives from different experts would be impractical and would result in discarding numerous brilliant ideas before they’ve had the time to develop.
Instead, I use ChatGPT to describe my problem and ask it to use the Six Thinking Hats framework to give different perspectives, or I consult expert perspectives on Delphi (an AI tool that “clones” the minds of experts).
Ultimately, creativity is limitless, and in order to leverage AI to enhance it, we must commit to continually evolving our understanding of it.
Creating good problems
Finding surface-level issues that users don’t care about won't move the needle. But complex problems that change user behavior or innovative ideas outside your local maximum can deliver massive improvements. However, as you provide advanced, personalized experiences, future problems become more complex. But some problems are good to have. As German’s recent experience shows,
By treating AI like a colleague in a whiteboard meeting, I’ve been able to put out more A/B testing charters, drastically increase the pace of making presentations to stakeholders, and create much more sophisticated problems for myself.
Problems akin to using click behavior and user profile information to augment website experiences in real time or having so many experiments in the pipeline the conversation for Level of Effort vs. Level of Impact is the most common weekly conversation.
Your AI assistant
It’s worth realizing the current limitations of AI to avoid getting swept up in exaggerated claims. No, you can’t ply it with realms of data and testing tool access and let it do all your work.
While there is a lot to get excited about as the technology continues to develop, it’s important to realize that AI currently requires human input;
It still needs a human touch to bring it across the finish line. Let's be honest, without this balance, the majority of people can spot what has been generated by AI. AI should be used as a means of brainstorming. Not your deliverable.
German put it wonderfully, “Don’t solely rely on AI but also don’t not use it. Let it be your colleague, the J.A.R.V.I.S to your Iron Man. Ask questions but question its responses, and brainstorm your own ideas.”
The future is just beginning
The current level of AI hype can be exhausting. But rather than disengaging, start focusing on small, practical ways you can use AI in your daily work, as the six groundbreakers above suggested.
5 ways to start using AI in experimentation
- Delegate data analysis tasks: let AI uncover the ‘what’ while you focus on the why.
- Automate away mundane tasks: from putting the next test live to updating the prioritization of new ideas based on the latest test results.
- Use AI as a brainstorming sounding board: Try using the Six Thinking Hats framework to get different perspectives or consulting expert perspectives on Delphi (an AI tool that “clones” the minds of experts).
- Create good problems, such as hitting test velocity thanks to AI-driven efficiencies.
- Have an AI sidekick: remember, while AI can help you come up with ideas, build structure, or debug problems, it still requires humans to review and check the output.
As you begin to understand where and how AI can assist you, it will be easier to stay up-to-date with this fast-moving technology. After all, things will only get more impressive from here on in.
Thanks to all the humans who contributed their thoughts and time: Ian Klosowicz, Data Analyst at DataDad.io, German Botello, Global Site Personalization Lead at Bose Corporation, Abhi Agarwal, Leader - Digital Transformation at IBM, Sim Lenz, Director of Experimentation at Conversion.com, Jim Sterne, President at Target Marketing of Santa Barbara, and Johann Van Tonder, CEO at AWA Digital.
