How can I create an A/B testing framework?

This interview is part of Kameleoon's Expert FAQs series, where we interview leaders in data-driven CX optimization and experimentation. Lucia van den Brink has worked in CRO for seven years, optimizing websites with millions of visitors. She currently works as an independent consultant at her company Increase Conversion Rate.com and part-time as a Senior Experimentation Strategist at Speero. 

What frameworks do I need to run an effective experimentation program? 

I’ll discuss some basics here.

Pre-test

To understand what your experimentation program might look like, you’ll have to be able to do bandwidth calculations. This allows you to figure out:

• How long tests should run for.
• How many tests can you run per page.
• What the estimated effect you can measure is (MDE).

This information helps you plan the cadence, velocity, and type of tests best suited to the website. You can use the Speero pre-test analysis calculator to do this for your business. 

Test Ideation

The next step is to gather test ideas. Ideally, these are backed by data and research insights but also allow yourself to develop ideas freely.

Take a walk or a shower to activate a default mode in the brain (Daniel Kahneman calls it System 1) that is highly associative and allows ideas to flow.

You can use my ideation framework to understand how the creative, associative brain works and how you can improve or validate your ideas with rationalization. 

Prioritization

You'll need to decide which test to run first when you’ve got a list of ideas (ideally from as many people as possible). For this, you can use a prioritization model. 

Start with simple models like ICE and PIE, but as your experimentation program matures, you’ll probably use PXL or other evidence-based frameworks. This will help you understand which tests can create the biggest impact.

Otherwise, you might start A/B testing the website footer with too much optimism. 

Evolve your prioritization from ICE or PIE to PXL with this framework.

Statistics 

Once you’ve run your tests, you’ll need to analyze the results. I think statistical models like Bayesian and Frequentist also qualify as frameworks, and I always suggest everyone be able to use both (or learn to analyze a test with both simultaneously). This will teach you the difference and similarities between the two statistical models. 

It’s also good to know about Sample Ratio Mismatch (SRM) - an error that can cause many false results, often due to technical problems or test setup.

There’s even a framework to determine your best next step if you’ve received a sample ratio mismatch error.

Why do I need A/B testing frameworks when there’s a test process in place? 

First, let’s establish the difference between a test process and an A/B-testing framework. 

A test process is generally the steps you take from A to Z to create an experiment.

A typical process could be: data/research → ideation → create mockups → design → development → test analysis → iterate, implement or move on (check out this framework by my colleague Shiva Manjunath to help decide on the following action). 

Now, this process can be a framework. We can label the steps, as I did, and create a visual image to make it into a framework. But labeling information always means leaving out information or shaping it. In this case, I used the label “development,” but this doesn’t really illustrate the development QA step, for example. 

So to me, a framework is just a concrete shape of information instead of an abstract process that’s not documented. You can also call it a blueprint or model. It’s important to have frameworks to make the process rigorous and repeatable across the business. 

What are the best A/B testing frameworks out there? 

If you think about it, the concept of an A/B test is a framework (and the concept of science too). So, where to start? I’d say: try to soak up as much as possible.

Below are a five notable examples. 

1. The strategic testing framework 

This framework helps you to create a strategy before launching into tactics. It includes:

• Metrics: these are objective-specific, and map to higher org goals intentionally. They help make the objective a SMART goal.
• Objective: written in a specific "How might we" format, and is derivative of the research insights ideally.
• Solution options: strategies and tactics that can effect the metrics of concern and thus the objective.
• Research sources: the objective and solution should be derivative of customer and market research.

2. The Research XL framework 

This framework helps you to start with data and insights before turning to experimentation. 

3. Learning, speed, and velocity framework

“Won’t running all those experiments slow down product development?” to answer this and many other questions about the difference and balancing act between learning, speed, and velocity, check out this framework from Lukas Vermeer. 

4. An evidence-based prioritization framework 

Ruben de Boer provides a partly automated, evidence-based prioritization framework with a double feedback loop to run better A/B tests and find more winners. You can follow these five steps to create your version of this prioritization framework. 

5. User research methods framework

I’ve already provided the Research XL framework, which helps to identify the range of research methods you can utilize. This framework by Emma Travis focuses on helping you determine which form of user research is best suited based on your research objective/business case, effort, and level of confidence (e.g. value) required.

How can I evaluate whether an existing framework is right for my organization?

There’s a short and simple answer to this question.

How can I develop an A/B testing framework?

I can only tell you how I do it. Here’s my approach. 

Step one

The most important thing for me is asking myself: how can I explain this to someone else? As I said, frameworks are information in a certain shape. 

Once I get a bit of a story in my head, like step 1…step 2, and so on, I leave it. I take a walk and let the framework outline stew for hours or days.

Step two

You’re trying to make a framework as concrete as possible from something that might be non-existent or abstract.

So, the next question I think about is: what would be a better shape for this information? A pyramid, a flowchart, or just a list of three bullet items? I play with the possible shapes in my head, try to draw (sometimes), and decide on one. 

Lucia, how do you balance writing novels and participating in the Karate World Championships with your day job? 

For many years I’ve worked for big corporations, and I’ve continuously warned myself not to only trade my time for money but also pursue whatever I feel I need to pursue simply because no one will do it for me. 

I trained in karate five times a week for several hours, alongside working 40 hours plus a three-hour daily commute. I did this for about five years. It’s not impossible, but honestly, the fact I was in my twenties probably helped.

I’m happy I did it. I quit a couple of years back after 15 years of practice, and I’m grateful for its lessons: the limits and possibilities of my body, how to improve the nitty gritty details like professional athletes do, prioritizing specific improvements over others, dealing with being nervous, and reaching peak performance. 

In an office job, we’re like walking heads. Our bodies don’t matter much. We’re expected to sit and think and talk sometimes while our bodily energy is wasted. It doesn’t need much, but stretching or walking has many benefits.

And then there’s writing, indeed. I have always felt a big urge to create and think on paper. Writing forces me to sit down with big subjects. Otherwise, I’d just run from party to dinner, to friend, without ever looking at what was inside of me. That felt empty at one point. 

Balancing the mind, body and soul is a daily quest for me, but I’ve gotten better at it year after year. I hope you will too.