Checklist: How to choose your A/B testing solution

January 28, 2020

Reading time:

8 min

A/B testing is now widely accepted and used by brands looking to driving constant improvements on their website. It’s THE conversion rate optimization practice that lets you base your UX decisions on exact data analysis, rather than guesswork.

However, while it common practice among big-name brands, the way each company manages its tests and the functionality it requires can vary. Essentially every brand must find the technological solution that is most appropriate for them and that will allow them to achieve their goals within their specific environment.

At Kameleoon, we’ve assisted over 450 brands with their A/B testing and CRO strategies since 2012. Based on this experience, we’ve drawn up a list of the main criteria brands should consider when choosing their testing solution.

In this article, written in collaboration with our expert consultants, you’ll find a checklist of these features. This will help you choose your A/B testing solution and find the right technology partner to support you in your digital conversion rate optimization projects, resulting in increases in online revenue and customer engagement.

5 key stages for A/B testing

To build our checklist, we broke down the A/B test implementation process into five stages that link to your UX optimization strategy.

Stage 1 : Setting your goals and your variants
Stage 2 : Preparing your tests
Stage 3 : Analyzing your results
Stage 4 : Ensuring you have the right support throughout the process
Stage 5 : Planning for future optimization: personalization and AI.

For each of these stages, several features are essential to successfully carry out your A/B tests and to obtain reliable results.

Want to download the checklist now? It's here.

1 Stage 1: Setting your goals and your variants

Any A/B testing strategy begins with setting your goals and variants. This stage is crucial for ensuring that your tests will be effective and target the right visitors.

Setting and configuring your goals

At the heart of the test configuration phase is setting your goal. What KPI are you going to measure to determine whether one variant is better than the other?

For example, you want to test two different wordings for a Call-To-Action on one of your pages. Before running the test, you should decide the goal to use when comparing the two variants. In this case you might select an engagement goal (the number of clicks on the CTA) and see which variant performs best for this.

Features to look for

Ability to select from a wide choice of goals (click on a particular page element, scroll par page, pages viewed, retention rate per page, time spent on website, number of pages viewed
The ability to create your own, personalized goals, such as around e-commerce transactions with the addition of personalized attributes (order amount, transaction ID, payment method, etc.)

Analyzing and segmenting your audience

To conduct testing on your website, you need to know your audience well. Your tests will not be carried out on all your visitors, and you need to be able to segment your visitors for certain A/B tests.

It is therefore important for the solution you choose to include a targeting engine capable of segmenting your visitors based on behavioral, technical or contextual criteria.

Features to look for:

Real-time collection of visitor data
Segment creation based on source acquisition (landing page, referring website, type of traffic - direct, paid-for, organic, social, referral)
Segment creation based on digital footprint (browser language, type of device, IP address, screen resolution, browser used, cookie value)
Segment creation based on browser history (New vs returning visitors, type of customer, etc.)
Segment creation based on external information (date, time)
Segment creation based on data provided by other solutions (integrations)
GDPR compliance: collected data should only be anonymous information linked to browsing that cannot be used to identify the visitor

Identifying the triggers to activate your experiments at the right time

Activating an A/B test can be contingent upon certain specific elements that evolve according to visitor behavior. We call these triggers.

Your A/B testing solution must include the most appropriate triggers for your needs.

Features to look for:

Generic triggers (mouse out, specific events)
Page triggers (number of pages viewed, time spent on site, presence of an element on the page)

2 Stage 2: Preparing your tests

Once you’ve set the goals for your A/B test, identified the target audience and when tests will be triggered, you need to build and launch the experiment itself. That means creating the graphics, setting the right parameters, even if these are complex, and managing your test launch.

Creating the graphic elements for your test

Your A/B testing solution must enable you to easily create variations of the graphics on your website that match your visual identity and that you can modify if necessary.

Features to look for:

An easy-to-use graphic editor for creating custom elements:

- The ability to browse on the website like a user and to activate the editor as an overlay on any page, in on- or offline mode.

- Automatic detection of element types (block, image, link, button, form) and contextualized menu with options appropriate to each element type.

- Access to a suitable editor for developers coding variations with CSS and/or JavaScript.

A library of ready-to-use widgets that can be easily adapted to your visual identity.
A simulation tool that lets you check all test parameters, such as the display of variants, targeting, KPIs and potential conflict with other experiments, before launching online for all your visitors.

Using advanced settings

An A/B test can be very simple, such as running an experiment around the color of the CTAs on your product information sheets. It can also be much more complex, such as a test on your entire conversion funnel or an experiment where the traffic distribution is changing and needs to be dynamically managed by your testing solution.

Your solution must allow you to manage all the advanced settings you’ll need for your tests.

Moreover, some brands want to run more technical projects or to manage large-scale projects internally with their development teams. In this case, it’s important to rely on a solution with a strong technology DNA. Server-side test management, which tests your optimization hypotheses on your back-end architecture rather than via your visitors’ browser, is a functionality that some clients can’t do without, and yet many tools don’t offer it. The absence of this ability can be a deal breaker, depending on your projects.

Features to look for:

Multivariate test management
Multi-site and multi-domain solution
Traffic allocation algorithm (linear or multi-armed bandit)
Test scheduling
Ability to estimate test duration
Ability to manage tests server-side with or without logic caching
Tools for developers (REST API and SDK library, developer portal)
Automatic application management "Single Page App" (React, Angular)

Managing and anticipating problems linked to A/B testing

If not conducted properly an A/B test may impact your website’s performance.

To prevent this you should choose a solution that has an optimized architecture for A/B testing.

Features to look for:

No flicker generation
ITP management allowing you to conduct A/B tests despite Apple’s reduction of cookie lifespan
Automatic management of user consent

3 Stage 3: Analyzing your results

Once the test is completed, you will have to learn from the results, both to decide what changes need to be made to your website and to update your conversion rate optimization roadmap going forward.

To do this, you need access to a clear, easy-to-understand and customizable reporting tool that lets you track the results that matter to you.

Features to look for:

A/B testing reporting tool with key performance indicators (calculation of conversion improvement, associated accuracy rates)
Choice of statistical and analysis methods for your tests (Frequentist or Bayesian, by visitor or by visit, by converted visit or for all measured conversions)
Visual tools to better understand your results

- Interactive graphics

- Ability to zoom in on a given period

- Personalization of reporting pages

Filter and breakdown by criteria
Comparative view of multiple time periods
Tools to facilitate report sharing (export, sharing by URL)
Creation of alerts based on business criteria
Decision support system
Integration of third party analysis solutions (analytics solutions, heatmaps)

4 Stage 4: Ensuring you have the right support throughout the process

It is vital to be supported through each stage of your A/B testing projects. Your support requirements are going to differ according to the maturity of your brand when it comes to testing and to the internal structure of your organization.

That’s why it’s important to work with a company that can adapt to your specific needs.

Criteria to look for

For more mature companies, the ability to be entirely independent (training via tutorials in the solution, permanent access to the helpdesk)
A dedicated Customer Success Manager assigned to your account, able to provide you with strategic support
Having a dedicated Technical Account Manager assigned to your account for more technical projects
The ability to collaborate with a wide network of agencies

5 Stage 5: Planning for future optimization: personalization and AI

A/B testing should be part of a more general optimization strategy. Therefore, when choosing an A/B testing solution, you should also be thinking about the next stages of your optimization strategy.

While A/B testing is crucial, your strategy extends beyond it, towards practices that create greater value. These include personalization, whether based on manual segmentation or on the real-time prediction of visitor intentions using AI.

There is a clear advantage in selecting an “all-in-one” solution for A/B testing that will enable you to manage your entire optimization strategy. It allows you to prepare for future activities and strategically prioritize your roadmap.