7 steps to start A/B testing your Shopify Plus store

The biggest mistake you can make as a Shopify store owner is to assume that customers will behave as you expect them to.

A/B testing your website without any insight and strategy however doesn’t always give you the results you want.

To be successful and learn from all your tests (even the ones that aren’t commercial winners), you need to understand which tests can help you achieve different business goals, and implement a framework for testing often, strategically and intelligently.

With a robust Shopify A/B testing framework and methodology, you can:

• Reduce shopping cart abandonment
• Maximize the value of your acquisition campaigns
• Increase sales and the average order value (AOV) per customer
• Align company messaging with customers’ values

Shopify recognizes the value of A/B testing to store owners. It has native integrations with different applications, including dedicated A/B testing platforms like Kameleoon, to make experimentation easier and more impactful.

In this article, you will learn:

• What Shopify A/B testing is.
• The benefits you can reap from testing on your Shopify Plus store.
• How to run Shopify A/B tests.

If you already do A/B testing, this article will also show you how to increase the variety and complexity of your testing—the two biggest factors in generating high ROI on CRO—using Kameleoon.

1 What is Shopify A/B testing?

Shopify A/B testing is an experimental process used by online store owners to compare the performance of two or more versions of a webpage and determine which one generates the most conversions or sales.

When conducting A/B tests, online store owners create two versions of a page differing in design, layout, copy, or other elements.

They randomly direct visitors to each version and compare the performance using metrics like click-through, conversion, or bounce rates. Results help owners make data-driven decisions and improve overall performance.

In Shopify, A/B testing can be conducted using built-in features or integrations with third-party apps, like Kameleoon. The built-in features are not particularly robust or comprehensive, so it is highly recommended to consider using a third-party tool for your A/B tests.

2 Benefits of A/B testing on your Shopify Store

There are many benefits to A/B testing on your Shopify store. When you run A/B tests, you can:

• Reduce cart abandonment. Find and fix issues that cause your customers to leave your checkout flow without completing their purchase.
• Improve the customer experience across your website. Minimize points of friction across your funnel to create a streamlined, user-friendly experience.
• Serve highly relevant product recommendations and content to your customers. Test different algorithmic recommendations or product messaging to deliver the right content at the right time.

All of these actions contribute to the biggest benefit, and number one goal of testing on your Shopify store - increased revenue.

When you fix issues that cause your customers to abandon their cart, you boost the number of customers who finish the checkout process.

When you fix friction across your customer journey, your customers enjoy a better experience on your Shopify store.

All of this can lead to an increase in sales and revenue.

3 Who should A/B test on Shopify?

Given the benefits of Shopify A/B testing, every store owner should be testing their Shopify website. However, this isn’t always possible. Shopify A/B testing is best for:

1. Stores with greater than 10,000 monthly visitors per page

To get a statistically significant result when testing on Shopify, the page you will be running tests on needs to get at least 10,000 monthly visitors.

It’s important to note that a specific page needs more than 10,000 monthly visitors - not the entire site.

If a page has less than 10,000 monthly visitors, tests will need to run for a long time to reach statistical significance. This may not be feasible for some kinds of tests, like client-side tests that rely on cookies to store customer data, as cookies are typically only kept for 30 days in most A/B testing tools.

It’s worth mentioning that Google will begin phasing out cookies by the end of 2024. As a result, A/B testing tools will need to find alternative methods of storing customer data.

Kameleoon solves this problem by using a server-side snippet that helps you identify returning visitors and bucket them persistently over time, even if your cookies have been removed. This will help ensure the highest data accuracy standards.

It’s easy to view A/B testing as something simple that anyone can do at the drop of a hat. However, running a comprehensive and effective A/B testing program is a lot more complex than simply testing button colors or a header on a homepage.

It requires time, resources, planning, and a deep understanding of statistics and data analysis in order to be successful.

Only those who are truly willing and able to commit to running a proper experimentation program should begin testing on their Shopify store. Otherwise, it’s possible that you’ll end up doing more harm than good.

If you are not able to currently invest the resources you need to succeed, there are optimization agencies and consultants who can work with you on implementing changes that will lead to increased revenues and better customer engagement.

4 How to Run Your First Shopify A/B test

Running your first Shopify A/B test requires a solid testing program you can refine and scale over time.

It starts with understanding the testing resources you have to work with (sample size being the main one), then determining the problem and the solutions you plan to experiment with. Here is a simple step-by-step framework to running an A/B test on your Shopify store.

1. Calculate your sample size

As discussed, the biggest hurdle to Shopify A/B testing is your monthly traffic.

To start testing, you need a large sample size that is representative of your entire customer base. While your website visitor numbers may cross the 10,000 monthly threshold, the traffic on the specific page you want to A/B test matters.

A lack of sufficient traffic to a page may increase the duration of your A/B test. In the case of insufficient traffic, it’s best to focus on surveys and focus groups to gather qualitative user feedback and implement changes accordingly.

Calculating the sample size you need to run split tests on the page is the first step. You can use the sample size calculator in the Kameleoon App to determine how many visitors you need for both variations in your A/B test.

2. Determine your business problem and create a hypothesis

Once you’ve determined that the page is getting enough traffic for your A/B test, you need to decide what business problem or KPIs testing on this page will address.

Maybe there are cart abandonment issues from a high-traffic product page identified through a funnel analysis. Maybe your recommendation section isn’t personalized enough for customers.

Study and critically assess your store’s data to identify issues you could solve with A/B testing on your Shopify website.

The data doesn’t always tell you the answer to your problem though.

Matt Abbott is the Head of Growth at Swanky, the largest Shopify Plus A/B testing agency in the UK. According to Matt, “You can get clouded by the data. And because Shopify stores have a reasonably consistent architecture, it’s easy to underestimate the impact clarity of messaging and design has on the user experience. If your products are right and your customers have a high determination to purchase, a customer will struggle through a terrible user journey.”

After identifying the issue, confirm what visitors are doing when they incounter that part of their customer journey and form a hypothesis.

Combine user experience platforms like Hotjar and analytics tools like Google Analytics to investigate user behavior. These tools will show you how customers behave when navigating your site. 

You can create several hypotheses based on these behaviors. A hypothesis is an educated guess as to the cause of an issue.

An example of a hypothesis for an ecommerce store could be:

• A more pronounced CTA will increase the number of customers adding products to their basket.
• Adding delivery information to a page will increase the number of customers who make a purchase.
• A more personalized experience will increase the number of purchases made from the product recommendation section.

Your hypothesis will then determine the type of test you need to run.

3. Decide which type of test to run

After you develop your hypothesis, you can decide what kind of test you’ll run on your Shopify page.

Options range from a basic split test to multivariate testing to testing algorithmic changes server-side.

Take the example above, where your hypothesis is, a more personalized experience will increase the number of purchases made from the product recommendation section.

In this case, the test you run will involve designing a variation where your product and personalization algorithms work together to deliver a better customer experience.

In another case, your hypothesis may be that removing the “add to cart” button (since your customers typically purchase 1 item per visit) would create a faster shopping experience.

In A/B testing your Shopify store, you can create a variation where the “add to cart” button is absent, then test it against your control where the button is present to see if it makes a difference to customer experience.

Which type of test you run will depend on the kind of data you’re using, the regulatory risk involved, ease of implementation, and many other factors. The image below gives you an idea as to how to think about optimization techniques.

Here, Matt provides some advice for how to approach designing your tests. “From experience, don’t design the alternative variation with a fixed mindset on what the solution will look like. Set the challenge to your design team and if they come back with three alternatives, test all three against the control (if traffic levels allow).”

4. Determine the segments, triggers, and data you will use

Your entire customer base may not be right for every experiment.

In one experiment, you may only need returning visitors to your Shopify store. In another, you may only want to target returning visitors who saw specific pages.

Determining which segment of your audience is right for your hypothesis is important before you run A/B tests in your Shopify store. Proper segmentation ensures you show the right experiment to the right customers.

You can target your segments further by including/excluding them based on the pages they visit.

An example could be to target only the French-speaking segment of your audience who visit your product pages in an experiment. Visitors from the French-speaking segment of your audience who do not visit any product page will not see the experiment.

Using segments and targeting conditions will give you data that is invaluable in helping you to make decisions for your business.

Matt continues, “Not only will it be important to ensure you are getting the language and tone right for your international stores, you might be surprised about what different markets react to.

I’ll never forget the results of a British English vs. American English test on an American storefront for a British company. The (sensible) hypothesis was that American spellings would be received better by the American market, but we were wrong, British spellings generated 13% more conversions. Proving that it’s good to test everything.”

5. Finalize your experiment

Now it’s time to put everything together. Log into your A/B testing tool and design your variations using the designated processes.

Add the variation you want to include in the experiment in the traffic allocation section and choose which segment of your customers you want to target with your experiment.

By default, Kameleoon splits the traffic on the testing page between your control and variation, but you can be more specific by choosing segments of your visitors for your experiment.

Simulate your experiment to check that all parameters are working correctly. Then launch your experiment.

6. Establish the duration of your test

Ideally, your A/B test should run for 2-3 weeks.

Depending on your traffic, the duration of your experiment may be shorter or longer than this standard period.

Leaving your experiment to run for longer will account for customers’ evolving behaviors.

There will be days when your traffic is normal and a good representation of your customers, then there will be other days where external factors like payday, holidays, weather changes, and major world events may spike or tank your traffic.

If you were running an experiment in March 2020 when the World Health Organization (WHO) announced the pandemic, you may have seen a traffic spike if your store sold groceries or medical essentials.

Conversely, your traffic may have tanked if your store sold luxury items, as customers focused on essential products.

Letting an A/B test in your Shopify store run for a proper amount of time will help balance out the highs and lows of traffic.

Having said that, running your experiment for too long will contaminate the data. If a tool is relying on cookies for testing, as is often done in client-side tests, then A/B tests that run past 30 days can pollute your data.

Apple's Intelligent Tracking Prevention (ITP) limits JavaScript-based cookies to seven days, causing A/B test results to be unreliable, which may lead to wrong conclusions and negative effects on the user experience, conversion rates, and revenues.

A server-side snippet, as used at Kameleoon, resolves this.

7. Learn from your insights

The major benefit of using a well-structured A/B testing framework for your Shopify store optimization is that every experiment can achieve something.

While it would be amazing for all your hypotheses to be true and your experiments to improve conversions or click-through rates, this is rarely the case.

Most A/B tests will not result in breakthroughs. In fact, as many as 70% of A/B tests “lose” from a commercial point of view. This is why you test — to find what works for your customer base and vice versa.

A better way to look at this is to imagine if you hadn’t tested your ideas in the first place. You would have implemented new changes that may have worsened the customer experience or negatively impacted KPIs.

While A/B testing your Shopify store, your variation may beat the control. In that case, you can start rolling out the changes to your audience.

In Kameleoon, you can gradually roll out your recent changes to your visitors. This way, you can track customer behavior to new features. And, if complex conditions arise, you can easily roll back new features.

If your control beats your variation, analyze the insights you gathered from your A/B tests. Store these insights and use them to inform future experiments.

If you believe a test is so successful that you wish to roll it out to 100% of traffic, you can do so with one click in Kameleoon. Often, this gives your dev teams time to code the changes in the backend.

5 How to scale your Shopify Plus A/B testing with Kameleoon

Kameleoon is the number one enterprise-level solution for Shopify Plus stores. It provides omnichannel, contextual, profile-based, rules-based, and algorithmic targeting, plus product/content recommendations and testing capabilities for customers.

Many A/B testing tools falsely claim to have an integration with Shopify or Shopify Plus. What they actually have is a script for ecommerce stores to install on their Shopify site, but these scripts tend to be buggy and may pollute the data you get from Shopify A/B testing.

Kameleoon’s Shopify integration is 100% approved and listed on Shopify’s App exchange. Kameleoon’s app makes scaling A/B testing and personalization for Shopify customers easy without involving multiple teams and monthly maintenance.

Set up Kameleoon with your Shopify Plus Store

You can set up Kameleoon directly from your Shopify Plus App.

• Click “Apps” in the left-hand corner and then “Customize your store.”
• Search for “Kameleoon.”
• Download and install the Kameleoon application from the Shopify App Store.
• Configure it using the steps outlined in this help article.

Kameleoon will automatically track the main ecommerce actions that take place in your store including transactions, product views, access to the cart page, and collection views.

Use Kameleoon’s Graphic Editor to build tests

Creating variations using the Graphic Editor is quick, convenient, and flexible.

Kameleoon’s graphic editor allows you to edit, delete and rearrange elements on a page without needing a developer. You can set up A/B tests with ease to find answers to pressing questions.

Here’s how it works.

• Log in to your Kameleoon App.
• Click on “New Experiment.”
• Select “In the graphic editor” from the two options shown.

• Choose which kind of experiment you’d like to create.
• You can start editing variations immediately as Kameleoon automatically creates two versions for you.

• Allocate traffic to your variations.
• Choose segments and goals for your experiment.
• Simulate your experiment to be sure that all parameters are correct.

• Launch your experiment.

To learn more about using the Graphic Editor, you can refer to this Graphic Editor help article.

Run A/B tests using the Code Editor

Kameleoon’s Code Editor offers you the means to run more complex experiments and test server-side.

You can roll out experiments by editing the JavaScript and CSS code directly in your variations. You can also A/B test using our Hybrid Experimentation capabilities, where you have the ability to run server-side tests with client-side capabilities.

To test using the Code Editor:

• Log into Kameleoon, and click on “New Experiment.”
• Select “In the code editor” from the options shown.

• Select what kind of experiment you would like to run.

Follow the steps in Kameleoon to design your code-based experiment and run more complex, server-side tests on your Shopify store.

To learn more about using Kameleoon’s Code Editor, review our documentation on using Kameleoon’s Code Editor.

Conduct feature experimentation on your Shopify store

As your testing program progresses, you may want to run more technical experiments, like testing which features your customers respond to the most. This is where feature experimentation and feature flagging come into play.

While technically advanced, using feature flags with your Shopify store is very powerful. It can enable you to manage features and release them progressively to your visitors.

Usually, to run feature flags or feature experiments, you will need to install an SDK on your application server. However, this is not easy to do in Shopify because you do not own the application server.

However, Kameleoon still offers an alternate way of running feature experiments client-side via our JS Activation API. This JS activation API works in almost the same way as a JavaScript SDK. It gives you the ability to get into your Shopify liquid files and make changes there directly.

To use this feature in Kameleoon, you can follow the steps above for creating an experiment in the Code-Based Editor and choose the “Hybrid” experiment type.

This article will take you through, step-by-step, how to use feature flags with your Shopify store to run feature experiments.

Identify elements that influence conversions with multivariate testing

With Kameleoon’s multivariate testing, you can test multiple variations of an element at one time to narrow down which elements of your website influence conversions.

Testing different combinations of website elements will help you find elements driving conversions and work to improve those areas of your website that aren’t working so well. This advanced testing method can be incredibly impactful, helping you get to significant wins faster and easier.

In Kameleoon, login and click on “New Experiment”. Then click on “Multivariate (MVT).”

Create your sections for different elements like button color and CTA wording. Then create the different variations for each section. Allocate your traffic, simulate, then launch your experiment.

Create personalized experiences based on users’ behavior, device, and location

Kameleoon’s personalization capabilities allow you to create tailored experiences for different segments of your audience. This in turn can improve the experience of users on your site and increase conversions and other key metrics.

Let’s say your Shopify store has visitors from the UK, US, Canada, and France. You can create personalized experiences for visitors from each of those countries.

For example, you can create a banner for American visitors offering discounts in celebration of holidays like 4th of July and Labor Day.

In the same vein, you can create personalizations based on users’ behaviors and devices.

Use AI to scale your Shopify A/B testing and personalization

Kameleoon’s platform uses predictive algorithms to identify visitors’ intent and interest in different products to enable you to personalize their experiences.

Kameleoon Conversion Score™ (KCS) uses a scale of 0 - 100 to show a visitor’s probability of converting.

You can also personalize a visitor’s experience on the website after 15 seconds based on behavioral signals like product categories viewed, time of day, weather, or price sensitivity.

For example, you can reorganize the homepage based on a visitor’s KCS, allowing you to scale personalization for millions of visitors without involving different teams in your company or worrying about endless maintenance.

6 Example: How The Wine Collective increased revenue with Shopify A/B testing

The Wine Collective is an online store on the Shopify Plus platform that offers premium wines to consumers.

The company wanted to understand its visitors better and provide a world-class customer experience worthy of its high-end wines. Optimizing conversions for better revenue was another goal they wanted to achieve.

Larissa Enbridge, CRO Lead at The Wine Collective, turned to Kameleoon as her Shopify A/B testing app of choice to help drive these business goals.

Using Kameleoon, The Wine Collective ran experiments that increased their conversion rate by 5.3 percent. Understanding their customers and providing a better experience through insights from testing with Kameleoon led to an 18 percent decrease in bounce rate.

Find out how you can optimize conversion rates and provide a better customer experience like The Wine Collective by using Kameleoon’s Experimentation and Personalization Platform with our success story.

Kameleoon is the only optimization solution with web, full stack, and feature experimentation capabilities in a single unified platform. Its powerful Shopify Plus integration makes customer experience optimization easy for ecommerce stores. Its unlimited, flicker-free A/B/n testing, advanced AI personalization, feature experimentation capabilities, and real-time data reporting helps mid-size and enterprise brands create world-class online shopping experiences.

To find out if your store can increase conversion rates and provide a better customer experience with an advanced experimentation tool like Kameleoon, book a demo call.