How to put analytics front-and-center when optimizing your customer experience

This interview is part of Kameleoon's Expert FAQs series, where we interview leading experts in data-driven CX optimization and experimentation. Brian Massey is the Managing Partner at Conversion Sciences, a CRO consulting agency that focuses on actionable insights that boost conversions.

Q: You previously said that, as experimenters, we don’t deal in tests but in ideas. As someone on the agency side, do you find that organizations are generally on the right track with the ideas they want to test, but fall short on execution? How can you come up with more effective test ideas?

There are a lot of ways to improve the effectiveness of a website. However, most organizations don't have a disciplined way to save and order their ideas. So, the boss comes up with an idea, and that is the one that is tested. Because she's the boss. Or there is a monthly brainstorm and a small number of people decide what to test next. These are great ways to come up with ideas but terrible ways to decide what to test.

Too often the most effective ideas were expressed and then forgotten. We recommend a simple spreadsheet for collecting and ranking ideas. We give ours away for free.

Q: We talk a lot about the importance of data science. You call yourself a conversion scientist. You wear a lab coat! But what if a team with zero quantitative background (of marketers, for example) wants to start A/B testing. Is it a lost cause to do it in-house? Where should they begin?

Conversion optimization requires some specialized skills. As social animals, we are used to making decisions based on the behaviors of others, which is the definition of behavioral science. Most of us have chosen products based on their star rating. Yet, we know that there should be a good number of reviews before we can believe that rating. Thus, we have a basic understanding of sample size. So, no one is a lost cause.

But conversion optimization is not something that can be done on the side. The biggest issue with conversion optimization is that the effort dies out before the team gets competent. This is a skill set that you must eventually have in-house if you are to survive in a digital marketplace. But I recommend starting with an agency that can get positive results before the program loses steam.

Q: Garbage in, garbage out. As the world shifts toward data privacy—e.g. Apple ITP, death of third-party cookies, GA4—many brands are worried that it will be harder to obtain useful user data. How do you reassure them that this is an opportunity?

We are going to have to change the way we use data, primarily because of the death of browser cookies. Yet, there is not, nor has there ever been a more rich source of behavioral data than that provided by websites and apps. So, we are going to have to incentivize our visitors to let us watch them as they solve problems on our website.

It's not so much that the data is going away, but that it is going to get more expensive. Technology will also serve to provide this data in an ethical way. GA4 is actually a response to the privacy issues. It uses machine learning to help identify visitors who may not have cookies enabled. The tools will change. The way we run experiments will stay mostly the same.

Q: How do you keep stakeholders and the highest paid people in the room focused and interested over time in the absence of quick wins?

There are three outcomes of experiments: win, tie and save. It's important to celebrate the wins, as an experienced team expects one in four AB tests to be winners. A tie can be seen as "insurance" against a design change lowering conversion rates. The save means that you avoided launching a stinker of a design that would have hurt sales.

Whenever someone comes to you with an idea, they need to know that it will be added to your list and evaluated for its potential impact. After a certain time, the problem shifts. Soon the organization will want you to experiment on everything they do.

Q: Are there any unique aspects of how you approach doing CRO for e-commerce stores versus SaaS companies or consumer brands?

The playbooks for ecommerce and SaaS websites are different, but the approach is the same. Most ecommerce sites sell multiple products, so we are often very focused on "findability." For SaaS companies, we have the advantage that a prospect can try the product before they buy. Yet, we are running two kinds of tests for each: Tests that increase transactions and tests that tell us something about our audience.

Q: You’re in charge of a Fortune 500 company and are tasked with creating a CRO dream team led by 3 people, but you only know their backgrounds and job titles. What kind of people would you pick? What skills would you look for?

• A world-class project manager with the ability and interest to learn analytics, experimentation, and statistics.
• A world-class front-end developer who feels humiliated if her code has even one bug.
• A designer who has done enough work with data to have had his ego substantially squashed.

Can your store can increase conversion rates and provide better customer experience with an advanced experimentation tool like Kameleoon? Book a demo call to find out.

Kameleoon is an advanced client and full stack A/B testing and personalization tool with Shopify Plus integration to make customer experience optimization easy for eCommerce stores. Its unlimited, flicker-free A/B/n testing, AI personalization and real-time data reporting help mid-size and enterprise brands create world-class online shopping experiences.