A/B testing & personalization: do you trust your data and results?

Today’s digital marketing strategies involve rapidly understanding, analyzing and acting on large (and growing) volumes of data. Brands therefore have to ensure that they are working with trusted martech partners to help them achieve success. Showing the rising importance of data, for the first time in 2020, Gartner even added the concept of customer data ethics to its Hype Cycle, demonstrating that it is now a major concern for brands, consumers and regulators.

A/B testing and personalization solutions use visitor data to enable brands to make the right decisions to improve the user experience and deliver a personalized customer journey. Given its importance the quality of this data is paramount - it must be collected, processed and analyzed correctly if it is to deliver value. Otherwise, how can you be certain that the results of your experiments are reliable?

To answer this key question, we spoke to Jean-Noël Rivasseau, Kameleoon’s founder and CTO. At Kameleoon Jean-Noël has built a feature-rich A/B testing and personalization platform that combines high performance, reliability and security. Today he heads an R&D team of over 60 people, all dedicated to meeting the changing technical, functional and business needs of the biggest global digital brands when it comes to experimentation and personalization.

What are the top criteria when choosing your A/B testing and personalization technology provider?

“Brands invest in an A/B testing and personalization solution so they can make decisions based on reliable data and consequently roll out the best possible online experience to their visitors in real-time. So, in my opinion, the most important criteria is the solution’s reliability when it comes to delivering trustworthy results. Without this, you can’t have faith that your optimization practices are actually going to provide real improvements.

This may seem obvious but this point is often overlooked by decision-makers. Is your supplier collecting data correctly? How are they processing it? Is your reporting tool reliable?

These technical questions are vital, since they carry big risks for brands — data loss, skewed reporting and thus biased decision making — all of which has a financial impact. To me, it’s obviously vital for brands to have complete trust in their solution, because it’s a decision-making tool that has a direct impact on increasing conversion rates and therefore sales and revenues.

What do you mean by a “reliable” solution?

A reliable solution is one with a high degree of uptime, and that delivers A/B test and personalization results that cannot be disputed. This reliability depends on several upstream factors within the tool supplier, such as the skills of the team involved, the technology leadership and whether it is based on a robust, scalable architecture. It is important to think ahead - some decisions made several years ago may have a long-term impact, leading to an accumulation of technical debt that becomes unmanageable over time.

Besides technology choices, it is crucial to have the right people in place at your supplier to manage the tool’s underlying processing engine, and who have the experience and skills to successfully administer security.

Can you tell us more about how data is managed with Kameleoon?

Where data collection is concerned, our servers can manage more than 30,000 requests per second and we are completely scalable and flexible due particularly to our use of the Apache Kafka open-source distributed event streaming platform.

For data processing, again it’s our technology choices that make the difference when it comes to guaranteeing reliability to our clients. Thanks to these our platform boasts an average availability of 99.89%. We are also transparent about any incidents that may occur, via our incident page which is updated in real-time.

Moreover, Kameleoon guarantees the smooth running of our processing engine, with:

• A team of senior developers in charge of the engine itself
• Set-up of automated tests & incident simulation
• 24/7 monitoring activity to prevent any data loss

Finally, in the event of an error, our processes lead to very short response times, limiting the impact on our clients.

All these measures, which we constantly seek to improve, enable us to guarantee maximum reliability to our clients.

Are there other criteria to take into account to ensure you have reliable and usable data?

Brands must understand the constraints that are inherent to digital marketing. For example, the restrictions on cookie lifespan imposed by browsers, the flicker effect and script performance are all elements that have a huge impact on experiment results.

From this point of view, not all testing and personalization solutions are equal.

The first element, which we believe to be the most critical is that our visitor-data collection and processing technology enables us to adapt to the constant changes of Apple’s Intelligent Tracking Protection (ITP) changes and to other browser restrictions. Today, up to 30% of A/B test data can be incorrect if ITP restrictions are not overcome. Adapting to these changes is therefore a real technical challenge.

For more details, read our in-depth article on what ITP is and how we overcome it.

The second element is that we have a 100% anti-flicker script to ensure that a visitor has the same quality of experience, whether they are shown version A or version B in an experiment.

Finally, the last key point: the ultra-optimized size of our script enables us to provide brands with maximum execution speed, ensuring an optimal visitor experience.

In summary, you should choose a testing solution as you would a SatNav or GPS: if the route directions are wrong and if the results are too slow, then you’re better off not using it at all. That’s why the reliability of our data along with the performance and availability of our platform have always been our primary concerns. We’ve remained true to these principles since Kameleoon began as we know that our clients need to have 100% trust in our platform if we are to help them grow.”