How Intelligent Tracking Prevention (ITP) impacts A/B testing - and how marketers can overcome its restrictions

Can marketers trust their A/B testing tool anymore?

That’s the key question given the latest update to Apple’s latest Intelligent Tracking Prevention (ITP) technology, which is having an enormous impact on CRO and experimentation practices.

We first wrote about the problem that ITP introduces into the reliability of A/B tests back in July 2019 and have now updated this blog following the latest release of ITP in March 2020 which has spurred changes in our approach and technology.

While ITP is a very technical topic it can potentially have an enormous impact on marketers carrying out A/B testing experiments – and given that these are used to underpin decisions on your entire website presence, this goes to the heart of your digital marketing strategy. For those that want a less technical explanation take a look at our other blog on ITP here.

To find out more, we spoke to our CTO, Jean-Noël Rivasseau, about ITP, and in particular the latest version, 2.3 which introduces major restrictions on the storing of visitor information, with a 7-Day Cap on all Script-Writeable Storage, including cookies but also session or Local Storage.

ITP is implemented within the Safari browser, and while this has around 8-12% of the desktop market worldwide, it is a completely different picture when it comes to mobile devices, where Apple has much higher market share (closer to 50%-55% in the US and UK). This means that half of your web traffic could be impacted by ITP, along with a double-digit percentage of desktop visitors.

Here are the key points we captured from our conversation with Jean-Noël:

• What is ITP?
• What are the main impacts of ITP on A/B testing tools?
• How has ITP changed since 2019?
• How we overcome this issue at Kameleoon

1 What is Intelligent Tracking Prevention (ITP)?

Apple's ITP aims to increase user privacy by preventing unwanted tracking by websites and the various scripts installed on them, and is implemented in all Safari browsers. The latest version, an upgrade to ITP 2.3 in March 2020, essentially limits all first and third party cookie storage to seven days, as this post on Apple’s WebKit developer’s blog explains. This includes Local Storage, the previous method used to overcome restrictions on JavaScript-based cookies.

Capping all script-writeable storage to seven days is very aggressive. This is because since the introduction of ITP, most players (ad platforms, analytics providers) have been moving to first-party storage (using cookies or Local Storage) to overcome the initial constraints of ITP on third-party cookies.This is because almost all analytics platforms (including Google Analytics) need to store visitor’s unique identifier to track them on their journey.

This identifier is randomly generated on the first pageview of the first visit of a user to your website. It is then reused on all subsequent page views and visits to identify the particular user. For example, this allows Google Analytics to know if a visitor is new (as a new identifier/cookie has been generated) or returning (as it finds the cookie that has been set previously).

Given that such storage now has a lifetime of seven days in Safari, this overrides the much longer default expiry date normally set by a solution such as Google Analytics. So, a visitor making a first visit on Monday, then returning on Tuesday of the next week will be considered as a totally new visitor. Their new visit won't be linked to the previous one.

As a result your new visitor metric, as reported by Google Analytics, is completely false for any Safari traffic. Many other metrics (such as number of unique visitors, time to buy, etc) are also completely distorted. Essentially, it's hard to trust any number from cookie-based analytics solutions anymore!

2 What are the main impacts of ITP 2.3 on A/B testing tools?

The majority of testing and conversion optimization platforms also bundle web analytics capabilities within the tool. This means they’ll suffer from the exact same issues we just described, which is problematic in itself. However, for A/B experiments, an even more serious obstacle appears.

This is because each time a visitor is bucketed into an experiment, a variation is selected for them. This variation must be remembered, since if they return, they must see the same variation as before.

To achieve this all front-end testing solutions store this information (the association between experiment and variation) on the user’s browser. So, with ITP, a visitor coming back more than seven days after they initially triggered an experiment runs the risk of seeing a different variation. A/B testing in these conditions is useless and does not produce reliable results. It is the equivalent of getting in a car where the fuel gauge can’t be trusted to be accurate.

4 Why is Kameleoon the only conversion optimization platform that automatically and transparently handles Apple ITP?

At Kameleoon we are strong believers in the power of experimentation to deliver improvements to the digital experience, benefiting consumers and increasing engagement. Making decisions based on partial/incorrect data from your A/B testing platform goes against the whole point of running such experiments – you need exact results.

We’ve therefore been focused for some time on helping our clients to mitigate the impacts of ITP when carrying out A/B tests and personalizations.

Cross-domain Local Storage

Since the origins of Kameleoon back in 2012, we have been using Local Storage (LS) to store all A/B testing related data, and were the first to launch a cross-domain LS solution when ITP 2.1 was released. Local Storage, a standard Web technology supported by virtually all browsers in 2019, is basically a data store in the browser, so it can handle data in the same way that cookies do. This article outlines why we chose Local Storage and how we overcome the constraints that come with it. Unlike others we were able to implement a cross-domain Local Storage, catering for the fact that most sites have more than a single exact subdomain.

To explain what this means, let's imagine that most of your e-commerce website is hosted on https://www.randomshop.com, but the conversion funnel is on https://transaction.randomshop.com. If your visitor ID is stored on the LS, your analytics solution will report two different visitors when a complete purchase occurs, one seen on www.randomshop.com and the other on transaction.randomshop.com. Data stored in Local Storage does not transition consistently from a domain to another, and such a solution would see the as two completely separate sessions, not a single one of the same visitor.

For A/B testing, this is - again - even worse. If you run an experiment modifying the navigation menu, with two variations, one user could very well be exposed to the first variation on the main site, but to the second one down the funnel! This is of course disastrous in terms of user experience, while also voiding the results of the experiment.

So we went further and implemented a full cross-domain Local Storage implementation, as even a small percentage of pages on a different subdomain on your website can mean unreliable and even false results for A/B experiments.

Solutions from A/B testing providers

Clearly, the latest Apple ITP update imposes the same constraints on Local Storage as other first and third-party cookies. That meant every testing tools vendor that has moved to Local Storage needs to evolve their approach to ensure their clients benefit from accurate A/B testing and personalization data.

ITP does not set any restrictions on cookies created server-side by the website, as they are usually needed to run the site. So some vendors have chosen to create server-side snippets that allow the set-up of cookies server-side. However, this could have an impact on performance, depending on the number of cookies and the data stored inside each of them. Others have not gone beyond Local Storage, meaning that the data they collect will be unreliable on Safari, particularly even more widely if they do not have a cross-domain approach.

Alternatives approaches exist and brands can also:

• Add a DNS entry on their main domain to setup server-side cookies.
• Modify the configuration of their front facing web server (or CDN) to generate and add the required cookie.

You can learn more about all options here: https://developers.kameleoon.com/visitorcode-synchronization-methods.html

Synchronized server-side cookies and Local Storage

Instead, we have gone further. Our solution creates a snippet server-side that automatically synchronizes with Local Storage. Our current recommended solution is thus to install a server-side snippet that automatically synchronizes our kameleoonVisitorCode cookie between the front and the back end. This contains the very important visitorCode identifier.

ITP does not impose any restrictions on server set cookies, so this cookie will have an expiration date set sufficiently far in the future.

This snippet will create the KameleoonVisitorCode cookie server-side when no Kameleoon cookie has been found (i.e. it has not yet been created front-side) OR retrieve the existing value and recreate the cookie server-side to avoid ITP issues. The synchronization means that not only will identifiers not be removed after seven days, but there is no impact on performance or the user experience as we will only store a single cookie.

However, as Kameleoon stores other data in the Local Storage, data that will be needed to trigger real-time experiments with no extra server-calls, we have also implemented a Local Storage synchronization mechanism.

On Safari, once Kameleoon obtains its visitorCode by reading the (server set, so reliable) kameleoonVisitorCode cookie, it will check if its current Local Storage is empty. If that is the case, which probably means that the last visit was more than seven days ago, Kameleoon will perform a Server Synchronization Call (SSC) to fetch all the data that was present in the Local Storage from our own backend servers. Once this call is over, data will be restored in the exact state it would have been if ITP did not wipe it. Normal operations can then resume.

Although we can guarantee that our analytics core is unaffected by ITP, and that all our reported traffic data is reliable, of course we cannot guarantee the same for third party web analytics platforms. Our integration bridges with partner tools continue to work in the same way as before. We recommend that customers discuss the situation with these vendors directly.

You can read more technical details in our developer documentation, but essentially our solution means that you can run A/B tests with confidence on your entire visitor base, whatever browser they are using. It brings reassurance that data is accurate, enabling it to be used to make key decisions to improve and optimize the digital experience you deliver to each and every one of your customers.

ITP moving forward

We will continue to monitor the evolution of Apple's ITP technology, as well as other initiatives from other browser vendors (Firefox has a similar technology to ITP, but this is not yet as extreme). Any new changes will be swiftly analyzed and we’ll deliver solutions as soon as possible to enable our customers to retain confidence in their experimentation and personalization data.