Everything you need to know about Mobile App Testing with Booking.com
62% of the members of Jorden Lentze's LinkedIn community (Jorden is a Senior Product Manager at Booking.com) say they are already experimenting on mobile. Are you? Would you like to get started? We've invited Jorden to present Booking.com's Mobile App Testing strategy and detail the main steps involved in Mobile App Experimentation.
Booking.com, a pioneer in Mobile App Testing
With 100+ million active users and 1.5+ million conversions per day, Booking.com's mobile applications now account for almost 40% of all tests launched by the brand on its digital platforms and generate 50% of total conversions. These figures mean that the company has access to large sample sizes in order to carry out numerous tests and to structure itself into teams that are increasingly specialized in the different paths and elements of the application.
Not all of Booking's digital interfaces are the same. Experimentation started small, focusing on the hotel applications side with 1,500 to 2,000 visitors a day. The first mobile experiment was launched in 2011, and then the company saw a real ramp-up in Booking's experimentation strategy in 2018, shortly after Jorden's arrival.
Currently, 60 teams are working together on the brand's two mobile apps. Thanks to their optimization campaigns and regular traffic gains, the teams were able to split the customer journey into well-defined stages and refine the optimization of the user experience thanks to increasingly precise expertise. Booking.com's strategy is organized around 2 product release cycles per week.
Why has Booking.com opted for mobile experimentation?
Jorden explains that the rise of mobile experimentation is, above all, guided by the behavior of Booking.com users:
Experimentation also makes it possible to limit the risks of adversely impacting the customer experience during releases. Features are tested and optimized to avoid bugs before being offered to the audience, and releases are more secure.
Jorden also says that experimentation is essential for validating if improvements to the visitor experience are actually relevant and helpful.
What types of tests should be run on mobile?
Measuring the impact of the smallest changes
Experimentation is a virtuous cycle: the more tests you run, the more you understand your product and learn how your customers behave, and the more traffic you generate, the more tests you can run. This way, you can observe the impact of smaller and smaller changes more and more accurately. The slightest change, however imperceptible, can have an astonishing impact on the user experience.
For example, Jorden and his teams observed a surprising negative impact on their business following the addition of a small icon. The icon seemed quite harmless and was suggested by designers to visually represent 2 adults next to the sentence stating that the price is ‘for two adults’.
How can a company with 10% of Booking's monthly traffic test details of the page, such as icons, without waiting for months for results?
If you want to run similar tests, there are several options:
- Increase the test time and reduce the MDE (Minimum Detectable Effect).
- Run a simple user test.
- Don't make the change if you can't measure its impact.
- Use a statistical algorithm that allows you to work on a small sample, such as CUPED
Jorden Lentze: “Low traffic reduces the scale of the effects you can measure. If you don't have enough traffic, concentrate on bigger changes. The example given above shows that even small changes can have an impact on your users' experience and your business objectives. The idea is to encourage you to keep an eye on every change you make, but you probably already have enough things to improve, without needing to go into such detail. If the problem is related to traffic and not resources, you can also run several tests simultaneously, but this is a rather advanced practice.”
Check the relevance of each element of your app
Each element of your page or screen must serve your objectives. Jorden, for example, has challenged its teams on a carousel of offers, which he wants to ensure is of value before investing in improving it.
Iterating to launch fine-tuned features
One of the main benefits of Mobile App Testing is the opportunity to progressively develop new features for a safer and truly value-generating launch. Jorden and his team recently launched a room customization feature, allowing users to choose the type of bed they prefer.
Managing different operating systems
One of the unique aspects of mobile experimentation is the need to manage the two main operating systems: Android and iOS. Here, Jorden and his teams are challenging the relevance of dark mode on Android. This feature is essential on iOS, as it is a system-wide appearance parameter. On Android, it can be defined at the application level.
Are there any differences between iOS and Android in the results of A/B tests?
Jorden Lentze: “Yes, there are generally many differences. To balance things out, you need to start by identifying whether they are due to an implementation problem, a bug, design differences inherent in Android, or iOS or intrinsic differences in user behavior. They are not necessarily the same persona. Several studies show, for example, that Apple users have higher conversion rates, on the average, than Android users.”
If you see significant positive results on iOS and no difference on Android, do you deploy on Android to maintain parity in experiences or do you continue to iterate on Android?
Where there are differences between platforms, it may be a problem of implementation. Depending on the implementation, you won't see the same effect. It may also be due to design differences, which are quite marked between iOS and Android.
Jorden Lentze: “As far as parity is concerned, unless it's a question of price, brand image or a key feature, for me that's not the objective. We have, for example, dropped filters on Android that worked very well on iOS, but not on Android. You always have to make sure that you're really adding value. If it only added value on iOS, I would only deploy it on iOS and not on Android. Most people don't switch between iOS and Android and the few that do probably use the same account on different phones.”
What are the differences between Web Experimentation and Mobile App Experimentation?
There are, of course, differences in the way an organization approaches experimentation on these two channels. The technological environment for websites and mobile applications can be very different, which generally creates more constraints for experimenting on mobile apps.
Should you adopt a strategy for each type of device?
Fixing a buggy feature is more difficult on a mobile application, because it requires submitting a new version of the app to the Store, waiting for its approval, and then relying on users to download the new version, which may not happen for several days. In contrast, on a website, a published patch is immediately available to all users.
To limit the impact of this problem, some experts suggest completely separating the two types of experimentation on two different platforms. For Fred de Todaro, Kameleoon’s Chief Product Officer, there is no single answer to this question. The decision depends on many factors, such as the customer journey, the main objectives of the experimentation strategy and, of course, the way your teams are organized. You don't want to risk complicating the user journey with inconsistent messages or offers between web and mobile, so most of your tests should be designed to be cross-device.
The key is to recognize the unique aspects of each platform. There are bound to be differences in user behavior between web and mobile, but also between iOS and Android.
Is it as easy to set up a test on mobile as on desktop?
The implementation of tests is necessarily less flexible on mobile applications than on a website.
On a web application, you can run tests yourself using client-side technologies, such as the graphic editor, and easily create layouts, popups, and banners, without having to involve a developer.
These highly flexible tools do not exist for mobile applications. You generally have to carry out feature flagging, then Feature Experimentation, and you have to push your code along with your application's release cycle. Bug management is more complex and requires more resources. You'll need dedicated iOS and Android developers, and you'll need two platforms for publishing.
The unified user view: an advantage of Mobile App Testing
On mobile, you will definitely have access to a unified view of the user journey. With mobile ID and no need for cookies, you can track all the interactions and behaviors of users from different touchpoints and compile them into a single coherent dataset. As traffic is generally quite high on mobile applications, your data is more reliable and you'll have a better understanding of the preferences and patterns of your mobile users.
Different technological environments: striking the right balance
As mentioned above, while you can publish tests at any time in a web environment, you need to follow strict publication sequences on mobile. Tests must be pushed into your application updates, and special attention must be paid to quality of the tests, which must be bug-free, as no immediate action will be possible to resolve them. This requires meticulous planning, organization, and structuring of tests.
Performance is key
Performance is essential on a mobile application. Users may be in difficult areas with poor Internet connections. So you need to take this into account and adapt.
Jorden’s team therefore set out to measure the minimum acceptable speed for each page, i.e. the lowest speed at which conversions are not impacted. The speed of the application was slowed down in 500-millisecond increments and the impact on KPIs and customer behavior was measured for each test.
Where to start with Mobile App Experimentation?
To embark on the Mobile App Experimentation journey, you need to master three fundamental principles of Server-side Testing and Feature Experimentation. Here they are, in the order of maturity, from novice to a pro:
Start using feature flags
While the primary aim of A/B testing is to test new ideas with users, experimentation also serves a second, equally important purpose: to manage the way in which changes, updates and new features are delivered to users. It's essential for you to understand the value of your features and their impact on your target users.
Start simply by setting up feature flags. This allows you to limit any impact on production. If your feature doesn't meet your expectations, you can simply delete it instantly.
Proceed to progressive delivery
Next, advance naturally towards the use of progressive delivery. When deploying, target a group of users to whom you will introduce the functionality with your release. You can then roll it out progressively by iteration, measuring the impact on your key indicators and performance before making it available to all users.
This approach will also enable you to prove the value of this practice to your managers, and then you can start introducing your first product experiments.
Prioritize
Obviously, not every idea or hypothesis needs to be tested. Make sure you test in those cases where you have no data or proof that your feature will have a positive impact.
The aim of AB testing is to confirm that a new idea is a good idea. So you need a good balance between all these strategies: feature flagging, progressive roll-outs, and Feature Experimentation, and you'll do great things with your mobile application.
What is the minimum number of views per week or per month required to launch an AB test on an application?
Jorden Lentze: “In reality, the volume of traffic is not the only data to focus on. You also need to look at the volume of conversions you achieve through your objectives or metrics. It all depends on these 2 indicators. The good news is that you generally have higher engagement on mobile apps, which means you'll have a higher conversion rate and need less traffic than you usually do on a website. You can calculate the sample size required using this type of tool: https://cxl.com/ab-test-calculator//”
How do you select your ideas for AB tests?
Jorden Lentze: “We use binomial prioritization frameworks to determine the problems we need to work on first, and then to prioritize the ideas we are going to work on. At Booking.com, we have lots of ideas for solving problems or improving customer behavior. The problem is therefore to understand what to work on first, i.e. which ideas could have the greatest impact."
Fred de Todaro: “The best thing is to keep it simple. Test the features/improvements you want to launch when you have no proof that it will have a positive result."
Mobile App Experimentation: Which tool should you choose?
Here are a few basic features of an experimentation platform that are essential if you want to get started with mobile experimentation:
A real-time streaming architecture
Essential for instant updates. You should be able to deactivate a feature instantly, but also have access to accurate reports, updated in real time, so that you can view the data instantly, particularly when you are rolling out a progressive delivery.
The solution also needs to integrate with your ecosystem of tools. You need a data warehouse to manage data flows. Pay particular attention to the integration of your solution.
Consistent bucketing of your users
It is essential to guarantee your users a uniform experience across all their sessions and all their devices. The platform must be able to show the same variant to this user, whatever devices they use.
A powerful, collaborative, and intuitive platform
If you want your product teams to be able to conduct experiments, the platform needs to be intuitive and flexible. Some solutions will streamline the implementation of experiments better than others, with no additional development required from your team. Rather than asking your developers to code your test, they can simply code the functionality as before, and then you should be able to create variants of that functionality in your experimentation platform without any additional development.
Offline management
You must always be able to show the variant to users of the application and continue to collect data, even when users are not connected. This is essential for the consistency of their user experience.
