6 Practical tips for getting started with mobile app testing

While you may have mastered the art of A/B testing on web experiences, mobile app testing presents new challenges. Screen size limitations, user behavior nuances, and platform-specific conditions demand a distinct approach. This article bridges the gap, leveraging your existing A/B testing expertise to unlock the secrets of mobile app testing. We'll delve into the key differences between web and mobile app testing and equip you with practical tips from seven mobile experimentation leaders. 

But before we dive into the ‘how,’ it’s good to understand why mobile app testing is useful. 

Why is mobile app testing useful?

Jorden Lentze, Senior Product Manager Apps at Booking.com, succinctly summarizes four core reasons to embrace mobile A/B testing:

1. Follow your customers: If your mobile app has a large number of users (or high-value customers), you should dedicate time and resources to optimizing that experience.
2. Risk mitigation: Making changes to mobile apps takes more time and resources than web experiences. Mobile app tests help you evaluate changes to ensure they will have a positive impact before they are fully implemented. 
3. Loss prevention: Identify any negative impacts from changes and avoid releasing bugs or bad UX into the app experience. 
4. Customer centricity: Continuously improve the experience and add value for your customers. 

The nuances of mobile app testing

While most of the core principles of A/B testing are consistent across web and mobile, the technical constraints of mobile apps necessitate different testing methods. For web-based experiences, tests can be deployed instantly. But on mobile, you're beholden to your development teams’ app release cycle, the app store approval process, and users updating their apps. 

Furthermore, website experimentation platforms typically include WYSIWYG editors and client-side technologies that empower non-technical teams to craft tests without developer support. But mobile app tests demand closer collaboration and, depending on your requirements, dedicated iOS and Android developers. 

It’s also important to remember that your users might transition between your app and website, so you’ll need to maintain a cohesive customer journey by working and aligning closely with other teams and considering designing tests that are cross-device.

While deploying mobile app tests might be more challenging than web tests, mobile apps typically boast more logged-in users. This translates to a more holistic view of user interactions and behavior across platforms, providing a richer tapestry of data and more reliable test insights.  

The other plus for mobile app testing is that while you might not have high traffic, apps generally boast higher engagement and conversion rates than websites, meaning it’s possible to achieve statistically significant results with less traffic.

Using feature flags to get started with mobile app testing

Rather than using A/B testing right away, you should consider using feature flags for mobile app tests.

Start deploying a change to a small set of users via feature flags. If the change you made related to the feature flag doesn't meet expectations, you can just turn it off instantly—giving you access to a kill switch that isn’t dependent on the app release cycle. 

If you see positive results, you can progressively roll out the change to more users while measuring the impact on your key metrics and performance. You'll be able to show the value of this approach to your managers and then make the final step towards running the change as a full A/B test. 

It’s worth remembering here that not all ideas need testing. A/B tests should inform decisions such as adding a feature or changing an experience but aren’t required for bug fixes or UX improvements generally.

To achieve the feature flag > progressive rollout > experimentation workflow, you’ll need an intuitive and easy-to-use testing tool that supports feature flags. Some tools will streamline the implementation of experiments better than others, with no additional development required, making it easy for non-technical users to self-serve. Ideally, rather than asking your developers to code an experiment, you’ll ask them to code the feature; then, the Product Manager can create the experiment using feature flags and test variants of that feature.

Six practical tips for mobile app testing 

In addition to the different technical approaches and methods used to conduct mobile app tests, there are also process-related tips that are valuable to know. Here are six of the best pieces of advice from seven industry leaders working in mobile app testing. 

1. Start Small and avoild the snowflake effect

No matter how mature your website-based experimentation program is, it’s wise to start your mobile app testing journey at a beginner level and then ramp things up toward more complex experiments. As Hannah suggests,

William also recommends a lean mindset when it comes to creating tests. Perfection is the enemy of experimentation, as he explains, 

But what does a simple Minimum Viable Test  (MVT) look like in reality? Jorden Lentze, Senior Product Manager Apps at Booking.com, suggests you could start by removing something from your experience rather than creating something new. To achieve this, you would need to put the element in question behind a feature flag and then deactivate it for a group of users. It’s easy to get your first mobile app experiment into production.

2. Track the right metrics

The whole point of experimentation is to gather data to inform decisions, so you have to ensure you’re tracking the right metrics and have access to the necessary data before you start your first mobile app test. It’s also crucial to define what success looks like before you begin to avoid bias analysis; as Hannah explains, 

Umashankar expands on this, 

But should your change be a test at all? To know whether your test design is fit for purpose, ask yourself the following,

Using an experimentation tool that enables you to get a single unified view of the user can help a lot when assessing the impact of an experiment and enable you to use historical data for targeting purposes. This is useful in scenarios such as the one Léa mentions; 

To get this unified view, you must use a testing tool that consistently buckets users. It ensures that users receive a uniform experience across sessions and devices. If you have logged-in users, you should also ensure that the feature flagging platform can show them the same variant, whatever devices they use. 

For mobile app testing at Booking.com, they don’t use cookies but measure mobile app users based on their device ID, giving them more reliable data. This unified user view helps you understand the real impact of your tests: 

3. Collect qualitative and quantitative data within your testing process 

A/B and analytics data are great but do not represent the whole picture. Often, qualitative data fails to uncover the why behind user behavior. Umashankar recommends the following,

Experiments, whether for mobile or web, that are based on signals across multiple data sources help reduce bias and enhance the validity of your hypothesis. So, collecting this type of quantitative data is worth the extra effort. 

4. Prioritize mobile app tests that matter to your customers 

Just as you would for web experimentation, you must prioritize what tests you’ll run. Inae provides a valuable way of identifying what tests are most important.

In addition to this, you can also use the same prioritization frameworks used for web-based experiments, such as PXL, PIE, or ICE. Once tests are proposed across experiences, you can add them to your A/B testing roadmap to help coordinate efforts and ensure there are no conflicting experiments. 

5. Remember not all mobile experiences are created equal

The key to mobile app testing is to recognize the unique aspects of each platform. When you see differences in test results between iOS and Android, they could be due to different audiences/behavior, but more often than not, there’s a more straightforward explanation; 

Check the test implementation difference before trying to understand if it’s due to the slight audience difference across platforms. 

6. Parity across mobile apps isn’t the goal

Many businesses, if they have native mobile apps, will create iOS and Android versions that are effectively standalone experiences. The innate feeling is that these apps should mirror each other so all users get the same experience, but Jorden has some advice on this, 

6 Practical tips for getting started with mobile app testing

1. Start small: Begin with simple A/B tests, focusing on one element at a time. This helps isolate the impact of changes.
2. Track the right metrics: Define clear success metrics upfront to objectively assess your tests' results.
3. Collect qualitative and quantitative data: Supplement A/B test data with user feedback to understand the "why" behind user behavior and ensure user-centric optimizations.
4. Prioritize mobile app tests based on customer needs: Focus on testing elements most valuable to your customers by prioritizing features aligned with user needs.
5. Remember, not all mobile experiences are created equal. While iOS and Android users might behave differently, most often, differences in results between the two are due to implementation discrepancies.

6. Parity across mobile apps isn't the goal: Don't chase parity across platforms; prioritize value creation over identical user experiences.

    

Thanks to all of the contributors: 

• Sylvain Gauchet, Director of Revenue Strategy, US at Babbel
• Umashankar Shivanand, Head of Product at SleepScore Labs 
• Inae Lee, Growth Product Manager at Capital One
• Léa Samrani, Product & Growth Consultant 
• William Klein, Product Growth at WeWard
• Hannah Parvaz, Founder at Aperture 
• Jorden Lentze, Senior Product Manager Apps at Booking.com

If you'd like to learn more about mobile app testing, check out this webinar recording, in which Jorden Lentze, Senior Product Manager, shares real-world examples of his work from Booking.com.