How to build a culture of product experimentation across your organization
Alexandre Suon Perhirin, Head of Experimentation at the Accor Group, explains how he moved the company towards a culture of product experimentation.
What’s the difference between A/B testing and experimentation?
A/B testing is the simple practice of splitting web traffic to compare a variant with its original version and analyzing the results. Experimentation, however, is a global approach. It's the entire process. From creating an idea, formulating a hypothesis, and running the A/B test, to analyzing the results and the lessons learned from them.
Experimentation is a useful approach within the product development cycle:
- In the Discovery phase: Experimentation is an excellent research tool. It enables you to validate hypotheses before product development and prioritize what needs to be developed according to its value.
- In the Delivery phase: It allows you to secure the launch of new features with a gradual or targeted roll-out and to iterate easily without altering the user experience.
Why put a product experimentation strategy in place?
The four main advantages of adopting a product experimentation approach are:
- Continuous improvement: The results of your tests enable you to optimize the user experience and your feature
- Better resource allocation: By testing your hypotheses, you can identify the best opportunities (UX, Business, etc.), prioritize your product roadmap and save on unnecessary and costly developments
- Secure launches: Thanks to Feature Experimentation, you reduce risks de facto by launching only market-ready and user-approved features
- A data-driven approach: Experimentation enables you to obtain a large quantity of data so your decisions can be based on facts instead of intuition.
What do you need for a product experimentation strategy?
There are three key elements you need to build a culture of experimentation:
- Platform: Choose the most comprehensive platform possible. One with the fewest limits in terms of your experimentation wants and needs, it needs to evolve as you evolve. To start with, you probably won’t need advanced functions like Feature Management, server-side testing or AI-driven personalization, but it’s likely your experimentation program will grow into them in the future.
- Process: Processes will depend on the level of experimentation and the environment (the number of people involved, the type of product, the structure of the teams) of your organization. Several processes may co-exist within the same company. Remain open and flexible – adopt the approach that best fits your organization and adapt as you gain maturity.
- People: Any large-scale transformation is a potential source of friction. Be diplomatic. Rely on the people who support you, and don’t antagonize your critics. You should be aiming to make them a supporter of your program.
How did Accor implement a product experimentation culture?
At Accor, we went through four stages. That won’t necessarily be the case for every company. Depending on the size of the organization and the maturity of your experimentation program, you may need to add or skip steps. Before, at Accor, there was no culture of experimentation. Our backlogs were full of features, but none addressed actual customer issues.
STEP 1 - Pitch
First, the company needs to familiarize itself with experimentation. Then gradually acclimatize the organization to the benefits of testing. At Accor, this stage lasted around a year, during which we ran close to 25 tests, without a real roadmap or prioritization.
- Platform: Start with the basics of web experimentation. Launch simple A/B tests and validate more advanced hypotheses by running multivariate tests, which involve testing multiple variables. At this stage, a simple graphic editor and a code editor should suffice.
- Process: When setting the direction you need to move towards, keep your business goals in mind. Generally, at this stage, the resources allocated to your experimentation will be limited. This is why the processes will be key to moving on to the next stage.
- People: Surround yourself with people who are prepared to support you and likely to bring others along for the ride. Be inspiring! At Accor, my first team consisted of a Consultant (me), a Business Owner and a Product Owner. We weren’t even a real team but more a branch of another team.
Example of a level-1 test:
Accor’s first experiment was a simple test that involved modifying some text and a CTA on the home page. The analytical data showed a low click rate on the offers found in this location.
Using Kameleoon’s WYSIWYG editor, the team implemented a no-code A/B test, which showed an increase of over 70% in clicks on the “Book” CTA. We then rolled out the variation to be visible to all site visitors.
STEP 2 - Share and square
Your organization’s stakeholders now understand the value of experimentation and are starting to request more experiments. The various teams – Product, Design and Marketing – are now putting forward hypotheses. At this point, it’s all about structuring and emphasizing efficient resource allocation. At Accor, this step lasted almost two years. The staff dedicated to experimentation were drawn away from the Analytics department to form an Experimentation team, serving all the other teams (Design, Dev, Product, and Business). At the time, they were launching between 25 and 50 tests per year.
- Platform: Tests are increasingly frequent, and you must carefully study the resulting data. Your testing solution must perfectly integrate into your tech stack (particularly with your analytics tools) and enable server-side testing with a smooth connection between your backend and the platform. Thanks to an adapted SDK.
- Process: This is where the process begins to develop. The number of tests means you need to create a backlog plus an order of priority. Then accurately communicate your needs and expectations according to the data and results of your testing.
- People: At Accor, the few people dedicated to experimentation now created a team of around ten staff, including the Head of Experimentation, a Product Owner and a group of developers.
Example of a level-2 test:
For this server-side test, Accor observed one of their competitors was showing the number of photos in the gallery on their hotel search. The experimentation team wanted to assess how this detail impacted visitor conversion rates.
This test showed that the number of visitors clicked on the gallery. However, those already ready to convert were less likely to do so, indicating the importance of calculating customer conversion probabilities.
STEP 3 - Develop
After establishing a testing practice, your team will shift your focus onto the development phase, following your key organization objectives (OKR). Focusing on the quality and impact of your tests will raise awareness among your teams of the “Test & Learn” concept. At this point, experimentation will become almost systematic in all business decisions. You understand which projects to concentrate on and which add the most value to the organization. Accor launched around 100 tests per year at this stage.
- Platform: As you’re now using increasingly advanced tests, you’ll probably soon need to test more “sensitive” parts of the product or launch new features with greater security. Feature Management will enable you to measure the impact of your new feature with a gradual and/or targeted roll-out.
- People: Your team is growing, and more people within the organization must understand experimentation. Here, it's important to educate teams on the importance of experimentation and how their team may be able to utilize it in the future.
- Process: The main difficulty in the growth of experimentation is aligning all stakeholders and teams with the same business and product priorities.
- The results of all the teams testing within the Kameleoon reporting and analytics allow us to create shared goals across the organization, regardless of their expertise.
Example of a level-3 test:
Accor was able to critically review the design of the hotel product page within its customer journey by taking inspiration from its competitors. This test required significant graphic and structural changes to the page that were impossible to implement with a graphic or simple code editor.
STEP 4 - Innovate
The last step happens when experimentation drives innovation. It has been fully integrated into the company culture and is a lever in making strategic business decisions. How do we decide whether a partnership would be beneficial? What ROI can we expect from a new feature? It is helpful to reinforce your experimentation strategy at this stage by broadening the scope with advisors and custom tech offerings. Accor has just entered this stage, reaching over a hundred tests a year.
- Platform: Now your experimentation solution must help you run increasingly specific tests. If you want to give your visitors an even more comprehensive experience, you can base your tests on AI-driven personalization. The platform must also be perfectly compatible with your Data Warehouse to ensure the accuracy and quality of your data. The tool must be flexible enough to adapt to all your use cases and create “customized” features.
- People: Experimentation has gradually become a business validation tool for leaders. At Accor, the team reports to the Senior Vice-President and the Chief Data Officer, who are actively involved in defining the team’s roadmap and strategy.
- Process: Experimentation now represents around 70% of the backlog for all teams, regardless of their field. For some teams, it’s 90% of the backlog, meaning that only the legal elements are integrated without being tested in advance. Make it a priority to allocate your resources to the tests that will have the greatest impact, following the company’s overall strategy.
Example of a level-4 test:
Accor recently launched two types of tests characteristic of this level of innovation.
In the first test, Accor compared a customer-opinion-collection solution built using Kameleoon with a solution offered by a SaaS tool.
In the second one, Accor tested visitors’ interest in taking out insurance before moving to the payment stage. This test enabled them to understand if there is a need for a partnership with AXA.
In both cases, experimentation enables us to decide whether an investment is beneficial. Following each of these tests, we can make business decisions based on data, not intuition. This kind of test can also be decisive for e-commerce businesses since it allows them to test the addition of a product to their catalogue before entering a commercial agreement. Thanks to a simple test, you can obtain a reliable conversion rate outlining whether you should or shouldn’t add the product.
What would you say to a product team that wanted to expand its experimentation strategy?
Above all, experimentation can make your life easier by providing the data you need to make the right decisions. Going forward, always focus on the "3 Ps: People, Process, and Platform", and not what you do nor the quantity, but how you do it.
