What is a “Center of Excellence” in experimentation?

This interview is part of Kameleoon's Expert FAQs series, where we interview leading experts in data-driven CX optimization and experimentation. Luís Trindade started out at FARFETCH leading the Data Products area, but his desire to understand how users interact with products led him to his current role as Principal Product Manager for the experimentation area, where he focuses on promoting a test & learn Center of Excellence across the company. FARFETCH has recently won the Experimentation Elite Award 2022 and the Experimentation Culture Award 2020.

What is a “Center of Excellence” experimentation function, and what would you say are the pros and cons of using this structure?

A Center of Excellence (CoE) is an organization model. It enables all other areas/teams transversally to do more, run better experiments, and nurture the creation of a mindset of experimentation. It defines best practices, provides guidance, processes, and provides tooling to achieve those goals.

This model works best in organizations with some structure and maturity. Otherwise, it may create additional friction and processes that a smaller organization may not need. For less mature companies in terms of experimentation, a centralized model helps define how experimentation can work in the organization, be inspirational to leadership and help accelerate initial adoption and practices. But a centralized model will quickly become a bottleneck when you start scaling up the number of experiments.

What steps should be taken to develop a CoE?

First, start by measuring your maturity level and know where you should focus. Maybe the first steps are not necessarily about changing your organizational model. The Experimentation Growth Model is a great example of a tool (and its underlying paper) to help you do that assessment for your organization.

The main challenge in setting up a CoE is the capacity to influence change and truly transform the organization's culture. Having support from the leadership is crucial to provide the resources (tooling and team) and embrace the need for change at different levels. For example, changing your ways of working with fast product discovery cycles, a single leading success measuring framework, and a mindset of learning instead of winning/losing.

Another pain point is the scalability of a CoE: the tendency is to fall back to the centralized model and rely on the team of experts that can help you do your job, but that model quickly becomes a bottleneck.

To help promote more sustainable growth and adoption of the CoE, its ways of working and mindset, focus on

• Building a network of champions/ambassadors inside each team/area who share knowledge regularly.
• Enabling those champions (who are subject-matter experts) to identify their pains/challenges or blind spots in the organization.
• Providing open forum sessions where recurrent knowledge is shared, questions are asked or hypotheses are peer-reviewed.

Where do you set the boundaries of what the CoE team does?

A CoE team's mission should be about enablement. It’s the same as the adage to teach someone to fish so they can eat for a lifetime, rather than providing the fish so they can eat for a day.

Providing tools and sessions/forums that promote visibility and collaboration of what is going/being tested will provide a continuous feedback loop elevating the quality of the experiments and therefore help increase the maturity of the teams. The CoE team should provide the guidance, tools, and support to keep pushing the boundaries for everyone.

To help create this type of ownership and responsibility amount teams, avoid using a central ‘sign off.’ Instead, provide guidelines on how to create good hypotheses, how to read test results, how to make rollout decisions etc. Be open and encourage the sharing of examples where something could be better for the next iteration. Teams cannot be afraid of failing, and the CoE needs to be there to support them and help them continually improve.

Accepting that the experimentation program will be a continuous learning process is important; learning and iterating on what works or doesn’t work is as important as the tests themselves. Some tactics used by other companies or teams/areas could work for some, by maybe not for your organization. One size doesn’t fit all.

What skillsets/roles make up an effective CoE experimentation team?

I’ll give you an example of how we are structured at FARFETCH:

• Lead for the Experimentation CoE. In this case, I lead the experimentation CoE with skills in Product Management, Engineering, and Data. The main focus is on defining practices and guidelines and coordinating the team. I’m also playing the role of a Product Manager for our in-house experimentation platform.
• Principal Engineer. Skilled in Data and Architecture and acts as an interface for all the supporting areas across the company.
• Two Engineering teams with 4-5 people on each team with Backend and Frontend skills. These people are responsible for evolving our Experimentation platform.
• One Data Science team. This team is responsible for building our own StatsEngine and reporting tools.
• Non-dedicated team members. This is our network of ambassadors spread across the different teams/areas, including Analytics, Data Platform, Product (multiple domains), Engineering, and Agile coaches.

In general, having a small team with strong knowledge about Product Management (as experimentation is a key element for any product development) and statistics is essential.

Likewise, change management skills are very important. Partnering with a Product Ops team can help here if you have them in your organization.

Finally, depending on the tooling decision (build vs. buy), you need to invest in having an engineering and data team to support and evolve it or at least integrate and maintain the external vendor tools.

What do you feel is the biggest challenge for experimentation teams working in fashion e-commerce today?

A key challenge we face every day is due to our luxury positioning in the industry, which requires us to offer an exceptional experience to our customers. Finding the right balance between running bold fast experiments with the natural expectations management that it implies vs. keeping an expected flawless experience is tricky and challenging. But experimentation also provides you with all the risk mitigation tools needed.

Being a platform player, we also run many experiments that are at the service level, which don’t impact the customer experience as much but allow us to learn and improve our platform services. This opens up the possibilities to test and optimize services for different tenants and understand how to personalize the offering for their specific customer base. Evolving an experimentation platform (and the corresponding data layer) to work at this scale is indeed one of the biggest challenges we are currently facing.

You’re a mentor at several startup accelerator programs. Can you share your best advice for supporting early-stage innovative ideas?

In the past, I’ve worked as part of a company builder venture, helping to bootstrap and validate many business ideas and opportunities, from a paper vision to a working prototype that we could test with real customers to validate product-market fit. And then sometimes launching successful products/companies, but many times failing.

The best advice I would give is that you need to embrace failure, rebrand it as learnings, and use those learnings as stronger insights for your next iteration. By applying the scientific method, always predict what the outcome will be and define your success metrics clearly BEFORE running your experiment, and then accept the learnings and iterate. By “experiment,” I’m not talking about A/B tests per se. You can test a business model/concept/idea or actually anything you do because a true experimental mindset can and should be applied to anything in our lives, becoming a way of living and learning continuously.